Amortising Inference and Meta-Learning Priors in Neural Networks
Authors: Tommy Rochussen
Vincent Fortuin
Venue: ICLR 2026 Poster (2026)
Keywords: neural processes
Bayesian neural networks
meta-learning
priors
variational inference
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)

Abstract: One of the core facets of Bayesianism is in the updating of prior beliefs in light of new evidence$\textemdash$so how can we maintain a Bayesian approach if we have no prior beliefs in the first place? This is one of the central challenges in the field of Bayesian deep learning, where it is not clear how to represent beliefs about a prediction task by prior distributions over model parameters. Bridging the fields of Bayesian deep learning and probabilistic meta-learning, we introduce a way to $\textit{learn}$ a weights prior from a collection of datasets by introducing a way to perform per-dataset amortised variational inference. The model we develop can be viewed as a neural process whose latent variable is the set of weights of a BNN and whose decoder is the neural network parameterised by a sample of the latent variable itself. This unique model allows us to study the behaviour of Bayesian neural networks under well-specified priors, use Bayesian neural networks as flexible generative models, and perform desirable but previously elusive feats in neural processes such as within-task minibatching or meta-learning under extreme data-starvation.

Published as a conference paper at ICLR 2026
AMORTISING INFERENCE AND META-LEARNING PRIORS IN NEURAL NETWORKS
Tommy Rochussen 12233 & Vincent Fortuin 1.2,4
1 Helmholtz AI, Munich Center for Machine Learning, 3 Technical University of Munich,
4 University of Technology Nuremberg.
vincent fortuinntteelmhhltzz munich de
ABSTRACT
One of the core facets of Bayesianism is in the updating of prior beliefs in light of new evidence- how can we maintain a Bayesian approach if we have no prior beliefs in the first place? This is one of the central challenges in the field of Bayesian deep learning. where it is not clear how represent beliefs about a prediction task by prior distributions over model parameters. Bridging the fields of Bayesian deep learning and probabilistic meta-learning, we introduce way to learn a weights prior from a collection of datasets by introducing a way to perform per-dataset amortised variational inference. The model we develop can be viewed as a neural process whose latent variable is the set of weights of BNN and whose decoder is the neural network parameterised by a sample of the latent variable itself. This unique model allows us to study the behaviour of Bayesian neural networks under well-specified priors, use Bayesian neural networks as flexible generative models, and perform desirable but previously elusive feats in neural processes such as within-tas minibatching or meta- leat-le under extreme sttaastarvation.
INTRODUCTION
While the ability to learn hierarchical representations of data has allowed neural networks to boast phenomenal predictive performance across many domains, enabling them to accurately estimate their predictive uncertainty remains generally unsolved. Bayesian deep learning (MacKay, 1992) promises theoretically sound approach for endowing representation learners with uncertainty quantification, but there are many difficulties with the approach in practice. Of all of them, one of the most dleep-deprivine is the question of how to choose appropriate priors. Neural network weights lack interpretability, meaning it is fiendishly difficult to elicit priors over them that are sensible in prediction space (Fortuin, 2022). The convenience priors that generally use are known to reduce large' Bayesian neural networks (BNNs) to the "simple smoothing devices" (MacKay, 1998) that are Gaussian processes (GPs; Neal, 1995; Matthews et al., 2018; Yang. 2019), meaning that in trying to achieve uncertainty quantification we inadvertently destroy the ability to learn hicrarchical representations (Aitchison, 2020). An increasingly popular approach to specifying priors in BNNs is to use smnetion-space priors However, these priors are most often chosen to GP priors which, yet again, reduce Bayesian deep learning to approximate GP inference. Even when priors and architectures are chosen specifically to avoid GP behaviour, the resulting stochastic process prior is not just poorly understood, but generally bad model of the real-world data- gettrrener process (Karaletsos and Bui, 2020).
et al., 2017; Sun et al., 2019; Cinquin et al., 2024).
On the other hand, neural processes (NPs; Garnelo et al., 201 20:8);) use the shared structure between related tasks meta-learn a free-form stochastic process prior. Unlike GPs and BNNs, they do not learn an explicit parametric prior that can be evaluated or sampled from, but, given some context observations. they learn map directly to the stochastic process posterior corresponding to the learned implicit prior process. For a sufficiently flexible NP architecture and with enough data, they can model the ground-truth data-generating process (Foong et al., 2020). Given this remarkable ability, neural processes have enabled practitioners to endow representation learners with uncertainty quantification to great success across a range of domains including weather and climate applications (Allen et al., 2025; Ashman et al., 2024b; Andersson et al., 2023), causal machine learning (Dhir
In the sense of infinite architecture width, architecture depth, number of channels, on.

---

Published as a conference paper at ICLR 2026
et al., 2025), Bayesian optimisation (Maraval et al., 2023; Volpp al., 2023), and cosmological applications (Park and Choi, 2021; Pondaven et al., 2022).
Drawing inspiration from neural processes, the shared structure across a related set of tasks. In other words, can we use meta-learning to design well-specified priors in Bayesian deep learning? To that end, devise scheme for performing per-dataset amortised inference in BNNs. This results in a neural process whose latent variable is the set of weights of a BNN, and whose decoder is the neural network parameterised by particular latent variable posterior sample. We refer to our model as the Bayesian neural network process (BNNP). The unique ability of the BNNP to amortise BNN inference and meta-lean BNN priors enables us to investigate such previously unanswerable questions as: "under well-specified prior, how important is the approximate inference method in Bayesian deep learning really?". Furthermore, as a new member of the neural process family (Dubois et al., 2020), the BNNP introduces some completely novel capabilities. These include the ability to perform within-task minibatching for scalability to massive context sets, and the ability to adjust the flexibilityy of the learned prior so that overfitting can be avoided settings for which only a few tasks have been observed.
interested in meta- meta-learning BNN priors that encode
THE BAYESIAN NEURAL NETWORK PROCESS
2.1 LAYERWISE INFERENCE
We start by considering the conditional posterior over the last-layer weights W E Rdd.-1 in an L-layered multilayer perceptron (MLP), where di denotes the number of units the /-th layer of the network. In this exposition we do not consider biases but they straightforward to include in practice. Given some data D = {X, Y} where X E R= and Y E gaxd as well as the weights of the previous layers WI:L11 = [= and assuming Gaussian likelihood, the conditional posterior over the last-layer's weights is of the form
dr II N (Innaa 2(x o d=1 n=1
p(W4/WHL-1 D) ((W)
where Vn.a is the d-th dimension of the n-th target, Φ6-) is the MLP's elementwise nonlinearity, x x denotes the activations of the penultimate layer for the n-th datapoint, w/ is the d-th column of W representing all input weights to unit d in the output layer, and aa is the observation noise level for the d-th dimension of the targets. Assuming that the prior p( p("") is conjugate, this posterior is available in closed-form via Bayesian linear regression (BLR; Bishop, 2007). Inspired by this result, Ober and Aitchison (2021) propose variational posterior for BNNs and deep GPs that decomposes into a product of layerwise conditionals
L
= II g 9 w 1=1
(2)
where the layerwise factors computed via exact inference between the layerwise prior and a set of iriiationllly-prameteriie pseudo-likelihood terms. We adopt a similar approach.
2.2 AMORTISING LAYERWISE INFERENCE
We generalise the likelihood of the last-layer weights seen Eq. (1) the weights of the (-th layer as follows
di II N (Uhai w'a onia d=1n=1
x w
(,(w))
where vha is a pseudo-observation corresponding to the d-th activation of the 1-th layer for the n-th datapoint, and oh is the noise level for the corresponding sseudo-likeihhod term. Note that x = X. These two parameters are obtained by passing the n-th input-output pair through an inference network 9a,:
y log(o",) = go, (Xa. Yn)
2

---

Published as a conference paper at ICLR 2026
((X)1)
XW
X
Amortised Linear Layer
Amortised
Linear
o(X 9)
BLR
(W! X !.)
Layer
X.
Amortised
Linear
BNNP
Layer
De
(a) Amortised linear layer
(b) BNNP
Figure I: Computational diagrams (a) the amortised linear layer, and (b) a BNNP with one hidden layer of activations. We use the context "e and target notation to distinguish between inputs with labels, on which we condition, and inputs without labels, which we predict.
where 0 denotes the parameters of the 1-th layer's inference network, which will be optimised during training". In the case of the final layer we can use the actual observations, that is, Y Y. With the pscudo-likelhhoad parameters, and assuming unitwise-factorised Gaussian the approximate layerwise priors Pe; ( (W") I = N' (w's pta E/) where t/n (112 E d' posteriors are computed in closed form:
(W W DD) D) x Y
II N (w's Eb) N ( (x w A/ d=1
di = II N (w/r m'y St))
(5)
where A E NxN is diagonal precision matrix with the n-th diagonal element given by and the posterior mean vectors m and covariance matrices S are given by
(o
/ -1 = Σ o A (xx''11)
(6)
m S E 14 o (x (x Aarl)
We consider inference under Gaussian priors with different factorisation structures in Appendix B.
This machinery, which call the amortised linear layer, enables inference the weights of linear layer that is situated arbitrarily within a neural network architecture, conditional on the weights of the previous layers. By stacking these layers and sampling from each layer's conditional posterior before computing the next layer's posterior, we can perform amortised inference in a BNN via the variational posterior
L II (w' wi:-11,2
(WID)
(8)
I=1
I ( Y!).
(9)
Since such model may be interpreted latent-variable neural process (Garnelo al., 2018b) in which the latent variable is the set of BNN weights and the decoder is the BNN itself, we refer to this model the Bayesian neural network process (BNNP). See Fig. for computational diagrams of the amortised linear layer and BNNP.
2.3 TRAINING
We adopt a variational approach to training, where the parameters be optimised are the inference network parameters θ = (0: M and the prior parameters V = { | The former take the role
Another option is pass the concatenation of an input-output pair well the previous layer's 's activation x into each inference network. It is possible this could allow for smaller inference networks.
3

---

Published as a conference paper at ICLR 2026
of variational parameters while the latter model parameters. We motivate choice of training objective by considering what behaviour we would like to encourage in the BNNP. We summarise these considerations into the following three desiderata:
(I) accurate approximate posteriors,
(II) a prior that faithfully encodes the data-generating process,
(III) high quality predictions.
Assuming access to a meta-dataset E = {D of tasks that share data-generating process
D(J) pDD,, and partitioning datasets into disjoint context " and target : sets, propose novel objective function for training the BNNP. We call it the ceterior-predictivv amortised variational inference (PP-AVI) objective, and it defined as
LPP-AVI (=) = =
Σ logg (X + CELBO (D!
(10)
where the first term is the log pesterior-predictiin density of the target set, and (E) denotes the usual evidence lower bound (ELBO) used for VI in BNNs. The first term isolation is the usual objective for NPs, reflecting that NP users often only care about predictions (Foong et al., 2020). Since we intend use the BNNP to study BNNs, we introduce the ELBO term to explicitly encourage high-quality approximate inference.
Proposition 1. For | + oc, maximisation of (Pr-Av(E)) directly targets the three desiderata.
proof of Proposition involving formal definitions our desiderata, practical guide to the PP-AVI objective, and discussion relating it to other objectives in the literature provided in Appendix A. We emphasise here, though, that (p)) can be unbiasedly estimated from a minibatch of tasks E C E via CPP-AvI(E), enabling training huge meta-datasets via stochastic optimisation.
2.4 WITHIN-TASE MINIBATCHING VIA SEQUENTIAL BAYESIAN INFERENCE
When making predictions on a particular task, the full set of real and pseudo observations must be stored memory order compute the posterior. So, while stochastic optimisation allows to scale to large meta-datasets, scalability to large context sets remains elusive. Fortunately however, we provide a solution this problem too. The high memory requirements can avoided by iteratively updating each layer's posterior with a minibatch of datapoints via sequential Bayesian inference (Bishop, 2007), temporarily discarding each minibatch after applying its update to particular layer's conditional posterior.
We partition a task D into minibatches { We use W 9 to denote a sample from the 1-th layer's posterior given only minibatch b, that is, W/ q(( 1,)). Since computation of each layer's posterior requires samples from the full-batch previous layer posteriors (i.c., W1:-11 ' the minibatching procedure must be performed in full for each layer before sampling and proceeding to the next layer. The ability minibatch forward pass context of neural processes, and it is valuable property as it allows us to scale to tasks with large and dineedimensional datasets without running into memory limitations. Although some extra stochasticity is incurred when training, at prediction time our minibatching procedure results in the exact same approximate posterior as the full-batch procedure. See Appendix C for detailed algorithmic description of the procedure and further discussion including complexity analysis of the use similar sequential inference trick devise an online-learning scheme
given context is rare in the
BNNP. In Appendix for the BNNP through which predictions for a given task can be updated in light of new data.
2.5 ADJUSTING THE FLEXIBILITY OF THE PRIOR
One limitation shared amongst neural processes is their poor performance when the number of observed tasks is limited (Rochussen and Fortuin, 2025). This happens because the model parameters are responsible for both amortising (predictive) inference and encoding a prior, such that there is way to discourage the learned prior from overfitting without also affecting the model's ability to
4

---

Published as a conference paper at ICLR 2026
amortise prediction. In the BNNP however, these roles are disentangled into two distinct parameter groups; those of the weights prior, and those of the inference networks. This separation enables us to limit the flexibility of the learned prior without affecting inference. We do this by fixing the prior over a subset of the weights and optimising only the remaining prior parameters. For example, we can fix the prior over the last layer's s weights while optimising the prior parameters for the carlier-layer weights. By varying the number of weights whose prior is fixed, we introduce new knob with which to tune the flexibility of the learnable prior. Such a scheme enables practitioners to balance between forcing broad/misspecified prior and learning an overfit prior, leading to better generalisation in the small meta-dataset regime.
zero-centered and unit- diagonal Gaussian
2.6 ATTENTION
One way extend the BNNP would be go beyond MLPs and incorporate a more sophisticated neural architecture, such as the attention layer ( (Vaswani et al., 2017). We outline two ways to do this. As is common in the NP literature, the encoder can be augmented with attention blocks (Kim et al., 2019; Nguyen and Grover, 2022). In the BNNP, this means parameterising the inference networks {g as transformers rather than MLPs, and processing the full context set together each layer. This could improve performance since modelling interactions between context points could lead to better pscudo-likelhhod parameter estimates than when we process each point independently, However, an attentive encoder incurs an extra computational cost of Onn)). Furthermore, application of our within-task minibatching scheme under an attentive encoder would lead to different posteriors to the full-batcl forward pass due to the new dependencies between context points. Nonetheless, we refer to this variant of the model as the attentive BNNP (AmBNNР).
Alternatively, we focus the decoder. The core technology that we introduce is way to amortise inference over the weights of a linear layer situated arbitrarily within a neural network. Since attention block is just a composition of linear layers and various nonparametrie operations?, it is possible to amortise inference in an attention block by amortising the inference in each linear layer therein. If such amortised attention blocks are stacked, we end up with transformer whose weights posterior is obtained in-context. A more sophisticated decoder could enable us to model more complex tasks. However, such a decoder processes the target locations together such that the prediction for a particular target depends on the other target locations. This property destroys the consistency of the model and therefore no longer results in a valid stochastic process. We therefore refer to this version of the model the Bayesian neural attentive machine (BNAM), omitting any reference to stochastic processes from its name. Note that the BNAM also incurs an extra O(n)) computational cost. For computational diagrams as well as further details on the lack of consistency of the BNAM including a simple demo, see Appendix E.
RELATED WORK
Neural Processes. The BNNP is new member of the NP family (Garnelo et al., 2018a; Dubois et al., 2020), and in particular the vatenl-v NP family (Gamelo et al., 2018b; Singh et al., 2019; Lee et al., 2020; Foong al., 2020). The AttBNN is also a member of the transformer NP family (Kim et al., 2019; Nguyen and Grover, 2022; Feng et al., 2023; Ashman et al., 2024a;c). Our model is different to existing latent-variable NPs in that our latent variable is the parameterisation of the decoder itself, and not an abstract representation of the context set that is passed to the decoder. While Rochussen and Fortuin (2025) adopt a similar approach, in this work the decoder is a BNN rather than sparse GP. Inference in the BNNP can be viewed as an instantiation of Volpp et al. (2021)'s Bayesian context aggregation mechanism- in both settings we have an encoder that maps from individual context points to pseudo-likelihood terms with which the latent variable's 's posterior is obtained through exact inference. While we choose to exclude the BNAM from the NP family due to its lack of consistency, some existing NP variants also lack consistency; Bruinsma et al. (2023)'s NP variants and Nguyen and Grover (2022)'s s TNP-A both fail to produce consistent predictive distributions due to autoregression amongst targets, while Nguyen and Grover (2022)'s 's TNP-ND is
Such as residual connections. We do not consider inference over the aaee-norm parameters, justifying deterministic treatment of them through their comparative inability to overfit.
5

---

Published as a conference paper at ICLR 2026
more similar to the BNAM, with attention performed between targets*. Our training and modelling setup is also closely related to that of Gordon et al. (2019), with the first term of our proposed training objective being exactly equivalent to theirs (see Appendix A).
Approximate Inference in BNNs. There have been considerable efforts to develop more accurate and scalable approximate inference methods for BNNs in recent decades. There are Markov-chain Monte-Carlo algorithms (Neal, 1992; Welling and Teh, 2011; Sommer et al., 2024), ensemble and particle approaches (Lakshminarayanan et al., 2017: D' Angelo and Fortuin, 2021; Liu and Wang, 2016), Laplace approximations (MacKay, 1992; Ritter et al., 2018; Immer et al., 2021), variational strategies (Hinton and van Camp, 1993; Graves, 2011; Louizos and Welling, 2017), as well as more standalone solutions (Maddox et al., 2019; Gal and Ghahramani, 2016: Hernandez-Lobtt and Adams, 2015). Our approach is most similar to Ober and Aitchison (2021)'s global inducing point variational posterior since we factorise the variational posterior into layerwise conditionals and compute each one through exact inference under pseudo-observations. Unlike their approach, in each layer we have a pseudo-likelihood corresponding to every datapoint rather than for limited set of inducing locations; we parameterise the pseudo-likelhhood terms via inference networks rather than directly; and we use broader class of Gaussian priors. Another related method is Kurle et al. (2024)'s BALI, which adopts a non-variational approach to parameterising the paeudo-likelihon terms in each layer. Separately, our work shares the use of secondary/inferenee""lyy with normalising flow-based approaches (I (Louizos and Welling, 2017) and Bayesian hypernetworks (Krueger et al., 2018), but the difference is in what the secondary networks map between-for from observations to pscudo-likelihodd terms and for them it is from base distribution samples to posterior samples. Finally, our approximate inference scheme is an instance of structured variational inference (Hoffman and Blei, 2015).
networks
is
Priors in BNNs. In general, there no widely accepted way to select well-specified priors in BNNs (see Fortuin, 2022). While mmeta-)learning priors is not a new concept (Rasmussen and Williams, 2006; Patacchiola et al., 2020; Fortuin et al., 2020; Rothfuss et al., 2021), it is relatively under- explored approach in the context of Bayesian deep learning. To this end, Fortuin et al. (2022) analyse the empirical distributions of trained neural network weights to construct new priors, Shwartz-Ziv et al. (2022) use the approximate posterior from large pre-training task as learned prior for downstream fine-tuning tasks, and Villecroze et al. (2025) adopt an empirical variational Bayesian approach to learning prior over the last-layer weights of a BNN. Ravi and Beatson (2019) and Rothfuss et al. (2021) consider meta-learning BNN priors, with the latter even doing so via AVI. However, they both do so within hierarchical Bayesian framework involving restrictive hyper-priors. Concurrently with this work, Park et al. (2026) also devise scheme for meta-learning BNN priors, but their weights posterior is obtained in-context via a network lincarisation strategy rather than amortisation. They also propose mixture of priors-out our results suggest this is unnecessary.
4 EMPIRICAL INVESTIGATION
4.1 HOW GOOD IS THE BNNP'S S APPROXIMATE POSTERIOR?
We begin by evaluating the quality of approximate inference in the BNNP. We adopt similar setup to Bui (2021) and, under a common BNN architecture with fixed hyperparameters (including V). measure the gap between the log marginal likelihood (LML) and the ELBOs achieved by various VI techniques. The difference between these two quantities represents the KL divergence KL| (W()) Il Pe (W[D)). giving us a clear metric of approximation quality in each case. To eliminate any bias in inference quality due to misspecified priors, we generate a dataset by 1.) sampling a function from the BNN's prior, 2.) uniformly sampling a set of inputs, and 3.) adding Gaussian noise with standard deviation 0.1 to the outputs. Since the data therefore has a high probability of being generated under the prior (because was), we can estimate the log marginal likelihood via Monte Carlo integration. We compare BNNP meta-trained across similarly generated datasets under CPP-AVI. a BNNP trained on just the evaluation task via the standard ELBO objective function, mean-field VI (MFVI; Blundell et al., 2015), Ober and Aitchison (2021)'s global inducing- poinin VI (GIVI), as well as a number of VI algorithms with increasingly high-rank Gaussian variational posteriors: unitwise correlated (UCVI), wyperwise-correltteee (LCVI), and cully-coreelate
In their case, attention is performed between the target tokens only for the predictive covariance module. The predictive means for each target location remain conditionally independent given the context set.

---

Published as a conference paper at ICLR 2026
(FCVI). The results, which are collected across range of likelihood noise settings and averaged over four repeat runs, are shown in Fig. 2. They demonstrate that the BNNP capable of high quality approximate inference. For all methods with smaller likelihood noises. This effect is more pronounced for the unstructured variational approximations (MFVI, *CVI), with these methods observe the same bias in model selection via the ELBO as Bui (2021)-over large are favoured (the true noise level is 10 10-
that approximate inference quality decreases
40
DCST
RXNP
IKNP SUNRO
LMI.
-20
CIV:
NETVI
20
LCVI
OCVI
-10
10
10
(a) (D)
KI. (l pe (WD))
Figure 2: ELBO and KL divergence between approximate and true posteriors for different VI methods. The ENNP aepl-approxsmates the true posterior.
4.2 DO MEANINGFUL BNN PRIORS EXIST?
Here we train BNNPs (including v) on meta-datasets with different data-generating processes. We then qualitatively compare functions sampled from the BNNP's learned prior with those from the true data-ges genta-gnnrating process. We consider synthetic meta-regression datasets generated from random sawtooth functions, Heaviside (/binary) functions, functions from a standard BNN prior'. and synthetic ECG signals, well the MNIST dataset (LeCun et al., 1989) with random pixel masking cast as pixelwise meta-regression dataset (Garnelo et al., 2018a), in which case we use AttnBNNP. The results are visualised in Figs. and 4.
Figure 3: Function samples from the true data-generating process (first column), learned BNNP prior predictive samples (second column), BNNP posterior predictive function samples (remaining columns).
We observe that the BNNP learns functional priors that are almost indistinguishable from the synthetic data-generating processes. The corresponding posterior predictive samples appear to be sensible well, with increased uncertainty away from observations while maintaining the underlying functional structure in each case. While the AttnBNNP's s ability to generate MNIST images might represent the state-of-the--rr, the fact that many of the samples are elearly recognisable digits further demonstrates this section's 's conclusions, which are that 1) BNN priors that faithfully encode complex data-generating processes do exist, and the BNNP can be used to find such priors.
4.3
DOES A GOOD PRIOR DOUBLE AS LUNCH VOUCHER?
To investigate whether the quality of approximate inference still matters under good prior, consider one real-world and three synthetic lata-generating processes. In each case, we meta-train
TTroouhoou empirical investigation, we standard BNN prior to refer to zero-centered fulto-factorised Gaussian priors with variance scaled inversely proportional to layer width.
7

---

Published as a conference paper at ICLR 2026
8
0
5
9
6
4
5
6
3
7
Figure 4: Generative modelling of MNIST digits with the AttBNNNP. Smaller images depict sampled prior functions evaluated at same x8 grid of inputs that training data lay on. Larger ones depict sampled functions queried a 100x 100 gid of inputs. The AttnBNNP's prior has encoded the functional behaviour of handwritten digits, so super-resolution is natively supported without needing further training.
BNNP in order to find well-specified BNN prior for the problem. For variety of approximate inference algorithms, we compare predictive performance achieved under a standard prior versus well well-specified one. This is repeated predictive density (LPPD) and mean absolute error (MAE) as metrics. We consider SWAG (Maddox et al., 2019), MFVI, Langevin Monte Carlo (LMC; Rossky al., 1978), GIVI, the BNNP, and Hamiltonian Monte Carlo (HMC; Neal, 1992). For our real-world setting, we substitute LMC and HMC for their stochastic- graddent counterparts (SGLD; Welling and Teh, 2011, SGHMC; Chen et al., 2014) due to larger context sets and architectures.
16 test datasets and
per-datapoint log posterior
(a) Context set
(b) Predictive mean
(c) Predictive uncertainty
Figure 5: Demonstration of an ERAS precipitation prediction test task with the BNNP. With no context points over Switzerland, the BNNP's predictive uncertainty is increased that region.
As synthetic data-generating processes, we use squared-exponential GP, Heaviside, and sawtooth functions. all three cases the inputs are sampled uniformly and the outputs are lightly noised. of Europe centered
The real-world setting consider is precipitation prediction over Switzerland. We use the ERA5 Land dataset (Muñoz Sabater et al., 2021) I) and use longitude, latitude, challenging,
and temperature input variables. To make the prediction task any context points from Switzerland in the test tasks, meaning predictions in this region are heavily influenced by the choice of prior. Fig. demonstrates the ERA5 test task setup and Fig. 6 displays the results across the four settings. While it is clear that better priors boost predictive performance, we also see that there remains considerable variation in performance amongst the methods when under learned priors. In other words; a good prior is not all you need when working with BNNs since high quality approximate inference is still necessary there no free lunch Bayesian deep learning.
omit
Heavidde
Sawtoot
ERA5
0.50
(7)
6.98
EWisMPnn IMC civi INSPSNI
S0AG toop
5900 MENT
LMC
GMI
Rep IMC
SWir SERVI ant ane nee scar
0.2
Lowand
00
00
5900 MIFV LMO
CAVI
5900 MPAT 1300
BMC
acaaer
Figure 6: Target-set performance of approximate inference algorithms under well-specified prior (blue) and standard BNN prior (brown). The learned prior almost always leads to improved performance.
4.4 CAN A RESTRICTED PRTOR IMPROVE META-LEVEL DATA EFFICIENCY?
We consider two problem settings in which the number of available datasets would typically be few for NPs to be applied. The first setting is recasting of the Abalone age prediction
8

---

Published as a conference paper at ICLR 2026
task (Nash et al., 1994) meta-regression. The three classes corresponding to a specimen's 's sex (male/female/inaant) are used to separate the data into three distinct datasets. The male and female datasets are used for meta- traiaiin and the infant dataset is reserved for testing. There are seven input features including various specimen size and weight measurements. The second setting is based on the Paul15 single-cell RNA sequencing dataset (Paul et al., 2015), and the task is predict how specialised a cell is from 3451 gene expressions. We perform PCA on the data to obtain 100 information-rich abstract features, and split the dataset into 19 subsets according to cell clusterings that Paul et al. (2015) provide. We randomly select ten for meta-traininn and one for testing.
LPPD
(t)
(1)
-1.5
CIVECIENE no at D2
03
de 1o SPIMPARTTO TNP
NPV GIVI 60
03
0.
60
10
KP
3AR TNP
(a) Abalone
(b) Paul15
Figure 7: Test-task target-set performance for meta-leaming prohlems with limited data. Decimals indicate the proportion of BNNP's prior that is trained. BNNPs with partially trainable priors perform the best.
At the bottom end of the "prior-learnability"" spectrum, we consider MFVI and GIVI baselines with standard priors (trained on the test-task's context set), while at the other, as comparable NPs which produce coherent function samples, we consider the original (latent-variable) NP (Garnelo et al., 2018b), the Bayesian NP (BNP: Volpp et al., 2021), and a transformer NP (Nguyen and Grover, 2022) under Bruinsma et al. (2023)'s state-of-the compare the baselines to the BNNP across its full range of prior-learnabilit where all non-learnable prior parameters are fixed to standard BNN prior settings, and prior parameters corresponding to earlier layer weights are made trainable first. The results are averaged over four trials. In Fig. 7 we see in both problem settings that the best-performing model is BNNP with a partially learnable prior (0.8 and 0.5 respectively). In the particularly data-scarce Abalone problem, we see clear evidence of overfitting in the fully-flexible NPs, including the BNNP-1.0.
autoregressive sampling scheme (AR-TNP) We
DISCUSSION, LIMITATIONS, AND THE BIGGER PICTURE
Why is the BNNP's approximate posterior good as it is? Each layer's approximate posterior is explicitly conditioned on the weights of the previous layers, enabling us model weight correlations between layers. While it is tempting to attribute the high quality to this as Ober and Aitchison (2021) do, FCVT's approximate posterior- -wich also models such inter-layer correlations very poor. One explanation could be that method is unaffected by the posterior multimodality induced by weight-space symmetries. This is because each layer is given access to the outputs of the previous layer as well as its own pseudo-outputs, meaning each layer only "sees" one mode of the posterior.
While it may be surprising that simple Gaussian priors can yield highly complex, even multimodal (Heaviside in Fig. 3, MNIST digits in Fig. 4) stochastic process priors, note that the universal posterior predictive approximation result for mean-field Gaussian posteriors of Farquhar et al. (2020) likely also applies Gaussian priors. We are encouraged by the demonstrated flexibility of Gaussian priors as it means finding good BNN priors is not as insurmountable a task as it might have been. The BNNP provides a solution to this problem for the case when multiple datasets are available, but the problem remains unsolved for the singlc-dataset case.
We highlight that our message is not that the BNNP is the one model to rule them all. When there is limited number of datasets, it is a very useful NP for practitioners to have in their inventory. Otherwise, our main focus in this work was in using it as a scientific tool with which study BNNs. Inference in the BNNP scales unfavourably with architecture width (Appendix C), so we leave investigation of the BNNP's performance as general-purpose NP to future work. Similarly, we introduce the BNAM for scientific interest but we do not include it in our experiments since it is of no use in answering our particular research questions.
Finally, we note that the strong performance of our explicitly Bayesian meta-learning setup hints at an underlying lesson; that Bayesians should be using the abundant data of today's 's world to learn powerful priors, not to be squeezing posteriors out of dubious priors. This message seems to be coming from an ever wver-growing chorus, with the posing of Bayesian inference as an in-context learning problem becoming increasingly popular (Mäller et al., 2025; Reuter et al., 2025; Chang et al., 2025).

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGEMENTS
TR is grateful Adrian Weller and Rich Turner for helpful comments and Matt Ashman for day-to- day supervision on an earlier version of this project (Ashman et al., 2023). VF was supported by the Branco Weiss Fellowship.
REFERENCES
Laurence Aitchison. Why bigger is not always better: on finite and infinite neural networks. In Proceedings of the 37th International Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research, pages 156-164. PMLR, 13-18 Jul 2020. URL html.
https::/proceeddin wtt.ssmmr..prrr..
Anna Allen, Stratis Markou, Will Tebbutt, James Requeima, Wessel P Bruinsma, Tom Andersson, Michael Herzog, Nicholas D Lane, Matthew Chantry, Scott Hosking, et al. End-to-end data-driven weather prediction. Nature, pages 1-8, 2025.
Tom Andersson, Wessel Bruinsma, Stratis Markou, James Requeima, Alejandro Coca-Castro, Anna Vaughan, Anna-Louise Ellis, Matthew A. Lazzara, Dani Jones, Scott Hosking, and et al. Environmental sensor placement with convolutional Gaussian neural processes. Environmental Data Science, 2, 2023. doi: 10.1017/eds.2223..2.
Matthew Ashman, Tommy Rochussen, and Adrian Weller. Amortised inference neural networks for small-scale probabilistic meta-learning, 2023. URL https: /atxiv.ivvvv/////////e//////a wwwrxiivorr//eeo abs /2310. 15786.
Matthew Ashman, Cristiana Diaconu, Junhyuck Kim, Lakee Sivaraya, Stratis Markou, James Re- queima, Wessel P Bruinsma, and Richard E. Turner. Translation equivariant transformer neural processes. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 1924-1944, PMLR, 21-27 Jul 2024a. URL htttps:/pproceeee
mlr .ww.pppesss.... http//2233a/aaaaaa ashman2 4am atmm..
Matthew Ashman, Cristiana Diaconu. Eric Langezaal, Adrian Weller, and Richard Turner. Grid- ded transformer neural processes for large unstructured spatio-temporal data. arXiv preprint arXiv:2410.0673// 2024b.
Matthew Ashman, Cristiana Diaconu, Adrian Weller, and Richard E. Tumne. In-context in-context learning with transformer neural processes. In Proceedings of the 6th Symposium on Advances in Approximate Bayesian Inference, volume 253 of Proceedings of Machine Learning Research, pages 1-29. PMLR, 21 Jul 2024c. URL https: httpp/poooeeeeggeeeemee/ee mlr. www.srressss... ashman2 2 html.
Christopher M. Bishop. Pattern Recognition and Machine Learning (Information Science and Statistics). Springer, edition, 2007. ISBN 0387310738. URL http: /www.
amazon http://www.o...oee..raaaaaaaaaaa
X0D000a0BBSTSuusttttnaatrrTrrrrn 3Dws 3gueeeeeelliikcccccooooee///s3e3 1bGreettieesSS1833033333
659534
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. Weight uncertainty in neural network. In Proceedings of the 32nd International Conference Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1613-1622, Lille, France, 07-09 Jul 2015. PMLR. htths::s: httttt::://pcccedd.nno /ttpp//ecceeengssee/ee mll. ...r.p2.. vttress//3//buuudel115 html.
Wessel Bruinsma, Stratis Markou, James Requeima, Andrew Y. K. Foong, Tom Andersson, Anna Vaughan, Anthony Buonomo, Scott Hosking, and Richard E Turner. Autoregressive conditional neural processes. In The Eleventh International Conference on Leurning Representations, 2023.
URL https:ttp///c /wpp/revviveeeeeeeeeeeeeeee.epee htt.httsss///peeeeeee net for/oorruutt id=OAsXFPafTHh.
T. Bui. Biases in variational Bayesian neural networks, Jan 2021.
10

---

Published as a conference paper at ICLR 2026
Paul Edmund Chang, Nasrulloh Ratu Bagus Satrio Loka, Daolang Huang, Ulpu Remes, Samuel Kaski, and Luigi Acerbi. Amortized probabilistic conditioning for optimization. simulation and inference. In Proceedings of The 28th International Conference Artificial Intelligence and Statistics, volume 258 of Proceedings of Machine Learning Research, pages 703-711. PMLR, 03-05 May 2025. URL www.t:tppp//t....t. http:::////c/ceeedppeee ng..eeeddig..mmm..... ml press v258// / chang25a. html.
Tianqi Chen, Emily Fox, and Carlos Guestrin. Stochastic gradient hamiltonian monte carlo. In Proceedings of the 31 International Conference on Machine Learning, volume 32 of Proceedings of Machine Learning Research, pages https://ppoceedingg. mar press / vv22 cheni14 html.
Bejing, China, 22-24 Jun 2014. PMLR. URL
https:
Tristan Cinquin, Marvin Pförtner, Vincent Fortuin, Philipp Hennig, and Robert Bamler. FSP- Laplace: Function-space priors for the Laplace approximation in Bayesian deep learning. In Advances Neural Information Processsing Systems, volume 37, pages 13897-13926. Curran Asso- ciates, Inc., URL http:: meuri reurips cc saraaperffllsspppppppr Confeere-ooeeeeeeeeco .dff.
2024 htt//file///// / htt.://eee.777722222202222222222
Anish Dhir, Matthew Ashman, James Requeima, and Mark van der Wilk. A meta-learning ap- proach to Bayesian causal discovery. In The Thirteenth International Conference on Learning Representations, 2025.
Francesco D) Angelo and Vincent Fortuin. Repulsive deep ensembles are Bayesian. In Ad- vances in Neural Information Processing Systems, volume 34, pages 3451-3465. Curran Asso- ciates, Inc., 2021. URL http:ttttt//pceeeeeteetete
neurips. www.eeewwweepppeeeeeeeee
Yann Dubois, Jonathan Gordon, and Andrew YK Foong. Neural process family. http:
//yanndubs. gi
wigggittthu thub. wacchu..oo/eeee /t/NNurall-Procssseeseeee Fami
Waamllly ly/ September 2020.
Sebastian Farquhar, Lewis Smith, and Yarin Gal. Liberty or depth: Deep Bayesian neu- ral nets do not need complex weight posterior approximations. In Advances in Neural Information Processing Systems, volume 33, pages 4346-4357. Curran Associates, Inc., 2020. URL
http:/ee..rip.pccppppppeeee..e files/ pttps//ppeee/////////// file/ hilee/dfee9440 /ttp://fcee6660000313311aaall11.
Papplll--apprreeee påf.
Leo Feng, Hossein Hajimirsadeghi, Yoshua Bengio, and Mohamed Osama Ahmed. Latent bot- tlenecked attentive neural processes. In The Eleventh International Conference on Learning Representations, 2023. URL https:ttp/// /ttp:////peeeeieeeeee wttpiiiwwweweeeeeeeee nnt ftte:/oormmiiiyIItttette
Daniel Flam-Shepherd, James Requeima, and David Duvenaud. Mapping Gaussian process priors to Bayesian neural networks. In NIPS Bayesian deep learning workshop, volume 3, 2017.
Andrew Foong, Wessel Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard Turner. Meta-learning stationary stochastic process prediction with convolutional neural processes. In Advances in Neural Information Processing Systems, volume 33, pages 8284 0244-299... Curran Asso- ciates, Inc., 2020. URL http: 2020 e 5/deee5df03333 F0355coo330522555 6a 35be5 6dbe28 7a -Paper.ddf.
i ngs. neuri ps. cc cc/paper___pee fi lelles/pae lsspppper/
Vincent Fortuin. Priors in Bayesian deep learning: A review. International Statistical Review, 90(3): 563-591, 2022.
Vincent Fortuin, Heiko Strathmann, and Gunnar Rätsch. Meta-learning mean functions for Gaussian processes, 2020. URL https: //atxii aawiiaiiivv.///a org/ abs /ats//19..1.. 08098.
Vincent Fortuin, Adrià Garriga-Alonso Sebastian W. Ober, Florian Wenzel, Gunnar Ratsch, Richard Turner, Mark der Wilk, and Laurence Aitchison. Bayesian neural network priors revisited. In International Conference on Learning Representations, 2022. URL https: //tp///opp//iieeeeeeeeee F JqJyqRJy.
Yarin Gal and Zoubin Ghahramani. Dropout Bayesian approximation: Representing model uncertainty in deep learning. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1050-1059, New York, New York, USA, 20-22 Jun 2016. PMLR. URL https:/ http:////occeeeeessseeee mlr press/ v4 //8/g8 // html.
11

---

Published as a conference paper at ICLR 2026
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Damilo Rezende, and S. M. Ali Eslami. Conditional neural processes. In Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pages 1704-1713. PMIR, 10-15 Jul 2018a. URL https:
ipproceedings ml p.reesppp//eee aatpasavvvv//aaaaa 8a..88.... html.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Tch. Neural processes, 2018b. URL https://aaat attp///aarx.oooo/b/////// 1807. 01622.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turner. Meta- learning probabilistic inference for prediction. In International Conference Learning Represen- fations, 2019. URL www.ttttt/////e.m //pp/////ee/eeeeee/////eeeeeeeee net/ forum f 2id==kkstttc..F...
Alex Graves. Practical variational inference for neural networks. Advances in Neu- ral Information Processing Systems, volume 24. Curran Associates, Inc., 2011. URL /it.es/pppee///////. cc/paper http://ppp...cpppppe.eeeeeeepppp Rebbccheeedd.1eeeebfbbbbbaaa
Jose Miguel Hermandez-Lobau and Ryan Adams. Probabilistic backpropagation for scalable learning of Bayesian neural networks. In Proceedings of the 32nd International Conference on Ma- chine Learning, volume 37 of Proceedings of Machine Learning Research, pages 1861-1869, Lille, France, 07-09 Jul 2015, PMLR. URL https: html.
BRLLttsss////oeeedi/ http:///ccceeeeeeeeeeeeeee/e.sss mmr. mrt.ppress/v7/
hernande z- -ooanteez-oobtoelle
Geoffrey Hinton and Drew van Camp. Keeping the neural networks simple by minimizing the description length of the weights. In Proceedings of the Sixth Annual Conference on Computational Learning Thewry, COLT '93, page 5-13, New York, NY, USA, 1993, Association for Computing Machinery. ISBN 0897916115. doi: 10. 00.1145/6633044166306 URL tttp:://doo..oom 1145/168 168306.
Matthew Hoffman and David Blei. Stochastic Structured Variational Inference. In Proceedings of the Eighteenth International Conference on Artificial Intelligence and Statistics, volume 38 of Proceedings of Machine Learning Research, pages 361-369, San Diego, California, USA, 09-12 vv88 /o foonfman11 5. html.
May 2015. PMLR. URL Uttpt::: pttettss:/prrceeedhiee htppp/occcdigggoeo ngs ml /
Alexander Immer, Maciej Korzepa, and Matthias Bauer. Improving predictions Bayesian neural nets via local lincarization. In Proceedings of The 24th International Conference on Artificial Intelligence and Statisties, volume 130 of Proceedings of Machine Learning Research, pages 703-711. PMLR, 13-15 Apr 2021. URL https:://// /tapp:://prooee ppoceed ngs. ml press/v330/ immer html.
Theofanis Karaletsos and Thang Bui. Hierarchical gaussian process priors for bayesian neural network weights. Advances in Neural Information Processing Systems. 33:17141-17152, 2020.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnllo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Teh. Attentive neural processes. In International Conference on Learning Representations, 01 2019. doi: 10.48550/arXiv.19011 1901.0057
Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In ICLR, 2014, URL html #K #KingmaW13 3.
http:/ dblpp.ppp.//////// uni trier. de/db/
David Krueger. Chin-Wei Huang, Riashat Islam, Ryan Turner, Alexandre Lacoste, and Aaron Courville. Bayesian hypernetworks, 2018. URL htt.https:/// http://:////xxioooo//aa/////////
www./o/////aaaeaeooeeoo.oeeeee/e wwtpe///////.////////////tttettt 04759.
Richard Kurle, Alexej Klushyn, and Ralf Herbrich. BALI: Learning neural networks via Bayesian layerwise inference, 2024. URL httph::t/a/aate wttp:////abb//1...../.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scal- able predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL http:eeeeddinn....eie..eee.e.eee neurips. cc/ htte//aaee__iesepppee///// haaper_flees/pppereeee 9 9ef2ed447fd2cc1 (tt.:eree2220004ffaafaaaooaa 3awpssaeccooppppp..pppp.
12

---

Published as a conference paper at ICLR 2026
Tuan Anh Le, Hyunjik Kim, Marta Gamelo, Dan Rosenbaum, Jonathan Schwarz, and Yee Whye Teh. Empirical evaluation of neural process objectives. In Bayesian Deep Learning workshop, Neural Information Processing Systems (NeurIPS), 2018.
Y. LeCun, Boser, S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. Backpropagation applied to handwritten zip code recognition. Neural Computation, 1(4):541-551, 1989. doi: 10. 162/neco. 1989. 1.89.1.4.541.
Juho Lee, Yoonho Lee, Jungtack Kim, Eunho Yang. Sung Ju Hwang, and Yee Whye Teh. Bootstrapping neural processes. In Advances in Neural Information Process- ing Systems, volume 33, pages 6606-6615. Curran Associates, Inc., 2020. URL /Ltes//ppppr/2222//LLL
https://proceeding hewpreeeeennnnnneeeeeeeeeeeeeeee 2at.99999555..a....a0aaaa22535
neurips cw/paper http://ae.ee.....a..aa..eeaa///a
Qiang Liu and Dilin Wang- Stein variational gradient descent: A general purpose Bayesian inference algorithm. In Advances in Neural Information Processing Systems, volume 29, Curran Asso- ciates, Inc., 2016. URL httpp:t:/tptteeeeeeeeeeeeee http://poooeddddoeooooooo neuri ps cc cccpappr_ffi fi filefiies/eeeeee selles/ppper www.eedddapppppp.......
Christos Louizos and Max Welling. Structured and efficient variational deep learning with matrix Gaussian posteriors. In Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 1708-1716, New York, New York, USA, 20-22 Jun 2016. PMLR. URL https: htttttts//ppcceeeeeeeeee ppoceedingsss ml mlr.r... www.prressvv.. louizos16 html.
Christos Louizos and Max Welling. Multiplicative normalizing flows for variational Bayesian neural networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 2218-2227, PMLR, 06-11 Aug 2017. URL https: htttp::/pproeeeinngg
mll. press ptpress/v0//ooee www//ouiiiiie htm.l...
David JC MacKay. A practical Bayesian framework for backpropagation networks. Neural computa- tion, 4(3):448 472, 1992.
David JC MacKay. Introduction Gaussian processes. NATO ASI series F computer and systems sciences, 168: 133-166, 1998.
Wesley J Maddox, Pavel Izmailov, Timur Garipov, Dmitry P Vetrov, and Andrew Gor- don Wilson. A simple baseline for Bayesian uncertainty deep learning. In Ad- vances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL
/ttlees/pppe//211// pdf.
Alexandre Maraval, Matthieu Zimmer, Antoine Grosnit, and Haitham Bou Ammar. End-to- end meta-Bayesian optimisation with transformer neural processes. In Advances in Neural Information Processing Systems, volume 36, pages 11246-11260. Curran Associates, Inc., 2023. URL https: FHle/2561721d00/ http://pooceeenn..ecee/e// h neurips http//ppr__fieeee filles/paper//022 Sateeeffcc-apeer-eee Conper--cnfeeeeceee.oo -df..
Alexander G de G Matthews, Jiri Hron, Mark Rowland, Richard E Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks, International Conference on Learning Representations, 2018
J. Muñoz Sabater, E. Dutra, A. Agustí-Panareda, C. Albergel, G. Arduini, G. Balsamo, S. Boussetta, M. Choulga, S. Harrigan, H. Hersbach, B. Martens, D. G. Miralles, M. Piles, N. J. Rodríguez- Fernández, E. Zsoter, C. Buontempo, and J.-N. Thépaut. Era5-land: state-of-the-art global reanalysis dataset for land applications. Earth System Science Data, 13(9):434943383, 2021. doi: 13-4499---3-44449999 13/ 4349/349/// /0021/
URL https:://///.... essd. sd. copernicus httpp::/ess....peeeiiiccm wwt.eeoogg//aaaeeeeeeeeeeeee
Samuel Müller, Arik Reuter, Noah Hollmann, David Rügamer, and Frank Hutter. Position: The future of bayesian prediction is frite- In Forty-second International Conference on Machine Learning Position Paper Track, 2025.
13

---

Published as a conference paper at ICLR 2026
Warwick Nash, Tracy Sellers, Simon Talbot, Andrew Cawthorn, and Wes Ford. Abalone. UCI Machine Learning Repository, 1994. DOL: https//bbiingg/022245///CCC
Radford Neal. Bayesian learning via stochastic dynamics. In Advances in Neu-
ral Information Processing Systems, volume 5. Morgan-Kaufmann, 1992. URL wwtppper_ppppppppeeeeeeeeeeeeeee fi httpae///////e//////11111 pdf. pdf.
Radford M Neal. Bayesian Learning for Neural Networks. PhD thesis, University of Toronto, 1995.
Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 16569-16594, PMLR, 17-23 Jul 2022. URL https::///..... /ttp//ppeceen.....eeepppoo mlr. prwpspessssse........ /v1
62 http//11///////// hwwp/n2uyyen2222. html.
Sebastian W Ober and Laurence Aitchison. Global inducing point variational posteriors for bayesian neural networks and deep Gaussian processes. In Proceedings of the 38th International Confer- ence on Machine Learning, volume 139 of Proceedings of Machine Learning Research, pages 8248 8259. PMLR, 18-24 Jul 2021. URL https: /ttp://e//ccceeeesse//eem / mwr. wwl..prees//vv3.
ober2la. html.
Young-Jin Park and Han-Lim Choi. A neural process approach for probabilistic reconstruction of no-data gaps in lunar digital elevation maps. Aerospace Science and Technology, 113:106672, 04 2021. doi: 20.10166..st.2221.1000
Young-Jin Park, Cesar Almecija, Apoorva Sharma, and Navid Azizan. Uncertainty-aware meta- learning with analytically tractable posterior. In The 29th International Conference Arti- ficial Intelligence and Statistics, 2026. URL https: http://pppereeeeeeeeeeeeee net. forum? idd EtAZdcNSII.
Massimiliano Patacchiola, Jack Tumnrr Elliot J. Crowley, Michael O' Boyle, and Amos J Storkey. Bayesian meta-learning for the few-shot setting via deep kernels. In Advances in Neural Information Processing Systems, volume 33, pages 16108-16118. Curran Asso- nwwrips cc htw.::occcccic.aa7aaaaeaarr...d
ciates, Inc., 2020. URL 2020
/paper/
Franziska Paul, Ya'ara ara Arkin, Amir Giladi, Diego Adhemar Jaitin, Ephraim Kenigsberg, Hadas Keren-Shaul, Deborah Winter, David Lara-Astiaso, Meital Gury, Assaf Weiner, Eyal David, Nadav Cohen, Felicia Kathrine Bratt Lauridsen, Simon Haas, Andreas Schlitzer, Alexander Mildner, Florent Ginhoux, Steffen Jung, Andreas Trumpp. Bo Torben Porse, Amos Tanay, and Ido Amit. Transcriptional heterogeneity and lineage commitment in mycloid progenitors. Cell, 163(7): 1663-1677, 2015. ISSN 0092-8674. doi: http:://boooor//l.0.016/.......d // //www. www.sceieenedecccooo rect. com/ wwt.sccinccceeeececceeeee http:///cc..//...e..///////////0
URL https:
K. B. Petersen and M. S. Pedersen. The matrix cookbook, October 2008. URL http::tttt/w/tt
imm. h
php?3274 Version 20081110.
Alexander Pondaven, Märt Bakler, Donghu Guo, Hamzah Hashim, Martin Ignatov, and Harrison Zhu. Convolutional neural processes for inpainting satellite images, 2022. URL https: //arxiv./
htg/abb/22055 12407.
Carl Edward Rasmussen and Christopher K. Williams, Gaussian Processes for Machine Learning. The MIT Press, 2006.
Sachin Ravi and Alex Beatson. Amortized bayesian meta-learning. International Confer- ence Learning Representations, 2019. URL https: /ttp:////peeeiiieeeeeeeeeeee htttttt::///peeeeerww rkgpy3C5tX.
wew.tt/wweeeeeeeee.eee......eeee wtt.eeeww.eteeetmteeeeeemm forumiid-
Arik Reuter, Tim G. J. Rudner, Vincent Fortuin, and David Rügamer. Can transformers learn full Bayesian inference context? In Forty-second International Conference on Machine Learning, 2025. URL https::/opeerereeeeeeeee /pwpnroeeeiieeeeeooooeeeee net forum?i d i hKbc.
14

---

Published as a conference paper at ICLR 2026
Hippolyt Ritter, Aleksandar Botev, and David Barber. A scalable laplace approximation for neural networks. In International Conference on Learning Representations, 2018. URL https:// openreview net/ forum21d=Skdvv22x3.
Tommy Rochussen and Vincent Fortuin. Sparse Gaussian neural processes. In Proceedings of the 7th Symposium on Advances in Approximate Bayesian Inference, volume 289 Pro- ceedings of Machine Learning Research, pages 194-219. PMLR, 29 Apr 2025. URL https:
//proceedings mlr presss / v229/222200 // rochussen25a html.
Peter Rossky, Jimmie D Doll, and Harold L Friedman. Brownian dynamics as smart monte carlo simulation. The Journal of Chemical Physics, 69(10))468-4463,, 1978.
Jonas Rothfuss, Dominique Heyn, Andreas Krause, et al. Meta-leamnin reliable priors in the function space. Advances in Neural Information Processing Systems, 34:280-293, 2021.
Ravid Shwartz- Ziv, Micah Goldblum, Hossein Souri, Sanyam Kapoor, Chen Zhu, Yann Le- Cun, and Andrew Gordon Wilson. Pre-train your loss: Easy Bayesian transfer learning with informative priors. In Advances in Neural Information Processing Systems, 2022. URL wwt. httt//foummiiixcnn......
httts://openrevttt
Gautam Singh, Jaesik Yoon, Youngsung Son, and Sungjin Ahn. Sequential neural pro- cesses. In Advances in Neural Information Processing Systems, volume 32. Curran Asso- ciates, Inc., 2019. URL https: http::::///pccceceg...s..s httptpsss//pppoeeeeeeeaao www.onngg...euiii neurips wwtpapperaaiiiieeeeeeeeeeeeeeeee filesspaaee. Pap.............p.......
Emanuel Sommer, Lisa Wimmer, Theodore Papamarkou, Ludwig Bothmann, Bernd Bischl, and David Rügamer. Connecting the dots: coneeceennst the key to feasible sample- based inference in Bayesian neural networks? In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 15998-460111 PMLR, 21-27 Jul 2024. URL https: htte::/prrooceeee procedd ngs ml
www.ml...pr.......eeeeemmmmmmmme w.l.rspres//ee peress/v235. v235 wwwmer html.
Shengyang Sun, Guodong Zhang, Jiaxin Shi, and Roger Grosse. Functional variational bayesian neural networks. International Conference on Learning Representations, 2019.
Terence Tao. An introduction to measure theory, volume 126. American Mathematical Soc., 2011
Ashish Vaswani, Noam Shazeer, Niki Parmar. Jakob Uszkoreit, Llion Jones, Aidan N Gomez, E ukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Ad- vances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. URL https: http://///oe//eeee////eeee neurips cc/paper_ppee.eeeeeeeeeeeeee wwslllsseep///ee tess/papr/22117 httee/3f5ee224547deeeee wtw.:/eeeceadodc1ecaccaaaeea Saapaaaaaaaaaaaaappppaea Paper.pdf.
Valentin Villecroze, Yixin Wang, and Gabriel Loaiza-Ganem.
Last layer empirical bayes. In I Can't Believe It's Not Better: Challenges in Applied Deep Learning, 2025. URL https:
//apenreview.oo net http://wwww...tt/tommdeeddd----e forummid-j533s1111 62RA.
62RA.
Michael Volpp, Fabian Flürenbrock, Lukas Grossberger, Christian Daniel, and Gerhard Neumann. Bayesian context aggregation for neural processes. In International Conference Learning Representations, 2021. URL https: http:///ppeeeeeeeeeeee fnwee//oooumiieeeeeeeeeeeeee ?ineruunmmiuuu aehF a.
Michael Volpp, Philipp Dahlinger, Philipp Becker, Christian Daniel, and Gerhard Neumann. Accurate Bayesian meta-learning by accurate task posterior inference. In The Eleventh International Conference on Learning Representations, 2023.
Max Welling and Yee Whye Tch. Bayesian learning via stochastic gradient langevin dynamics. In Proceedings of the 28th International Conference International Conference Machine Learning, ICML'11, 11, page 681-688, Madison, WI, USA, 2011. Omnipress. ISBN 9781450306195.
Alexander Wolf, Philipp Angerer, and Fabian J Theis. Scanpy: large-scale single-cell gene expression data analysis. Genome biology, 19(1):15, 2018.
Greg Yang. Scaling limits of wide neural networks with weight sharing: Gaussian process behavior, gradient independence, and neural tangent kernel derivation. arXiv preprint (tt2:1900.04760 2019.
15

---

Published as a conference paper at ICLR 2026
A OBJECTIVE FUNCTIONS AND TRAINING
In this section we justify our choice of objective function, we discuss practical implementation of it, and we discuss alternatives. Throughout, we q to denote distributions that depend the approximate posterior q( (WI.) and pa to denote distributions that depend on the parameters of the prior without depending -W).).
A.1 PP-AVI
We begin by repeating Proposition for the reader's benefit.
Proposition 1. For E + oo, maximisation of (PP.Aw(EE) directly targets the three desiderata.
order to prove Proposition we first provide formal definitions of the three desiderata. Throughout, we assume true underlying data-generating process 'D ~ p(D).
Definition 1 (accurate approximate posteriors). A probabilistic mela-learner produces accurate approximate posteriors if the task-averaged KI. divergence between approximate and true posteriors
E B)D)) KL (L(g(W(D.) Po P. D.)
(11)
is small.
Definition 2 (faithful prior). A probabilistic meta-learner has prior that faithfully encodes the data-generating process the lask-averaged KL divergence between the true generative process p(Y. and the model's 's marginal likelihood (Y Xe)..
Ex))) KL(p(Y.(X.) (Y.(x.)
(12)
small.
Definition 3 (high quality predictions.). A probabilistic meta-learner produces high quality pre- dictions if the task-averaged KL divergence between the true conditional generative process (D X.) and the model's ' posterior predictive ql Y (D,
ExpDe.xe) (L. (D, (.. ]
(13)
is small.
Proof. Recall the definition of (E):
(P) log a (X() + CELBO (D!
(14)
16

---

Published as a conference paper at ICLR 2026
Taking the limit of infinite datasets, proceed as follows
lim EPP.AVI(3) P)D) log (Y.)p, X,) LELRO (D.) B 000
(15)
PPCD) [oo q PP: X,) (Yooloooooo
-KL(g pv (W]D:)]
(16)
Ep(Y.p..x.) [log (( (Y|,,, x.)]
+Bp(x.)
E [P(Y.(X.) [oggo ( xXx))l]
- -E
KL (D(W)Dd) || pv (W[D:]]
(17)
log (YY (Y) (.)
-P(D..)))
BMYIID..X.)
Xi)
pv X Y. (x.)
+Bp(X.)
EP EXY.)
log
Ep(pa) KL (O WI))) || popppppo (WID.)]
(18)
HH(Y, X.)
E Ep(D..x.)
KL(p(YY/Pe..1 gl,,
coust.
(
Bx(x.)
KL. (I.(p(Y.(X.)) (Y.
const.
KL [g(W
|| pv (W[P.]]
(19)
where [-I(-I-) denotes the Shannon conditional entropy. Since the entropy terms are constant with respect to the variational and model parameters (O, } we reach
arg max lim AV) = arg min (0.0) |l|+00
E BP(D....) KL X,) (Y X.]]
(0,0)
+ [x()))
KL[P(Y P (Y.(.))
+ Ep()))
KL (g(g)DD)))
|| (WID.]]
(20)
Practical Details. Unfortunately, both terms in LPP-AVI are analytically intractable. As first step, we decompose the ELBO into the usual reconstruction error / complexity penalisation form:
LVP.AvI(3) logg (D()) x(o) E (WID()) [Logr (Y!",,Xy)
KL (WID!")
|| pv (w)]
(21)
We deal with these three terms in order of increasing difficulty. The final term becomes tractable by decomposing it into the sum of layerwise KI. divergences
KL a (WID!)) pw) = Σ KL l (w'l D(.)) || Per (w")]
(22)
where each term in the sum is available in closed form as the KL divergence between two multivariate Gaussians.
17

---

Published as a conference paper at ICLR 2026
The middle term, the (context set) expected log- ooglikelihood, is unbiasedly estimable via Monte Carlo integration with a finite number of samples K
K log P
E a (WiDD! logr (
( W x
~
Y W(k), x()
(23)
K
where W(() W )! and low-variance gradient estimates available via application of the reparameterisation trick (Kingma and Welling, 2014).
The posterior predictive term, which can equivalently be referred to the (target set) log expected likelihood, is defined as
D (D!.xx) log / (x (
w. W.X!") (WID!") dw.
log q
(24)
Again turning to Monte Carlo integration with K samples, we estimate this term via
logg q (x())D!)) x
Σ xx (x()) (())
= log K
(25)
{l0gy (r w(()) x K
= LogSumExp
log K
(26)
b=11)
where W(k) q (WID!") and we use the log-sum-exp trick to maintain stable computations log-space, Unfortunately, this Monte Carlo estimate is biased and so we cannot get away with single-sample estimates as we might with standard variational inference. In our experiments we used between 8 and 32 samples.
A.2 STANDARD AVI/ META EMPIRICAL BAYES
Perhaps the most obvious alternative objective function to consider would be the standard AVI objective given by the task-averaged ELBO:
CELBO (D!!).
CAvi (E)
(27)
In the infinite-dataset limit, this objective function corresponds minimising two of the three desired expected KL divergences; E,tx.) KL (L(p(YY((..) pv (Y,|x.)]]. and
EptD.) (L(T())).) |l pv p (WID.)) Supposing have globally optimised this objective function for an infinitely flexible model (not necessarily a BNNP) under infinite datasets, the trained model would have perfect approximate posteriors and prior that models the true (context) gett-generating process exactly. Perhaps the missing term would then be minimised for free- we should expect extremely high quality posterior predictives from this resulting model. While this may be true (proper analysis left to future work), in practice we found LAVI to be significantly harder to globally optimise than LPP-AVI- When training our BNNPs, we initialised the parameters the prior to those of the standard isotropic Gaussian prior. For some tasks (particularly the heaviside and sawtooth function regression problems) seems that this was a particularly adversarial initialisation in that maximisation of Cxvi seemed not to be able "break free" of the predictive behaviour induced by the standard BNN prior. So, even the extra expected KI. term from the PP-AVI objective is not strictly necessary, it seems simplify the lass-landscape by suppressing the non-globally optimal modes via penalising miscalibrated posterior predictives.
Since the ELBO is a lower-bound to the log marginal likelihood LELRO (De) log Y Xx). under infinite datasets we have that the AVI objective is lower bound to the expected log marginal likelihood across tasks:
lim (Avi(E) < Ep(x,) [log Y (oog((YBli [X.)]
(28)
This then corresponds to meta-level type-Il marginal likelihood scheme, or, meta-level empirical Bayes. Note that this is the objective function used in Rochussen and Fortuin (2025) and Ashman al. (2023).
18

---

Published as a conference paper at ICLR 2026
A.3 NP MAXIMUM LIKELIHOOD/ / POSTERIOR PREDICTIVE MAXIMISATION / ML-PIP
Neural processes tend to be used as probabilistic predictors more than anything else (such generative models), so the only distribution users really care about is the posterior predictive q( Y [D.. X²). Note that while this distribution is referred to the posterior predictive in the context of varialee NPs, for the conditional family of NPs it is referred to as the predictive likelihood. The obvious training scheme would then be to maximise the (log) posterior predictive likelihood of the target set over many tasks. This is the de-facto approach in conditional family NPs, and Gordon et al. (2019) demonstrated that it is sufficient for performing amortised variational inference in latent-variable meta-learners too. Gordon et al. refer to this scheme as meta- meta-learning probabilistic inference for prediction (ML-PIP). We refer to this objective as the neural process maximum likelihood (NPML) objective, and it takes the form
CNPML (=)
log q( q Y x
(29)
In the infinite-dataset limit, maximising LNPMI. corresponds to minimising the expected KL. term that is missing from LAVI- In other words, we have that
(PP-AvI(E) = LNPMLL (E) + LAv.(E).
(30)
While NPML has been demonstrated work well in NPs5, with the BNNP found it would lead either to solutions that modelled the data-generating process very badly, or to high numerical instability causing training runs to fail. As with LAVI. we suspect this is because of loss-landscape multimodality and difficulty ensuring global optimisation. In particular, we suspect that the globally optimal modes in LAvi and LNPML correspond to each other but that the secondary modes in each loss-landscape do not, meaning their sum is more straightforward to globally optimise.
A.4 NEURAL PROCESS VARIATIONAL INFERENCE
Another way to train latent-variable NPs is through a more conventional variational approach, where the objective function is given by a lower bound to the conditional marginal likelihood pv (D,, averaged over tasks. We refer to this as the neural process variational inference (NPVI) objective, and it defined as
E
(E) =
Σ E (w)))))
llog (Y! (, [ x
KL
(W)) pv (WID!")]
(31)
While NPVI is sensible in the infiiite-datsett limit, being equivalent to minimising
Ep(.....) x.) KL [D(YYY X,) || pv (Y, (Y, [D,] Ep() KL [L(g(W1)) pu (WID)]
(32)
it is intractable due to the presence of the true posterior pv (D)).) While this is typically circum- vented by approximating the true posterior with the approximate posterior (W (D.)). NPVI only targets Eq. (32) if the approximate posterior is exact. Under a poor approximate posterior, such as at the beginning of training, the lass-landscape is then quite different to what it should be and unclear how close the implemented version of LNPVI ever becomes to the ideal one. Indeed, the NPVI approach is known to yield suboptimal predictive performance (Le et al., 2018). Furthermore, the KL term involves computing the two approximate posteriors q( (W[D) and q( (P.). Since the computational bottleneck associated with the BNNP is in computing the approximate posterior, the double approximate posterior computation renders CNPVI even more inappropriate for the BNNP, and so we did not attempt to use it at all. Note that LXPVI was the objective function used in the original (latent-variabll neural process (Garnelo al., 2018b).
'Nte that, for latent-variable type NPs, the latent variable is denoted by rather than W.
19

---

Published as a conference paper at ICLR 2026
A.5 TELL-AVI
Motivated to find an objective function that can be ubiasedly estimated and which has similar infinite- dataset behaviour to LPP.AVI, we introduce the tagge--enpeied-loo-iikiiiiio (TELL) AVI objective. We define it as
AVELL =
Σ ( wi)D
[LogP (r!²) W. [.. + LELBO (D())
(33)
where note the only difference between LPR-AVI and LTELL-AVI is in the ordering of the log and expectation E *( (W)))!) in the left-hand term'. The right-hand term is estimated exactly as in
LPP-AVI. but the left-hand can be unbiasedly estimated by its Monte Carlo estimate from K samples, R E log (1) [W(s), | xx!)) for W (W enabling the use of very few samples
for decreased computational cost.
Secking to rewrite the expected log-likelihood term in terms of the log posterior predictive (log expected likelihood), we have
pYY/W, X.) ( D.) YY, X,)
(.(w)).) plog (Y, (Y X,)] (II)) log
(34)
( De)g (Y (D,, X.)
1 (Y, W Drr X,)
(WDD) [log (Y. X,)] + BO(WD)))
log 4 (W(D) a (Y, (De, Xt)
(35)
9 9( X, Y, D.) log
=logg (Y, X,) E (WI.))
(36)
(W))))
logg Pr, X.) KL (o((DD)) ((W())),
(37)
which gives us the following relationship between CTELL-NT and LPP-AVI:
lim LTELL-AVI = lim LPP-AVI -+00
Ep(D)
KL (L(a(W(D.) (W[D))
(38)
Though this expected KL term is somewhat sensible when interpreted as encouraging approximate pos- teriors from partial datasets to be similar their full-data counterparts, the reverse interpretation (that it encourages full-data approximate posteriors to be only as good as their partial-date counterparts) highlights dubiousness. Furthermore, the extra term would serve as distractor from the more im-
portant "posteriors KL term" that LPr-Avi already involves; (wIp)) KL [L[gg( (D.)) pv P (W[D.)]
and it is unclear what behaviour the sum of the two KL terms involving (W (.)) leads to. In prelim- inary experiments we found that LTELL-vI led to reasonable performance terms of predictions and the resulting learned prior, but it was never quite as good as LPP-AVI-
A.6 TRAINING
Algorithm BNNP training algorithm.
Require:
meta-dataset number of training steps T, number of Monte Carlo samples
for t= 1,..., do
Randomly select & from =
> select minibatch of tasks initialise batch objective
Lracc <- 0.0
for dataset D in minibatch do
compute layerwise posteriors, obtain is samples from each
c.g. by using algorithm b.c.g. using practical details from section A.
compute LvP.vv (D) using it samples
Lrauch +- Lruch
Lre-xv((D)
end for
Use Loaca to update θ and V end for
c.g. using Adam, gradient descent,
i.c. Eep AVI might have equivalently heen called the arrge-log-eppocca--lkiiiioooo (TLEL) AVT objective.
20

---

Published as a conference paper at ICLR 2026
B INFERENCE WITH ALTERNATIVE PRIORS
Fully factorised. The simplest form of prior is fully factorised one Pa; (W)) o'a,")
I1 I N N Waaaiiaa "'a ) Inference with this prior is performed similarly to the unit-
wise factorised prior, except that the unitwise covariance matrix is given by Σ = diag (a') and can therefore be inverted by taking the reciprocal of the diagonal elements, taking (dd-1) time rather than ( 9) time.
Layerwise matrix- Gaarss-G While seemingly obvious prior to consider is the matrix-Gaussian over layerwise weight matrices (Kurle al., 2024; Louizos and Welling, 2016; Ritter al., 2018), this prior turns out not to fit nicely into our amortisation framework. The pseudo-observation noise variances in each layer would have become part of the prior rather than the pseudo likelihoods, meaning our inference networks would lose their ability to up- or down- down-weight pseudo -ouudoobseevations by predicting their corresponding noise level. In other words, using matrix- matrix-Gaussin Gaussian priors would force us to destroy the Bayesian context aggregation behaviour of our method.
Layerwise full-rank Gaussian. A richer prior than the unitwise factorised one is layerwise full- rank Gaussian defined over vee (W)) To lighten the notation, we define w := vee (W') The prior then Por (W") p(w") N (wo, je', E.) In order apply the standard (single-output) Bayesian linear regression results, we need to augment the data matrices X Y' to some matrix-vector pair X y where y' := vec(Y'), that the likelihood model can be written as y X w It turns that the "vec trick" (Petersen and Pedersen, 2008) the Kronecker product gives us what we need:
Y = =
(39)
(40)
y' vec (6(x'-1)
(41)
vec (6(x -
(42)
09 (( wi
(43)
giving that x ; I (x(X Note that x RddiNx The posterior is therefore
p (w'1x' x') -N (w': (w':
(44)
with mean vector and covariance matrix given by
s s'' -'='' x
X I A'x 1-1 m' = s' (I x N'y)
(45)
(46)
where A' NdixNd is diagonal precision matrix formed by stacking the A matrices along the leading diagonal an Nd, x Nd, zeroes matrix. More specificcll. A diag (concat (1x))=)))
where the n-th element of A E R.N is given by بي To avoid direct matrix multiplications with
the NeN x did:11 matrix x Eq. (45) and Eq. (46) can re-written as
s'1 1
s
Σ' _'1 T
(47)
m' = s' (xr (x"
n² +b)
(48)
where Φ is constructed by stacking the d:- x di-1 matrices (6(X ( ) along the leading diagonal of a didi-1 x didr-1 zeroes matrix, and Φ is given by vec (x(X)-11 x where
vh.
the elements of Y are given by Al.d == at
21

---

Published as a conference paper at ICLR 2026
Note that inference with this type of prior is significantly expensive than with the unitwise or weightwise factorised priors. This is because the computation is dominated by the inversion of did:-1 x djd-- covariance matrices, requiring O (df (ddd)_1) operations. For network with uniform hidden layer width, the cost of inference therefore scales sextically (d") with network width.
Full-rank Gaussian. The richest prior consider is a full-rank Gaussian defined over all network weights w := concat (Wv) i... p(W1:)) = p(w) = N (wi μu, Σ). Inference then performed in the same way with the layerwise full-rank Gaussian prior, save that the layerwise factors {Po. ( (W') are replaced with the layerwise conditionals (p(W) W(]-))) Each
layerwise conditional takes the form
p( (W
= / plw/ - = N (w<: (e EdPr)
(49)
where e and p denote the indices of the current and previous layer weights respectively, and the conditional mean and covariance are given by
μ c Me Eep D-1 pp
(w) up))
(50)
Ed E.. Erp
D pp
Epe
(51)
where w denotes the (vectorised) weights of layer a, pa denotes the subvector of μ indexed by a. and similarly Eas denotes the submatrix of E indexed by a and b.
22

---

Published as a conference paper at ICLR 2026
C MINIBATCHING
Our minibatched posterior sampling algorithm for scalable inference in the BNNP is detailed in Algorithm The algorithm describes the usual full-batch posterior sampling procedure if the number of minibatches is set to B = 1. We use part to the propagation of imputs X forward to the 1-th layer of the network using weights W and compute e posterior to refer to computation of Eq. (5).
(X, W to refer
Algorithm 2 A minibatched BNNP posterior sampling algorithm.
Require: inference networks {ge, 1 and priors (W))) for 1,... T. do
for = 1 1,..., do Obtain D>
if then X X
else
x + partiall.forwarddpsss... W
end if
if 1
<Wi 1:8 Di) < compute.posterior (x) D1,90, (Wi))
else
( 1:8 Dia) + compute wwwpputooppottrroorroooo x Do, o( (W W11:1 1.B 1)a-11)
end if Discard D,
end for Sample Wi. (W Wll-1 end for return (Wi 1=
LB DLD)
The time complexity associated with a forward pass through the (-th layer is O(ndi-1 dd), and the corresponding space complexity is ( The superlinear scaling with the number of input neurons di-1 arises from inverting di-1 x di-1 matrices. The purpose of our minibatching algorithm to reduce the linear scaling the space complexity with context size to constant, i.e. convert the space complexity to ((b]di-1 2dd), where [6] the batch size.
While this is greatly beneficial at prediction time, during training it is necessary to store in memory the gradient information corresponding all batches. This reverts the memory complexity to O(n,d-1dd)) dd) in spite of the minibatching procedure. To remedy this, we propose randomly select just one of the minibatches to compute gradients with respect to at every layer (i.c. the same minibatch for all layers), discarding gradient information for the other minibatches. While this leads noisier gradient update steps, reduces the memory complexity back down (ib]d ( as desired. Although we leave detailed analysis of any biases that this might introduce to future work, we emphasise here that we expect this scheme maintain distinct advantage over a naive minibatching in which we simply limit the maximum context set size to the batch size size-our our scheme should preserve the BNNP's ability generalise predictive inference to context set sizes larger than the batch size.
23

---

Published as a conference paper at ICLR 2026
D ONLINE LEARNING VIA LAST-LAYER SEQUENTIAL BAYESIAN INFERENCE
As mentioned in Section 2.4, any form of posterior update due to new data must be applied in full each layer's posterior before computing the next layer's conditional posterior. Unfortunately, this means that sequential Bayesian inference cannot be used naively for online learning in the BNNP. To see why, consider a partioning of task's s dataset into "original" and "update" subsets D = D. U Du. Assume we have already computed the layerwise posteriors for the original data and have since discarded the original data. Without access to the orig- inal data we can only obtain the conditional posteriors {g( Wh:-1, D)} and not what we need, which is WWltt 1= propagating the full collection of data.
DD)) W
This is because conditioning on samples W requires
o.
6,
However, we can approximate what we need by updating just the last layer's posterior while preserving the existing previous layer posterior samples, giving us
o( (WID) wW D)
II (W w .D))
(52)
p <P 1 a (WL W 1--1- D.) II q ( (,))
D.)
(53)
We can interpret this the first inferred from just the original data, while the weights of the prediction head (last layer) are updated light of the new data by sequential Bayesian inference. Although this might seem like tenuous approximation, we find it works well in practice. We found that the approximation deteriorates for increasingly deep architectures this unsurprising given that corresponds to the posteriors of an increasingly small proportion of weights getting updated.
layers being a feature selector whose weights have been
Figure 8: A demo of our online learning algorithm for the BNNP. Orange dots are context points, blue lines are posterior predictive samples, black dots are context points that have lost access to. Each row incorporates more context data. The left-hand column corresponds to a BNNP given access the full context set at each increment, the central column corresponds to our online learning algorithm, and the right-hand column corresponds to the BNNP given only the new context data.
24

---

Published as a conference paper at ICLR 2026
E
THE BAYESIAN NEURAL ATTENTIVE MACHINE
E. ARCHITECTURE
Amortised Linear
K
Amortised Attention
Layer
K
Layer
X!
MHA
Anartined Linear Layer
Anortised
Linear Layer
Q
X-
MHA
Layer
Vi
(a) Amortised attention layer
X,
Amortised Attention Layer
Layer Korz
Layer BMMP
Layer
X]
Amortised Linear Layer
Amortised Attention Block
Anortised Linear
X!-1
Norz
-X
Layer
Amortined Block
BNAM
D.
(b) Amortised atlention block
(c) BNAM
Figure 9: Computational diagrams of the amortised attention layer (a), amortised attention block (b), and BNAM (c). Due the numerous crossing lines in (a), we colour code the context and target input data paths orange and light blue respectively. Arbitrarily many amortised attention blocks can be stacked sequentially in the BNAM; our diagram shows the simplest possible BNAM architecture.
We see in Fig. (()) that amortised inference can performed in an attention layer by using amortised linear layers in place of standard linear layers, where MHA is the usual multi-head dot-product attention mechanism acting on keys K, queries Q, and values V. Similarly, Fig. 9(b) we follow the standard approach (Vaswani et al., 2017) for constructing stackable attention blocks from attention layers, residual connections, layer norms, and 2-layer MLPs, but replacing each of the attention layer and MLP with their amortised counterparts. In Fig. ((c) we show how amortised inference can be performed in transformer by composing amortised linear layers and amortised attention blocks. We note that the resulting model can only used in somewhat unusual way for transformers; map from test inputs X, to predicted test outputs Y, where attention is performed between the test inputs, and where the posterior over the transformer's weights estimated from context set.
E.2 LACK OF CONSISTENCY
As mentioned in the main text, the BNAM does not produce consistent predictive distributions over target outputs. As we shall see, is the attention between test inputs that causes this. For any finite set of target locations X the joint distributions pX.... (Y and px. (Y for m (marginalisation) consistent if
PX (Yr.)))
PX (Yu..) )d Emp )dY.)dYm
(54)
In the case of the BNAM, we are interested in joint distributions of the form
p (Y tasm DD X(y..) (Yi... ts:n. W, Xt..)..) (W
(55)
where the weights posterior pv (W)..) is approximated by the variational posterior q( gW( q( (W''[wl--1.. (D)) that we developed in the paper, and where the likelihood p (Y.... W. X parameterised by transformer Tw with weights W. Plugging these terms in,
25

---

Published as a conference paper at ICLR 2026
verify that the BNAM does not produce consistent joint predictive distributions
dmm-inn J p (YYtn. cD X dY Im-lin
= - p ( biin ( (x (WID.) dwaY, Im+lis
(56)
/ p (Ytti (T (y. (W
(57)
= P (Yi De, Xtnin)
(58)
# p (Y (Des Xtim.)
(59)
and visualise the consequences of this in Fig. 10. By contrast, the BNNP models target outputs as conditionally independent given a weights sample for the MLP fw. This means the BNNP's likelihood decomposes p (Y Ess.. Ifw (X,,
I P fyy.lfw (xt,)) which in turn ensures consistency of the model through the fact that p (Y c. x. =
(Y fm [D X By the Kolmogorov extension theorem (Tao, 2011), consistency of a collection of joint distributions is needed for them to define valid stochastic process, and it is for this reason that the BNAM does not define stochastic process. Note that the other condition required is exchangeability; both the BNNP and BNAM exhibit this through permutation invariance with respect to the context observations and permutation equivariance with respect to the target inputs. While the lack of consistency is generally disadvantage, in settings for which the target inputs are always the same, this behaviour of the BNAM would not matter. An example of such a scenario the common NP task of image completion via pixelwise meta-regression- in this case the target inputs are always the complete set of pixel coordinates.
(a) p (Y, fum De, X
(b) (Y 1mm De. X
Figure 10: Inconsistent predictive distributions BNAM trained on GP-prior generated data. The orange dots are context observations and the wiggly blue lines are predictive samples. X Xt is generated as uniform grid of 100 locations between and 5 via torch. inspace (-5. 0, 5.0, 100). while X generated by appending value of 100.0 Xt, (i.e., 100 and n = 101). Observe that the joint distributions over Y Yi..m are clearly very different. other words, querying just single extra target location drastically change the BNAM's predictions, especially the additional target location is OOD.
26

---

Published as a conference paper at ICLR 2026
F EXPERIMENTAL DETAILS
The codebase can be found at github.com/Sheev 13/meta-bdl.
F.1 How GOOD IS THE BNNP'S APPROXIMATE POSTERIOR?
The BNN used to both generate and model the dataset had two hidden layers each with 20 units (i.e. overall architecture of [1, 20, 20, IJ) and ReLU nonlinearities. The prior was a zero-centered fully-factorised Gaussian with variances equal to the reciprocal of the number of input units for each layer. As described in the main text, the dataset was generated by sampling set of weights from the BNN prior, uniformly sampling N inputs X from (1 4.0) where N itself was sampled from (21, 42), passing these inputs through the MLP parameterised with the sampled weights to obtain 24 corresponding clean outputs, and finally adding Gaussian noise of standard deviation 0.1 to each output to obtain the outputs Y. The dataset is visualised in Fig. 11.
0
2-
Figure 11: The synthetic dataset used in the investigation of approximate posterior quality.
For all approximate inference baselines, the standard deviation of the Gaussian likelihood was fixed to particular value along the x-axes of Fig. 2, i.e. from torch. logspace (-2, 1, 40), and the approximate inference algorithm was run four times using random seeds [21, 42, 69, 420] and the below training parameters. The dataset was the same throughout.
Two BNNPs were trained in this experiment; one as meta-learner and one as traditional learner. Both BNNP's had fixed i.e. their priors were fixed that of the generainng BNN, and both used inference networks with two hidden layers of 50 units. The meta BNNP was trained using meta-dataset of 50, 000 process to the test dataset, save that the number of inputs was sampled from (5, 50). The PP-AVI objective function was used, with the proportion of context points sampled from (0.77, 0.9) with the remaining points being used targets. A (meta-level) batch-size of 5 was used, and 8 samples from the posterior were used in each step for Monte Carlo estimates of the objective function. The optimiser was Adam under Py PoToch defaults but with learning rate that was linearly tempered from 5e-3 to 5e-5. The model was trained for 20, 000 steps, or equivalently (for a batch-size of 5 and 50, 000 datasets) epochs.
Meta BNNP.
cetheetccalll-gnerrtte
datasets with identical generation
Non-meta BNNP and everything else. These models were trained by optimising the standard ELBO objective function that was estimated at each step using 8 samples from the posterior. All models were optimised via Adam under the PyTorch defaults and for 20, 000 steps of (within-task) full-batch training, i.e. for 20, 000 epochs. MFVI, UCVI, and LCVI used learning rates linearly tempered from 5e-3 down to 5e-5, FCVI from 5e-4 down to 5e-5, and GIVI from 1e-2 down 5e-4 (it was much more stable to train, especially compared to FCVT). GIVI used 10 inducing points, with the inducing inputs initialised random (size-10) subset of the training inputs
The obtained ELBO for each method computed by estimating the ELBO with 10, 000 samples from the posterior. The naive Monte Carlo estimator the log marginal likelihood used 10M samples (and the log-sum-exp trick). The KI. divergences between approximate and true
Evaluation.
27

---

Published as a conference paper at ICLR 2026
posteriors were then computed by taking the difference between the log marginal likelihood and ELBO.
F.2 Do MEANINGFUL BNN PRIORS EXIST?
F.2.1 ID REGRESSIONS
For all data-generating processes in Fig. 3, BNNPs with two hidden layers of 64 units were used (i.e. overall architecture of [1, 64, 64, 1)). with inference networks also with two hidden layers of 64 units. In all cases, the Gaussian likelihood had initial standard deviation of σy = 0.05 but this parameter was jointly optimised with all inference network weights @ and prior parameters V, Throughout, meta-level batch size of 5 was used.
Sawtooth, Heaviside, standard BNN prior functions. For these data-generating processes, 100, 000 synthetic datasets were generated where in each case the number of datapoints was sampled from (440, 100) and the proportion of these points which were used as the context set was randomly sampled from 2/(0.1, 0.5). The PP-AVI objective function was optimised via Adam under Py Torch defaults, with a learning rate linearly tempered down from 3e-3 to le-4 for 50, 000 steps (making for 2.5 epochs of the meta-dataset), and where 32 posterior samples were used to estimate the objective function at each step. For the sawtooth data and standard BNN prior data, ReLU activations were used throughout. For the Hcaviside data, tanh activations were used. Although the results probably reproducible with any activation functions under a large enough architecture, we found in our finite-network-wiiee width setting that picking the "right" activation function for the problem was quite helpful. For the Heaviside setting, multiplying the posterior predictive and KL terms within the PP-AVI objective by 0.1 yielded slightly nicer-looking prior-predictive samples. We hypothesise that this is because the true uuta--ennrating process does not sit within the set of those that can be represented by BNN prior, perhaps due the discontinuities when the function alternates between -1.0 and +1.0. Note that an infinitely wide BNN might be able to represent such stochastic processes in theory, but our finite-width case this objective function tempering seemed to help.
The data generated from standard BNN prior followed an identical generation procedure to that of Fig. 11, save that here the BNN architecture had hidden layer width of 64. The sawtooth data were obtained by randomly sampling inputs from (( (-0. 2.0), obtaining a random sawtooth function via
0.75
Ssan (.) =1)=11
+ 0.2562 + 1.33 x Remaiiner(--000 0.75)
(60)
3
2
where é and E2 were independently sampled from NNO,, 1), and then adding Gaussian noise of standard deviation 0.05 to the outputs the random function. The Heaviside data were generated by sampling inputs from Z1( 2((-5.0, 5.0), sampling function from GP prior with sguared-exponential covariance function unit lengthscale (and outputscale), passing the inputs through this function, subtracting the mean of the outputs from each of them, and then mapping all positive outputs to +1.0 and all negative outputs to 1.0. Finally, Gaussian noise standard deviation 0.01 was added to each output.
ECG data. The ECG data were synthetically generated using neurokit2's ecg_esiiee simulltt function. At a high level, for each dataset single "heartbeat" was isolated via signal cropping and then placed at a random location within six second window, with the resting voltage level used to pad either side of the heartbeat to fill the window. More specific details are best conveyed via the following code snippet.
deL Lt__y_ Log fa=100, nai 3010 Nong
"Gnnerate one synthet RCG saveforn and recurn (6, V) torch tenaors.
6.0
signal nk.eng sinclate! innes. hampling_rrteeece
heart heart__ratee00 rate=20, noisenncise, methoder simple"
28

---

Published as a conference paper at ICLR 2026
signal up.nean
signal /e 12-rp.nax (S1
singlo_ signal;in:: (3.5xfa) (6.5x2a) aoros_Dio2k cp.concatenaze (int
axismo) leng
spiit int (ap, rendon, und formi0.0, 3.0) * 26)
full op.ccncatenate baoocos_boock]sppae
siveie_vave_bookk, burcs_Liock(appittt)) axis=0)
normalize and convert Lo Leasors
ap ap.iinspacs (-d/2, 4/2, len(fulli, endpoinz=False)
x cerch.tcascrinn dtypertorch. float64) (aatett)_ueqquueee unequeezz
(full, diypes.orch (lo..s))...sseeeee
1t n_range is not Hone:
Lorch, candint (lovenin rangel,,,, high.max sixe- inds beschb.andpernnCC... (Cen (fuil) return X X[inda), Y[inds)
seturn X, Y
100, 000 datasets were generated in this way for training. The PP-AVI objective function was estimated via 24 posterior samples at each step and it was optimised via Adam under Py defaults with learning rate linearly tempered from 1c-3 to 5e-5. Between 2 and 64 context points (uniformly chosen) were randomly selected from the total of 6 x 100 = 600 datapoints in each task, with the rest being used as targets. 500, 000 training steps were carried out, making for 25 epochs. This BNNP used tanh nonlinearities.
F.2.2 IMAGE COMPLETIONS
The MNIST dataset of 28 x 28 pixel images of handwritten digits (LeCun et al., 1989) was used for this experiment. 2D regression datasets were generated by creating 2D grid of coordinates X in [-1.0, 1.0J² and using each image pixel as a y value corresponding to its respective grid coordinate x. The y values were mapped to the [0, 1] range. Repeating this procedure for every image in the default MNIST training set resulted in a meta-dataset of 60, 000 with which to train BNNP.
The BNNP's primary architecture consisted of four hidden layers of 64 units, so the full sequence of layerwise activations was [2, 64, 64, 64, 64, 1]. The inference networks also had four hidden layers of 64 units. ReLU nonlinearities were used throughout. Although the task was pixelwise regression, Bernoulli likelihood was used since the outputs took values exclusively in [0. 1J. Note that this does not turn the problem into a classication problem since the output values were not strictly or 1, but rather anywhere in the interval between 0 and 1 (inclusive).
The PP-AVI objective function was used to train the model, with the proportion of points used as context points uniformly chosen from the interval [0.01, 0.6]. Training was conducted over two episodes, each of 250, 000 steps and taking roughly 20 hours on an NVIDIA H100 GPU, The first episode saw the learning rate linearly tempered from 5e-4 down to 1e-5, and the second from 5e-5 down to le-6. Both episodes used 16 Monte Carlo samples, a meta-level batch size of 5. The purpose of the two training episodes due cluster usage rules limiting GPU job length to 24 hours.
7
7
7
177
7
7
8
0
8
3
3
8
(a) full
(b) context
(c) pred. mean (d) pred. std
(c) sample
(f) sample
(g) sample
Figure 12: Two example image completion tasks exccuted by the trained BNNP. We see that with limited
context data, the BNNP maintains distribution over different types of digit that might explain the data; in both cases it possible that the complete image could depict three.
29

---

Published as a conference paper at ICLR 2026
F.3 DOES GOOD PRIOR DOUBLE AS A LUNCH VOUCHER?
F.3.1 SYNTHETIC DATA SETTINGS
Data. The GP, Heaviside, and sawtooth training datasets were generated by uniformly sampling a number of datapoints from M(40, 100) and the sampled number of inputs were then uniformly sampled from [-5.0, 5.0). [-5.0, 5.0). or [-2.0, 2.0] for the cases of GP, Heaviside, and sawtooth data respectively. The proportion of these points used as context points was sampled from (0(111 0.6) in all cases. For sawtooth and Heaviside data, the noiseless outputs were obtained in the same way as in Section 1.2.1, and for GP data the noiseless outputs were obtained by sampling a function from zero-mean and squared-exponential kernel GP prior with a lengthscale of 0.5 and passing the inputs through the sampled function. The Heaviside data were corrupted by adding independently sampled Gaussian noise of standard deviation 0.01 to the outputs, whereas the GP and sawtooth data were corrupted by Gaussian noise of standard deviation 0.05. For the experiment itself, 16 datasets of each kind were generated once and saved, but with the difference that in all cases the context proportion was uniformly sampled from (0.055, 0.5) and, where the input domain was previously 1-500, 5.0), it was changed to 4.0..
BNNP configuration and training. The BNNP had two hidden layers of 48 units in the primary architecture as well as the inference networks. We used SiLU, tanh, and ReLU activations for each of GP, Heaviside, and sawtooth data respectively. In each case, meta-dataset of size 100, 000 was constructed and 100, 000 training steps performed over meta-level batch size of 5, giving a total of 5 epochs. The PV-AVI objective function was used with 32 Monte Carlo samples, learning rate that was linearly scheduled down from 3e-3 to 5e-5, and Adam under PyTorch defaults. The likelihood was Gaussian whose variance was learned jointly with vi and O. Training took one or two hours on an NVIDIA A100 GPU.
Baselines. For each of the 16 test datasets, each approximate inference algorithm was run twice; once for a BNN with a standard prior (zero-mean, standard deviation equal to the inverse of the square root of layer width) and once for BNN with the corresponding learned prior from the BNNP. All BNNs had the same architecture of two hidden layers of 48 units and with the same nonlinearity as the BNNP in each data setting. Each approximate inference algorithm used Gaussian likelihood with standard deviation 0.05 and Adam under PyTorch defaults was used in all cases where optimisation was required. The remaining settings were as follows:
SWAG. The network parameters were initialised by sampling from the prior before executing 5, 000 training steps with a learning rate of Se-3, The SWAG algorithm itself (Maddox et al., 2019) was then used to compute
approximate posterior by updating the SWA statistics 100 times at every 25-th iteration of PyTorch default gradient descent (not Adam) with a learning rate of 1e-2. The rank of the low-rank correction to the diagonal covariance matrix, K. was 64. The objective function was the unnormalised log posterior density of the training density.
MFVI, GIVI, and BNNP. To ensure a fair comparison to the con-meta-models the pre- trained BNNP itself was not used as a baseline; instead a new BNNP (with fixed likelihood noise of 0.05) was trained in each case with the prior parameters Vi fixed and only the inference network parameters θ set to trainable. For all three of these models, the (standard VI) ELBO was optimised for 75, 000 steps under a learning rate linearly tempered from 5e-3 down to 1c-4. 8 Monte Carlo samples were used in estimating the ELBO via the reparameterisation trick. GIVI used 24 inducing points, the inputs of which were initialised at a random subset of the context observations.
MCMC. Because it allowed for easier tuning of hyperparameters, LMC was implemented as HMC with a single leapfrog integrator step for each proposal (rather than the usual discretised Langevin dynamics). HMC was performed using a step size of 5e-4 for 5, 000 steps each with 100 leapfrog integrator steps. We used burn-in period of 500 and thinned the samples by keeping only every 50-th sample. These hyperparameters were selected by looking at unnormalised log- posterior/log- sasteriorloo-liikiiiooo///gs prior iteration plots and sample autocorrelation plots respectively. In the case of the learned prior for Heaviside data, a step size of le-4 was used instead. For LMC used a step size of 1e-3 (except for learned
30

---

Published as a conference paper at ICLR 2026
Heaviside prior, which had stepsize of 5e-4) for 250, 000 steps. The burn-in and thinning parameters were 10, 000 and 5, 000 respectively, chosen via the same method as for HMC. accept-reject step performed after each new proposal, and in all
A Metropolis-Hastingg cases the average acceptance frequency never fell below 0.99 (which is how the step sizes were tuned). There was no minibatching.
F.3.2
ERAS SETTING
Data. The precipitation data were obtained from the ERAS Land dataset (Muñoz Sabater et al., 2021). We focussed on an area of central Europe defined by coordinates [50", 5,, 45" 12"] corre- sponding to the rectangle borders in the order of [N, W. S, EJ. We used ten years of data from the beginning of 2010 through to the end of 2019. The grid granularities were 0.19 and 12 hours in the spatial and temporal axes respectively. The weather prediction task was spatial interpolation, where the goal is to impute missing grid values. The features used were 2D spatial coordinates 2m above ground-level, and the target was total precipitation in mm. All
and air temperature features and the targets were normalised, meaning each feature (and the targets) that were fed to the model had zero mean and unit standard deviation. Each timestep corresponded to single task, giving collection of about 24 12 X 365 x 10 7300 tasks, each with observations. Each task is therefore an image completion, save that the auxiliary information of ground air temperature was also provided to the model. A randomly chosen 16 datasets were reserved for testing, with the rest being used for meta-training the BNNP. For the training tasks, each time the task was encountered by the model (i.e. in different epochs) random 25% of the observations were selected, and of the selected observations the context points were randomly selected a proportion sampled from (00001, 0.5) with the rest serving targets, meaning that during training each task had between 0.01 x 0.25 x 3500 Re and 0.5 x 0.25 x 3500 s 438 context observations. The subsampling was done purely for faster training. For the test tasks, the proportion of context points for each task was sampled from (0.025, 0.25) and these points were selected randomly out of the points not in Switzerland. These splits were performed once and saved for the later evaluation across different models. The omitting of Swiss context points was done to ensure presence of a larger region without any context, in which the choice of prior would heavily influence predictions.
12-5 = 3500 total 50-44 x 0.1
0.1
BNNP configuration and training. The BNNP had three hidden layers of 64 units in the primary architecture as well as the inference networks, with ReLU activations throughout. The prior was learned by training the BNNP on the metadataset for 250, 000 training steps with a meta-level batch size of 5, giving a total of roughly 70 epochs. The PV- objective function was used with 24 Monte Carlo samples, a learning rate that was linearly scheduled down from 1e-3 to le-5, and Adam under Py PoTrch defaults. The likelihood was a Gaussian whose variance was learned jointly with V and O. Training took around 22 hours on an NVIDIA H100 GPU.
Baselines. For each of the 16 test datasets, each approximate inference algorithm was run twice; once for a BNN with standard prior (zero-mean, standard deviation equal to the inverse of the square root of layer width) and once for a BNN with the corresponding learned prior from the BNNP. All BNNs had the same architecture of three hidden layers of 64 units and ReLU nonlincarities. Each approximate inference algorithm used a Gaussian likelihood with standard deviation 0.05 and Adam under Py Torch defaults was used in all cases where optimisation was required. The remaining settings were as follows:
SWAG. The network parameters were initialised by sampling from the prior before executing 10, 000 training steps with a learning rate of 1e-3. The SWAG algorithm itself (Maddox et al., 2019) was then used compute an approximate posterior by updating the SWA statistics 200 times at every 25-th iteration of Py Torch default gradient descent (not Adam) with a learning rate of 5e-3. The rank of the low-rank correction to the diagonal covariance matrix, K, was 64. The objective function was the unnormalised log posterior density of the training density.
MFVI, GIVI, and BNNP. The treatment of BNNPs was the same in the synthetic setting. For all three of these models, the (standard VI) ELBO was optimised for 75, 000 steps under learning rate linearly tempered from 5e-3 down to 1e-4. 8 Monte Carlo samples were used in estimating the ELBO via the reparameterisation trick. GIVI used 24 inducing points, the inputs of which were initialised random subset of the context observations.
31

---

Published as a conference paper at ICLR 2026
MCMC. For the same reason in the synthetic setting, SGLD implemented as SGHMC with a single leapfrog integrator step for each proposal (rather than the usual discretised Langevin dynamics). SGHMC was performed using step size of lc-4 for 5, 000 steps each with 100 leapfrog integrator steps. We used a burn-in period of 2, 000 and thinned the samples by keeping only every 50-th sample. These hyperparameters were selected by looking at unnormalised log wwwrrioologglikilhoooo///oo
vs iteration plots and sample autocorrelation plots respectively. For SGLD we used step size of 5c-4 for 200, 000 steps. The burn-in and thinning parameters were 50, 000 and 2, 500 respectively. chosen via the same method as for SGHMC. No Metropolis Hastings accept-reject step was performed. and minibatch size of 128 was used in both cases. In cases for which the total number of context points was less than or equal to the batch size, the full batch was used.
F.3.3 EVALUATION
In all data settings, the metrics of mean absolute error and log posterior predictive density were used. In both cases, they were computed on the target set in each of the 16 datasets after training. For computing the metrics, 1000 posterior samples were obtained from the variational and SWAG posteriors, and all the available samples after burn-in and thinning for the MCMC methods were used (60 samples in the ERA5 setting, 48 LMC samples and 90 HMC samples the synthetic setting). The mean absolute error for each test task was computed as the absolute error between the true targets and the predictive mean (of the predictive samples), averaged over all target points in the task. These 16 scores were then averaged to give the final metric, with the standard error of these 16 values used as the errorbars. The log posterior predictive density for each task was computed the likelihood density of the full target averaged over posterior samples (i.e. Monte Carlo estimate of the posterior predictive integral), and then divided by the number of target points (for easier comparison between tasks with different number of targets). Of course, these computations were performed in the log-domain via the LogSumExp trick. These 16 scores were then averaged to compute the final metric for each method, with the standard error of the 16 values used as the errorbars. Note that for the MAE metric in the ERA5 setting, the data were transformed from the normalised domain to the original domain.
32

---

Published as a conference paper at ICLR 2026
F.3.4 CLOUDY WITH CHANCE OF MEANINGFUL PRIORS
In this section we qualitatively compare precipitation patterns sampled from each of 1.) the real world (i.e., ERA5), 2.) standard BNN prior, and 3.) our BNNP-learned prior. The comparison is shown in Fig. 13. We see from the examples in the left-hand column that real-world precipitation can be somewhat spatially sparse, with much of the reflected in the samples from the learned prior in the right-hand column, but not in those of the standard BNN prior. Furthermore, precipitation values must be non-negative- ttire is observed in the real-world examples as well as those of the learned prior, but not for the standard prior (see the minimum values the colour-value key for each example). Finally, the alps cause variations in temperature with a relatively high spatial frequency (on the scale of our area), and the models are given realistic settings of this feature from the ERA5 dataset. It is the variation of this feature that causes the rough patterns over southern Switzerland in the sampled precipitation patterns from standard BNN prior, which appears be overly/naively sensitive it. By contrast, the BNNP seems to have learned prior that models this feature more appropriately, most likely by encoding the interaction between this feature and the location. For example, we know that altitude varies across our region and that precipitation depends on both temperature and altitude, suggesting the temperature feature's importance might vary across the region. We suspect the BNNP's learned prior incorporates such notion of maatially-variable feature importance.
receiving precipitation. This behaviour
is clear that the learned prior more faithfully models real-world precipitation behaviour than the standard BNN prior does. Indeed, model for precipitation that allows for negative values-raindropa flying up from the earth and into the clouds-is somewhat questionable. It is no wonder, then, that predictive performance is also superior when using the learned prior than when using naïve one (Fig. 6). In general, we would encourage researchers and practitioners to question their use of standard BNN priors, and consider the possibility of meta-learning their priors instead.
(a) ERAS
(b) standard prior
(c) learned prior
Figure 13: Sampled precipitation patterns over an area of central Europe centered on Switzerland. The samples from the learned BNN prior much more realistic than those of the standard BNN prior.
F.4 CAN A RESTRICTED PRIOR IMPROVE META-LEVE DATA EFFICIENCY?
F.4.1 DATA
Abalone. We downloaded the Abalone dataset (Nash et al., 1994) from the UCI repository. These data have 8 input features such as height, diameter, shucked weight. One of the features is the specimen's sex, which is of three values: male, female, or infant. We separated the data into three
33

---

Published as a conference paper at ICLR 2026
partitions based the value of the sex feature, and then discarded this feature in all cases to leave only features. All imput features were then normalised to have zero mean and unit standard deviation, and this done globally (rather than within each of the three datasets). The output variable for each specimen is the number of rings, which is closely related to the age of the specimen. This variable was globally normalised in the same way as the features. The datasets corresponding male and female specimens were used as the meta-dataset for training the meta-lcarners, so | = 2. The male and female datasets had 1528 and 1307 observations respectively. The dataset corresponding to infant abalones was used as the test dataset, and it contained 1342 points. Similarly to how the ERA5 test data included large region (Switzerland) with no context points in order to emphasise the impact of the prior on predictions, in the infant abalone daaset we constructed a hypercube in the feature space that contained roughly 20% of points and set all of these to targets. Of the remaining points, a random selection were selected as context points such that the overall proportion of context points was 0.25, This split was performed once and saved so that all baselines were evaluated on the test task with the same context/target split.
Paul15. We downloaded the Paul15 15 mouse bone marrow cell dataset (Paul et al., 2015) using scanpy (Wolf et al., 2018). The raw dataset contains roughly 3, 400 features, where each one represents the degree to which certain gene is expressed. After normalisation, we used PCA to map this down to the top 100 eigen-genes. The target feature for this dataset is the pseudotime, a value computed by biologists that serves as a metric for cell age/development stage. A young stem cell will have a low pseudotime, whereas a cell that has already specialised for a certain role will have a higher pseudotime. We normalise the psuedotime feature well. Paul et al. (2015) provide partitioning of the data into 19 clusters, broadly corresponding to cells of the same type/specialissaiin (or lack thereof for young cells). We use these clusters to convert the problem into meta-learning one. We randomly selected 10 of the clusters to use as the training meta-dataset, and of the remaining 9 we selected the task with the most datapoints to use as the test task. This test task and target sets with a context proportion of 0.2, and this split was saved and used for all baselines. The numbers of datapoints the 10 training tasks were 153, 69, 373, 164, 9, 43, 246, 180, 167, and 63. The test task had a total of 329 datapoints.
split into context
F.4.2 MODELS
BNNP. For the Abalone setting the BNNP had three hidden layers of 64 units, as did its inference networks. For the Paul15 setting, there were four hidden layers of 96 units throughout instead. SiLU nonlincarities were used in both cases. The likelihood was Gaussian with a learned standard deviation that was initialised to 0.1. The prior-learnability decimal represents the proportion of all the parameters of the prior V for which parameter requi i res_grad was set to True. Each weight in the BNNP's s primary architecture has a prior mean, a prior variance, and prior covariances with the other weights corresponding to the same output neuron. To set a fixed proportion p of these to trainable, we found the total number of weights [W], found the nearest integer to p) W and then, proceeding through all weights in the network from the first layer through to the last, we set the prior parameters associated with each weight to trainable until we reach the integer. Training was performed by optimising the PP-AVI objective. The objective was estimated from 16 Monte Carlo samples at each step, and optimised for 100, 000 steps. For the Ahalone setting, the meta-level batch size was (i.e. full-batch training) but for Paul15 it was 5. meaning training was performed for 100, 000 epochs for the Abalone setting and 50, 000 for the P one. In both cases, every time a training dataset was encountered, it was randomly split into a new context-target partition where the proportion of context points was itself randomly sampled from (0. 0.6) each time. The optimiser was Adam under Py defaults, with learning rate that was linearly scheduled down from 5e-4 (Abalone) and le-4 15) to 5e-5.
Baselines. In all cases, optimisation was performed with Adam under Py Torch defaults.
BNNs. As with the BNNP, SiLU nonlinearities were used in all cases, and the BNNs had three hidden layers of 64 units in the case of the Abalone data, but four hidden layers of 96 units in the case of the Paul15 data. GIVI had 128 inducing points in the Abalone setting and 196 in the Paul15 one, and in both cases the inducing inputs were initialised to random subset of the test set context set. As non meta-leamers, these models were trained only on the test-set context set in each case. As demonstrated in Bui (2021) or in first experiment,
34

---

Published as a conference paper at ICLR 2026
GIVT's approximate posterior is of high enough quality that the ELBO can be used for model selection, and so the Gaussian likelihood's 's standard deviation was learned in the case of GIVI (initialised to 0.1), and the learned/optimal values were then used for MFVI (but frozen in place). The standard variational BL was optimised for 75, 000 steps, each time being estimated from 8 Monte Carlo samples. The learning rate linearly tempered down from 5c-3 to le-4.
NP and BNP. These models used encoders and decoders each with three hidden layers of 256 units and SiLU nonlinearities throughout. They used a Gaussian likelihood with a standard deviation initialised to 0.1 but optimised with the rest of the parameters. The objective function was the average target-set log posterior predictive density (see Section A.3), which, for each task, was estimated from 16 latent variable posterior samples. The meta-level batch sizes were 2 for the Abalone setting and 5 for the Paul15 one. Optimisation was performed for 500, 000 iterations with a learning rate linearly tempered down from le-4 to 1e-5. These models were very slow to converge, hence the very high number of training steps. Every time a training dataset was encountered, it was randomly split into new context-target partition where the proportion of context points was itself randomly sampled from (0.1, 0.6).
AR-TNP. This model consisted of three self-attention blocks where every layer had dimen- sionality of 256 and SiLU nonlinearities were used throughout. The objective function was again the average target-set log posterior predictive density, but as a type of conditional NP, the model outputs the marginals for each target and so no sampling was required, just multiplication of the (independent) marginals. The meta-level batch sizes were 2 for the Abalone setting and 5 for the Paul one. Optimisation was performed for 50, 000 steps (this model reached convergence much quicker than the other NPs), and every time a training dataset was encountered, it randomly split into proportion of context points was itself randomly sampled from (0.1., 0.6). Not that this model was trained in the same way regular TNP(-D).
context-targe' partition where the
F.4.3 EVALUATION
For all models, training was repeated four times for seeds [21, 42, 69, 420). In each of the four cases and for all models, 1000 predictive samples were used to compute the metrics, where for AR-TNP the samples were obtained via the autoregressive scheme developed by Bruinsma et al. (2023). For each data setting, the metrics were computed as follows. The mean absolute error for a test task was computed the absolute error between the true targets and the predictive mean (of the predictive samples), averaged over all target points in the task. The 4 scores obtained across the four seeds were then averaged to give the final metric, with the standard error of these 4 values used the errorbars. The log posterior predictive density for each task was computed as the likelihood density of the full target set averaged over posterior samples (i.e. Monte Carlo estimate of the posterior predictive integral), and then divided by the number of target points (for easier comparison between tasks with different number of targets). Of course, these computations were performed in the log-domain via the LogSumExp trick. These 4 scores were then averaged to compute the final metric for each method, with the standard error of the values used as the errorbars. Note that for the MAE metric, the data were transformed from the normalised domain to the original domain.
35


Reviews:
[{"review_id": "fbXVJ7XgkQ", "reviewer": "Reviewer_G27C", "summary": "This paper introduces a new model called a Bayesian deep neural process (BDNP), which combines aspects of the global inducing point variational posteriors (Ober and Aitchison 2021) and the (latent) neural process (NP) (Garnelo 2018). It is a member of the neural process family, although it differs in that the latent variable is the parameters of the decoder rather than an input to the decoder. Like Ober and Aitchison 2021, the variational posterior uses a psuedo output and a pseudo noise term to exploit the conjugacy of Bayesian linear regression per layer. Unlike Ober and Aitchison 2021, there is a psuedo output and a pseudo noise term for each data point and, instead of treating them as trainiable parameters, the BDNP uses amortization. That is, they are outputs of a trained “inference network”, which is analogous to the encoder of an NP. The model is trained by minimizing a log posterior-predictive density, as in an NP, and a ELBO term, as in VI. In the experiments, the paper shows better KL to the true posterior on a toy example, the ability to meta-learn BNN priors in 1D regression tasks, a comparison of different inference methods under the same meta-learned prior, and an experiment restricting the learned prior.", "questions": "- Unlike an NP, there is no aggregation of encodings in the BDNP, to ensure permutation invariance. In an NP this is needed to ensure exchangeability of the stochastic process. Instead, the aggregation happens in the loss function. In the appendix, you write that “BDNP and BDAM exhibit this through permutation invariance with respect to the context observations and permutation equivariance with respect to the target inputs”. Can you explain how BDNP defines an exchangeable process without the aggregator? \n- This relates to the above question: I’m confused about how exactly the model meta learns the prior. In an NP, the information from the multiple datasets comes from the encoding. The global latent variable is conditioned on the encoding. My understanding is that this is the “meta” learning done in an NP. In the BDNP, the encodings are, I would argue, variational parameters (as in how they are used in Ober and Aitchison 2021). The prior parameters of the BNN are trained similarly to how they would be trained in standard VI, using the fact that the ELBO is a lower bound on the marginal likelihood, which is used for model selection. Note that in an NP, only the log predictive term is in the loss function, not the ELBO term. Can you comment on how the prior meta learning is different from a standard NP? Am I understanding the BDNP correctly? \n- You train a separate inference network $g_l$ for each layer, is that correct? This seems like it would create a lot of parameters unless each network is small.\n- Can you explain again why the BDNP scales favorably in network width? How do you get around the computation of the mean and covariance in equations 6 and 7 being cubic scaling in the width?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- This is an interesting paper that addresses shortcomings of Bayesian deep learning in a new way, as far as I know. The complexity of the model is both a pro and con. \n- I like the idea of amortizing the psuedo observations of Ober and Aitchison 2021\n- Figure 6 was interesting. This shows the learned prior is useful regardless of the inference method. \n- The experiments focus on meaningful questions, rather than the largest scale experiments", "weaknesses": "- There are a few differences with a standard NP, enough that I wonder if this is really an NP (see my questions below). \n- Overall, this is a fairly complicated model, which makes it difficult to understand what each component is doing. I appreciated the discussion of the objective function in the appendix, but I think some of this discussion should appear in the main text. The objective doesn’t feel well motivated as it’s written now. More discussion of how this differs from a standard NP would help. \n- The introduction discusses how BNNs revert to GPs in the infinite width limit but in the finite width limit considered in this paper, BNNs are not GPs. There are many criticisms to make of BNNs, but I don’t think this one is relevant. \n- Unless I missed it, the experiments seem to lack details (architectures, optimizers, etc)\n- I found figure 1 (b) a bit confusing. Is this for a one hidden layer network? \n- Definitions 1, 2, and 3 aren’t well-defined because they reference a quantity being “small”, but it’s not clear what small means. \n- There are a few statements that are too strong for me, e.g. “the impressive performance of our explicitly Bayesian meta-learning setup”, “The BDNP’s approximate posterior is a very good on”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oE1ifpBky7", "reviewer": "Reviewer_Kn36", "summary": "This work aims to incorporate a meta-learning prior into the weight parameters of Bayesian Neural Networks (BNNs). In doing so, it builds BNNs that can be interpreted as a type of latent neural process model, where the latent variable is replaced by the posterior distribution of the BNN’s weights, which varies depending on the input. Specifically, it introduces an inference network that generates pseudo-labels for each linear layer and updates the layer’s posterior using these pseudo-labels and the prior. Once the parameters of the inference network and the BNN prior distribution are learned, indicating that the BNN has a well-posed prior, it evaluates the corresponding BNN's ability to perform various tasks effectively. Furthermore, it investigates the crucial role of approximate Bayesian learning under a well-posed prior.", "questions": "* It appears that the parameters of the inference network and the prior distribution are jointly trained according to Eq. (10). The ELBO term seems primarily used for training the parameters of the prior distribution, while the conditional likelihood for the target task appears to update both the inference network and the prior distribution. Is this interpretation correct? If not, could you please clarify this point further?\n\n* In this training procedure, was the inference network pre-trained, or was it trained jointly from scratch?\n\n* How is the scale of prior learnability (ranging from 0.0 to 1.0) measured in Figure 7? Moreover, if the amount of data is limited and the prior is well-posed, shouldn’t the prior be beneficial? Why does the performance of BDNP with a prior-learnability of 1.0 degrade so significantly in Figure 7(a)?\n\n* The following works seem relevant to the concept of meta-priors for BNNs and neural processes, and might be useful to include in the related work section:\n\nHierarchical Gaussian Process Priors for Bayesian Neural Network Weights, NeurIPS 2020\n\nBayesian Convolutional Deep Sets with Task-Dependent Stationary Prior, AISTATS 2023", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* This work introduces an interesting idea of incorporating a meta-learning prior into BNNs and establishes a connection to neural processes.\n*  It also presents an amortized linear layer structure that enables each layer to function as a Bayesian layer with an amortized prior.", "weaknesses": "* Although the proposed method is technically sophisticated, it is unclear what specific problem it aims to address with the proposed structure. For instance, it is not evident whether the main contribution lies in emphasizing the amortized prior for BNNs and its benefits, or in investigating approximate training for BNNs with a well-chosen prior.\n\n* The training procedure for the parameters of the inference network and the prior distribution is insufficiently described. Beyond presenting the loss objective, a pseudo-code illustration or detailed algorithmic explanation would be necessary for clarity.\n\n* Moreover, the quality of the learned prior and the corresponding posterior is likely influenced by the design of the inference network. However, this aspect appears to be underexplored in this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ptlYan7ypE", "reviewer": "Reviewer_poex", "summary": "This paper addresses the challenge of defining useful priors in a Bayesian deep learning setting by introducing the \nBayesian Deep Neural Process (BDNP). The central idea is to meta-learn a parametric prior for Bayesian Neural Network \n(BNN) weights, coupled with amortized variational inference, using a meta-dataset of related tasks. Conceptually, the \nBDNP can be viewed as a neural process (NP) where the latent variable represents the weights of a BNN, and the decoder \nis the BNN itself. The paper presents compelling empirical evidence demonstrating the BDNP's ability to achieve \nhigh-quality approximate inference compared to other VI methods and to learn meaningful priors effectively.", "questions": "- In the experiments comparing BDNPs to standard NPs, did you also learn the prior for the latent variable?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The presentation of the paper (language and structure) is well done. The theoretical section is sound and nicely concise. \nThe experimental section is comprehensive and well-executed. Presented ablation studies, particularly those evaluating the \nquality of approximate inference against other VI methods and the qualitative and quantitative analysis of learned priors, \nstrongly support for the paper's claims. The authors provide a commendably clear and realistic discussion of the \nlimitations of their approach, such as potential scalability issues for very large or complex architectures.", "weaknesses": "- **Training Loss Justification:** The paper's core training objective, PP-AVI, deviates from standard NP-losses like NP-VI. \nWhile Appendix A.4 provides a detailed justification for this choice over NP-VI, this reasoning should be\n(at least partly) integrated into the main text. Furthermore, the PP-AVI 'loss' is formulated as a maximization \nobjective rather than a loss.\n- **Clarification on Within-Task Minibatching:** The claim that \"the ability to minibatch a forward pass over a given \ncontext set is rare in the context of neural processes\" requires more elaboration. The authors acknowledge\nthat inference in BDNP can be viewed as Bayesian context aggregation in NPs [1]. Consequently, iteratively updating the\nlatent posterior over multiple mini batches via sequential Bayesian inference is also applicable to classical NPs. \nThis also works for other context aggregation mechanisms such as mean-aggregation or max-aggregation. Only context \naggregation in the Attentive NP can not be minibatched, but the same holds for tha Attentive BDNP (AttBDNP). Furthermore, \nthe method of using gradients from a random minibatch during training (Appendix C) is a significant detail. This \napproximation introduces potential biases and its implications (e.g., impact on convergence, generalization) should be \nmore prominently discussed in the main text. Additionally, the assertion that \"our scheme maintains the ability for the \nBDNP to learn to generalize predictive inference across various context set sizes during training\" requires theoretical \nor empirical evidence to support it.\n- **Evaluation Metrics for Correlated Samples:** The per-datapoint Log Posterior Predictive Density (LPPD) is used as one\nprimary metric without a formal definition in the paper. More importantly, a key advantage of latent NPs (including BDNPs) \nover conditional NPs (CNPs) is their capacity to produce correlated function samples. LPPD, being a per-datapoint metric,\nmay not fully capture or adequately evaluate this crucial aspect of model calibration and uncertainty representation. \nIt is recommended to also evaluate the posterior predictive log-likelihood over the entire target set, i.e., \n$p(y^{1:M}_T\\mid x^{1:M}_T, D_C)$, as this metric would provide a more holistic assessment of the model's ability to \ngenerate coherent and well-calibrated correlated function samples.\n\n[1] Volpp et al., 'Bayesian Context Aggregation for Neural Processes'", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H8fP2Prpmt", "reviewer": "Reviewer_xjWL", "summary": "The paper introduces a meta-learning model to infer better priors for Bayesian neural nets (BNNs) using a neural process (NP)-based approach, which treats the BNN's weights as latent variables.\nTo ensure scalability, it relies on layerwise factorizations and introduces within-task minibatching.", "questions": "- Q1: The paper doesn't discuss computational cost and runtime requirements. How expensive is the approach?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes an interesting and novel combination of BNNs and NPs for an important unsolved task in the Bayesian deep learning literature.\n- The proposed objective (eq. 10) consists of interpretable and well-motivated terms.\n- The approach is scalable and seems to provide promising results across a range of experiments.", "weaknesses": "- Proposition 1 is promising in that the objective is well-motivated but does not provide any guarantees that the vague definition of \"small\" in Definitions 1–3 is actually achievable. For example, are all three terms minimizable at the same time, or do they balance each other?\n- The empirical evaluation is promising but currently rather unusable, as it is completely devoid of any information on architectures, hyperparameters, training procedures, etc.\n- No code is provided.\n\n## Minor weaknesses\n- Hiding the definitions of the three desiderata (167–170) in the appendix makes that part of the paper feel rather handwavy, even though it isn't.\n- Details are missing from the results, e.g., over how many seeds are the error bars in Figure 6?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)