Martingale Posterior Neural Processes
Authors: Hyungi Lee
Eunggu Yun
Giung Nam
Edwin Fong
Juho Lee
Venue: ICLR 2023 notable top 25% (2023)

Abstract: A Neural Process (NP) estimates a stochastic process implicitly defined with neural networks given a stream of data, rather than pre-specifying priors already known, such as Gaussian processes. An ideal NP would learn everything from data without any inductive biases, but in practice, we often restrict the class of stochastic processes for the ease of estimation. One such restriction is the use of a finite-dimensional latent variable accounting for the uncertainty in the functions drawn from NPs. Some recent works show that this can be improved with more “data-driven” source of uncertainty such as bootstrapping. In this work, we take a different approach based on the martingale posterior, a recently developed alternative to Bayesian inference. For the martingale posterior, instead of specifying prior-likelihood pairs, a predictive distribution for future data is specified. Under specific conditions on the predictive distribution, it can be shown that the uncertainty in the generated future data actually corresponds to the uncertainty of the implicitly defined Bayesian posteriors. Based on this result, instead of assuming any form of the latent variables, we equip a NP with a predictive distribution implicitly defined with neural networks and use the corresponding martingale posteriors as the source of uncertainty. The resulting model, which we name as Martingale Posterior Neural Process (MPNP), is demonstrated to outperform baselines on various tasks.

Published as a conference paper at ICLR 2023
MARTINGALE POSTERIOR NEURAL PROCESSES
Hyungi Lee', Eunggu Yun', Giung Nam', Edwin Fong", .Jhoo Lee
KAIST, Novo Nordisk, AAITRCCS
(1hk2 708, eunggu. yun, giung, juhoLeeJekaist.ce Peteefnoovonordakk.com
ABSTRACT
A Neural Process (NP) estimates stochastic process implicitly defined with neu- ral networks given stream of data, rather than pre-specifying priors already known, such as Gaussian processes. An ideal NP would learn everything from data without any inductive biases, but in practice, we often restrict the class of stochastic processes for the ease of estimation. One such restriction is the use of fiiitedimensional latent variable accounting for the uncertainty in the func- tions drawn from NPs. Some recent works show that this can be improved with more "data-driven" source of uncertainty such as bootstrapping. In this work, we take a different approach based on the martingale posterior, a recently developed alternative to Bayesian inference. For the martingale posterior, instead of speci- fying prior-likelihood pairs, a predictive distribution for future data is specified. Under specific conditions on the predictive distribution, can be shown that the uncertainty in the generated future data actually corresponds to the uncertainty of the implicitly defined Bayesian posteriors. Based on this result, instead of assum- ing any form of the latent variables, we equip implicitly defined with neural networks and use the corresponding martingale pos- teriors the source of uncertainty. The resulting model, which name Mar- tingale Posterior Neural Process (MPNP), is demonstrated to outperform baselines on various tasks.
with predictive distribution
INTRODUCTION
A Neural Process (NP) (Garnelo et al., 201 2018a;b) meta-learns a stochastic process describing the relationship between inputs and outputs in a given data stream, where each task the data stream consists of meta-training set of input-output pairs and also meta-validation set. The NP then defines an implicit stochastic process whose functional form determined by a neural network taking the meta-training set an input, and the parameters of the neural network optimized to maximize the predictive likelihood for the meta-validation set. This approach philosophically different from the traditional learning pipeline where would first elicit a stochastic process from the known class of models (e.g., Gaussian Processes (GPS) and hope that it describes the data well. An ideal NP would assume minimal inductive biases and learn as much possible from the data. In this regard, NPS can be framed as "data-driven" way of choosing proper stochastic processes.
An important design choice for NP model is how capture the uncertainty in the random functions drawn from stochastic processes. When mapping the meta-training set into a function, one might employ deterministic mapping as Garnelo et al. (2018a). However, it more natural to assume that there may be multiple plausible functions that might have generated the given data, and thus encode the functional (epistemic) uncertainty as a part of the NP model. Garnelo al. (2018b) later proposed to map the meta-training set into fixed dimensional global latent variable with a Gaussian posterior approximation. While this improves upon the vanilla model without such a latent variable (Le et al., 2018), expressing the functional uncertainty only through the Gaussian approximated latent variable has been reported to be a bottleneck (Louizos et al., 2019). To this end, the
Lee et al. (2020) and Lee et al. (2022) propose apply bootstrap the meta-traie meta-training set uncertainty arising from the population distribution as source for the functional uncertainty.
In this paper, take rather different approach to define the functional uncertainty for NPs. Specif- ically, we utilize the martingale posterior distribution (Fong et al., 2021), a recently developed alternative conventional Bayesian inference. In the martingale posterior, instead of eliciting

---

Published as a conference paper at ICLR 2023
likelihood- prior pair and inferring the Bayesian posterior, elicit joint predictive distribution future data given observed data. Under suitable conditions on such a predictive distribution, it can be shown that the uncertainty due to the generated future data indeed corresponds to the uncertainty of the Bayesian posterior. Following this, we endow a NP with a joint predictive distribution defined through neural networks and derive the functional uncertainty as the uncertainty arising when map- ping the randomly generated future data to the functions. Compared to the previous approaches of either explicitly positing finiee-imensional variable encoding the functional uncertainty deriv- ing it from a population distribution, our method makes minimal assumptions about the predictive distribution and gives more freedom to the model choose the proper form of uncertainty solely from the data. Due to the theory of martingale posteriors, our model guarantees the existence of the martingale posterior corresponding to the valid Bayesian posterior of an implicitly defined param- eter. Furthermore, working in the space of future observations allows us to incorporate the latent functional uncertainty path with deterministic path a more natural manner.
We name our extension of NPs with the joint predictive generative models the Martingale Posterior Neural Process (MPNP). Throughout the paper, we propose an efficient neural network architecture for the generative model that is easy implement, flexible, and yet guarantees the existence of the martingale posterior. We also propose a training scheme to stably learn the parameters of MPNPs. Using various synthetic and real-world regression tasks, we demonstrate that MPNP significantly outperforms the previous NP variants in terms predictive performance.
2 BACKGROUND
2.1 SETTINGS AND NOTATIONS
Let X' = Rd be an input space and y = Rdu be an output space. We are given a set of tasks drawn from an (unknown) task distribution, 71, T2, i.d.
(T) A task T consists of a dataset Z
and index set e, where Z {=>)²= with each z; = (Ni, mt) E X x y is a pair of an input and an output. We assume Z are i.i.d. conditioned on some function f. The index set e Ç [n] where [n] == (1, . .n) defines the context set Ze {=)ise- The targel set Z₃ is defined similarly with
the index f = [n]] \
2.2 NEURAL PROCESS FAMILIES
Our goal is to train class of random functions X' y that can effectively describe the relationship between inputs and outputs included in a set of tasks. Viewing this as a meta-learning problem, for each task T, we can treat the context Ze as a meta-train set and target Z, as meta- validation set. We wish to meta-learn mapping from the context Ze to random function f that recovers the given context Ze (minimizing meta-training error) and predicts Z₁ well (minimizing meta-validation error). Instead of directly estimating the from Ze to a predictive distribution for finite-dimensional observations.
f. we learn mapping
p( 2o) /
[II p(yilf, xx))
II p(u: 24) p Z.)df,
where we assuming the outputs Y are independent given f and X. We further restrict ourselves to simple heteroscedastic Gaussian measurement noises,
p(ylf,z) (z))
where Ho X y and oj X + R map an input mean function value and corresponding variance, respectively. R is a parameter indexing the function f, and thus the above predictive distribution can be written as
/
p( X, 2.)
II N N(uuluo(Z:). (z) p(0 Zc)d0.
NP parametric model which constructs a mapping from Z, to 0 as a neural network. The simplest version, Conditional Neural Process (CNP) (Garnelo et al., 201 8a), assumes a deterministic mapping from Z, to 0 as
p() (f), Te Senc (Zei Deme),
2

---

Published as a conference paper at ICLR 2023
where (x) is the Dirac delta function (which gives zero if & and f (2(2drr 1) and fene is a permutation-innvnn neural network taking sets as imputs (Zaheer et al., 2017), parameterized by Penc. Given summary 0 = Te of a context Ze, the CNP models the mean and variance functions (14, 0²) as
(og - (er
(5)
where faee is feed ferdaforward neural network parameterized by Pdec. Here the parameters (denes Qsec) are optimized to maximize the expected predictive likelihood over tasks, E-[log Z.).
Note that in the CNP, the mapping from Ze to 0 is deterministic, so it does not consider functional uncertainty or epistemic (model) uncertainty. To resolve this, Garnelo al. (2018h) proposed NP which learns mapping from arbitrary subset Z' C Z to variational posterior q(0) Z') approxi- mating Z') under an implicitly defined prior p(0):
(mz fizr,log log sz') = fene (Z'; dene), p(0]Z') (00)Z)) = (00m.. 4n).
(6)
With fene. the Evidence Lower BOund (ELBO) for the predictive likelihood is written as
Lo) N
12(012) (ool:)/loooo/ylllueee (NNiloo of (x) DRL D
2E|m]
(012) (og -
iE[m]
An apparent limitation of the NP is that it assumes a uni-modal Gaussian distribution as an approx- imate posterior for z(0) Aside from the limited flexibility, it does not fit the motivation of NPs trying to learn as much as possible in a data-driven manner, as pre-specified parametric families are used.
There have been several improvements over the vanilla CNPs and NPs, either by introducing attention mechanism (Vaswani et al., 2017) for sene and saee (Kim et al., 2018), using advanced functional uncertainty modeling (Leee et al., 2020; Lee et al., 2022). We provide a detailed review of the architectures for such variants in Appendix A. Throughout the paper, we will refer this class of models as Neural Process Family (NPF).
2.3 MARTINGALE POSTERIOR DISTRIBUTIONS
The martingale posterior distribution (Fong et al., 2021) is recent generalization of Bayesian in- ference which reframes posterior uncertainty on parameters as predictive uncertainty on the unseen population conditional on the observed data. Given observed samples Z {= i.i.d. from the sampling density Po. one can define the parameter of interest as functional of Po. that is
0o (po) = arg min f ((2, Po(d)))
where is a loss function. For example, ((200 = (a - 0)² would return 0g as the mean, and ((z, 0) = -llog 0) would return the KL minimizing parameter between p(- 0) and po-
The next step the martingale posterior is to construct a joint predictive density on Z' = (2 for some large N, which we write as p( Z' Z). In a similar fashion
bootstrap, one can imagine drawing Z' p(Z' Z), then computing ((gN) where gN(z) 1 > dz, The predictive uncertainty in Z' induces uncertainty in ((gN) conditional on Z. The key connection is that if p(Z') Z) the Bayesian joint posterior predictive density, and
N
log p(= 0). then 0(gN) is under weak conditions. In other words, posterior uncertainty in 0 equivalent to predictive uncertainty in {-
distributed according to the Bayesian posterior (0 Z) as N
Fong et al. (2021) specify more general p( Z' 7) directly beyond the Bayesian posterior predictive, and define the (finite) martingale posterior as n (0 E Á Z) = f [0(gg) E A) p(dz' Z). In particular, the joint predictive density can be factorized into sequence of 1-step-ahead predictives, p(Z' Z) II p(z; 21:i-1), and the sequence {p(z: 2 (1:5-1) ) is elicited directly, removing the need for the likelihood and prior. Hyperparameters for the sequence of predictive distributions can be fitted in a data-driven way by maximizing
log p(
log p(z (14-1);
3

---

Published as a conference paper at ICLR 2023
which is analogous the log marginal likelihood. Fong al. (2021) requires the sequence of predictives to be conditionally identically distributed (c.i.d),, which is a martingale condition on the sequence of predictives that ensures 9N exists almost surely. The Bayesian posterior predictive density is a special case, as exchangeability of p(Z' 2) implies the sequence of predictives is c.i.d. In fact, De Finetti's theorem (De Finetti, 1937) guarantees that any exchangeable joint density implies an underlying likelihood-prior form, but specifying the predictive density directly can be advantageous. It allows for easier computation, as we no longer require posterior approximations, and it also widens the class of available nonparametric predictives which we will see shortly.
2.4 EXCHANGEABLE GENERATIVE MODELS
To construct a martingale posterior, we can either specify a sequence of one-step predictive distri- butions or the joint predictive density distribution directly, long as the c.i.d. condition is satis- fied. Here, we opt to specify an exchangeable p(Z' Z) directly, which then implies the required c.i.d. predictives. We briefly review exchangeable generative models which can be used to specify the exchangeable joint predictive. For a set of random variables Z = {=x)== with each 2 Z = Rd, we say the joint distribution p(Z) is exchangeable if it is invariant to the arbitrary permutation of the indices, that is, p(Z) p(ı I Z) for any permutation of [n]. A simple way to construct such exchangeable random variables to use a cammuutaioo-equivvraae f(Z) for any л. Given f,
mapping. A
mapping f n + 2" is permutation equivariant if f( Z) we can first generate i.i.d. random variables and apply f to construct a potentially correlated but exchangeable of random variables Z as follows:
: (e))/ Lid..
Z = f(E)-
po:
For f. we employ the modules introduced in Lee al. (2019). Specifically, equivariant module called Induced AelfnAnttntion Block (ISAB). An ISAB mixes input sets through learnable set of parameters called inducing points via Multihead Attention Blocks (MABs) (Vaswani et al., 2017; Lee et al., 2019).
permutation
ISAB(E)= MA E Raxd where MA e R=dd
Here, E R.xd is a set of m inducing points and MAB(;)) computes attention between two sets. The time-complexity of ISAB is (nm)), scales linear with input set sizes.
METHODS
In this section, present novel extension of NPF called MPNPS. The main idea is to elicit joint predictive distributions that are constructed with equivariant neural networks instead of assuming priors for 0, and let the corresponding martingale posterior describe the functional uncertainty in the NPS. We describe how we construct a MPNP in Section 3.1 and train it in Section 3.2.
3.1 MARTINGALE POSTERIOR NEURAL PROCESSES
Recall that the functional uncertainty in a NP is encoded in a parameter 0. Rather than learning an approximate posterior ( Z.), introduce a joint predictive p( Z' /Zo épea)) generating psendo
context set 7' = {=(] N- of size (N - -cl) 2 1. Having generated a pseudo context, we combine with the existing context Zo. and construct the empirical density as
N lol
gN (z)
1 N
Σ IEc
82
(2)
+
Σ
8e (x)
(10)
Given gN, the estimate of the function parameter 0 is then recovered as
0(gN) = arg min / ())
(11)
where our case we simply choose E(2, 0) == - log (w)lua(z), (z) (x))). (ggx) is thus induced by the uncertainty in the generated pseudo context 7!.
The uncertainty in
4

---

Published as a conference paper at ICLR 2023
fenc( Pene)
fdee Pelee)
c
p (R' Rei Spred)
X e
Re
f+t.
MLP
Re
ISAB
MLP
YO
R'c
concat
concat
X
Figure 1: Concept figure of our feature generating model applied to CNP (Garnelo et al., 2018a). We first convert given context dataset Ze to the representation R. using Multi-Layer Perceptron (MLP) layers. Next we sample from a simple distribution (e.g. Gaussian). Then we generate the pseudo context representation R", using generator one layer ISAB (Lee al., 2019) in our experiment.
Amortization The procedure recovering via Eq. 11 would originally require iterative opti- mization process except for simple cases. Fortunately, in our case, we can amortize this procedure, thanks to the mechanism of CNPs amortizing the inference procedure of estimating 0 from the con- text. Given Zeo CNP learns an encoder producing 0 that is trained maximize the expected likelihood. That is,
((Z.) = Senc (Zej (enc), ((Z.) zs arg min / f(22, (()()))
(12)
where ge is the empirical density of Ze- Hence, given Z' and Ze. we can just input Z' U Ze into fene and use the output 0(Z' U Ze) as proxy for (ggv). Compared exactly computing ((gN), obtaining 8('' Zz) requires single forward pass through fene, which scales much better with N. Moreover, computation for multiple Z' required for bagging can easily be parallelized.
Specifyyng the joint predictives We construct the joint predictives p pZZ Ppee)) with a neural network. Other than the requirement of Z' being exchangeable (and thus c.i.d.), we give inductive bias to p(Z'| Ze; pped) and let the model learn Opeed from the data. We thus use the exchangeable generative model described in Section 2.4. Specifically, to generate Z', first generate & Hi from some distribution (usually chosen to be unit Gaussian (O, Ia)), and pass them through equivariant ASAB block form Z'. To model the conditioning Zes set the inducing point in the ISAB transform of Ze. That is, with an arbitrary feed-forward neural network h,
ISAB(E) MAR(E, H), H (MAR(hZe),,5)
(13)
where h(Z.) = {h(z:))ee The resulting model is an implicit generative model (Mohamed and Lakshminarayanan, 2016) in sense that we can draw samples from it but cannot evaluate likeli- hoods.
Generating Representations When 2 is low-dimensional, it would be moderately easy to learn the joint predictives, but in practice, we often encounter problems with high-dimensional 2, for instance when the input x is high-resolution image. For such cases, directly generating 2 may be harder than the original problem, severely deteriorating the overall learning procedure of MPNP. Instead, propose generate the encoded representations of The encoders of the most of the NPFs first encode an input 2; into representation ri. For the remaining of the forward pass, we only need r,s instead of the original input 2. Hence we can build joint predictives p( Re; Oprea) generating R' [r(),N- conditioned on R. {ri)iee as for generating Z' from Z.. In the experiments, we compare these two versions of MPNPS (generating Z' and generating R'), and found that the one generating R' works much better both in terms of data efficiency in training and predictive performances, even when the dimension of is not particularly large. See Fig. for our method applying to CNP model (Garnelo et al., 201 8a).
5

---

Published as a conference paper at ICLR 2023
3.2 TRAINING
With the generator p( Zei Opred), the marginal likelihood for a task 7 - (Z, c) is computed as
logp (X, Zo)
log exp
Σ
OZZ. (2)) pl p(z'I Opea)dZZ (14)
VE(x)
Note that Zei ppeed) is c.i.d., so there exists a corresponding martingale posterior ay such that
log X, Zo) log /
exp
Σ
f(z,;
N (6) Z.)d0.
(15)
kEim]
We approximate the marginal likelihood via consistent estimator,
log (Y),,, Ze) ~ log
exp
Σ SEin
l(2, 0(0e ux(0))) = -marg (T, Ø),
(16)
K
where 7²(1) (*(K) Lid.
p( Ze: Ppeed). This objective would suffice if we are given suff-
ciently good (22, U (,)),, but we have to also train the encoder to properly amortize the parameter
construction process Eq. 11. For this, use only the given context data optimize
Pegr (Y), Zo)
Σ ((z,0(Ze) SEIn]
Lamon (T,
(17)
that is, we train the parameters (deac: ddee) using CNP objective. Furthermore, we found that if we just maximize Eq. 16 and Eq. 17, the model can cheat by ignoring the generated pscudo contexts and use only the original context to build function estimates. To prevent this, we further maximize the similar CNP objectives for each generated pseudo context to encourage the model to actually make use of the generated contexts.
log Pexe (1IX,
Σ f(2,, ((z00((Z((0)) -psendo (T,
(18)
K
SE(m]
Combining these, the loss function for the MPNP is then
E, [,[[(T, 6) = E, [Cmarg (T, σ) + Lamon (T, 6) + Cpacudo (T, 6)}.
(19)
RELATED WORKS
CNP (Garnelo al., 2018a) is the first NPF model which consists of simple MLP layers as its en- coder and decoder. NP (Garnelo et al., 201 also uses MLP layers as its encoder and decoder but introduces global latent variable model functional uncertainty. Conditional Attentive Neural Process (CANP) (Kim et al., 2018) and Attentive Neural Process (ANP) (Kim et al., 2018) are the models which apply attention modules as their encoder block order to well summarize context information relevant target points. Louizos et al. (2019) proposed NPs model which employs local latent variables instead of a global latent variable by applying graph neural network. By applying convolution layers as their encoder, Gordon al. (2020) and Foong al. (2020) introduced transla- tion equivariant CNPS and NPS model, respectively. addition to these works, Bootstrapping Neural Process (BNP) (Lee et al.,, 2020) suggests modeling functional uncertainty with the bootstrap (Efron, 1992) method instead of using a single global latent variable.
5 EXPERIMENTS
We provide extensive experimental results to show how MPNP and Martingale Posterior Attentive Neural Process (MPANP) effectively increase performance upon the following baselines: CNP, NP, BNP, CANP, ANP, and Bootstrapping Attentive Neural Process (BANP). All models except determin- istic models (i.e., CNP and CANP) use the same number of samples; K 5 for the image completion task and = 10 for the others. Refer to Appendices A and C for more detailed experimental setup including model architectures, dataset and evaluation metrics.

---

Published as a conference paper at ICLR 2023
ground-truth context points
2
Figure 2: Posterior samples of MPANP for 1D regression task with RBF kernel. The black dashed line is the true function sampled from GP with RBF kernel, and the black dots are context points. We visualized decoded mean and standard deviation with colored lines and areas. (Left) MPANP posterior predictions using the combined features of the original contexts and the generated pseudo contexts. (Right) Predictions using only the generated pseudo contexts without the original contexts. The pseudo contexts decoded into reasonable functions, especially with high uncertainty for the region without context points.
Table 1: Test results for ID regression tasks on RBF, Matern, Periodic, and t-noise. 'Context' and 'Target' respectively denote context and target log-likelihood values. All values are averaged over four seeds. See Table for the task log- ooglliilld values.
RBF
Matern
Periodic
I-noise
Model
Context
Target
Context
Target
Context
Target
Context
Target
CNP
1.09619.0s
0.515
1.031
0.347.0.00
.0. 10112110.20
1..729.0.0
0.03210014
0.8161100z
NP
L.22::..05
0.498
0.94820.08
-0.26740.224
-0.668 so.on
0,201
-0.333200rs
BNP
1.112.0000
10.00t
1.057
0.418
00.006 10.001
0.705
0.009 10.022
0.6191..1
MPNP (ours)
1.189:0.00
1.123
0.205 22.020
-0.668
0.145
=0.329
CANP
1.304=0.027
0.847
1,26420.041
0.662e0.015
0.527
-0.592=0.002
0.410+0.165
-0.577=0.022
ANP
1.380=20.000 0000
0. 0.00000007
1.380=0.000
0.663=5.004
0.583+0.011
-1.019=0.025
0.836+0
-0.415=0.1a1
BANP
1.380=0.00m
0.84620.001
1.380=0.000
0.662e0.005
1.354+0.000
-0.49622.oos
0.646440.02
=0.422500000
MPANP (ours)
g0.09
0.881 10.00s
1.380
0.692
1.3481o
-0.494:0.00
0.842 touoz
-0.332 20025
5.1 1D REGRESSION
In this section, we conducted 1D regression experiments following Kim al. (2018) and Lee et al. (2020). In this experiments, the dataset curves generated from GP with 4 different settings: i) RBF kernels, ii) Matérn 5/2 kernels, iii) Periodic kernels, and iv) RBF kernels with Student's noise.
Infinite Training Dataset Previous works (Garnelo et al., 2018b; Kim et al., 2018; Le et al., 2018) assumed that there exists GP curve generator that can provide virtually infinite amount of tasks for training. We first follow this setup, training all models for 100,000 steps where a new task is generated from each training step. We compare the models by picking checkpoints achieving the lowest validation loss. Table clearly shows that our model outperforms the other models in most cases. This results show that our model well captures the functional uncertainty compared to the other methods. In Appendix B, we also report the comparison with the baselines with increased number parameters match the additional number of parameters introduced for the generator in our model, where ours still significantly outperforms the baselines.
Finite Training Dataset We also compare the models on more realistic setting assuming fi- nite amount of training tasks. Specifically, we first configured the finite training dataset consisting of (51200, 102400, 256000) examples at the start of the training, instead of generating new tasks for each training step. We then trained all models with the same 100,000 training iterations or- der to train the models with the same training budget as in the infinite training dataset situation. Fig. clearly shows that our model consistently outperforms other models in terms of the target log-liken likeliiiiiood a predictive distribution of unseen dataset from a given dataset with small number of tasks. Refer to Appendix B for more detailed results.
when the training dataset is finite. This indicates that MPNPs effectively learn
7

---

Published as a conference paper at ICLR 2023
RBF
Matem
Periodic
t-noise
target lop--loo-eeeeeee lop-likellhout
0.69
0.51
-0.66
-0.20
0.58
0.40
.0.55
0.47
0.29
-0.74
-0.90
0.36
0.18
-0.78
-1.25
x1
X2
X5
x1
X2
K5
Inf
x1
X2
x5
Inf
x1
X2
x5
Int
train
-0- CNP
-o--
- BNP
-
MPNP (ours)
RBF
Matern
Periodic
t-noise
target hopdd
0.89
0.70
-0.40
-0.22
0.05
0.66
-0.65
-0.48
0.81
0.62
-0.90
-0.74
0.77
0.58
-1.00
x1
X2
x5
x1
X2
K5
Inf
x1
X2
x5
int
x1
x2
x5
inf
deta
-E CANP
-B-
ANP
-D DANP
-0
MPANP (ours!
Figure 3: Test target log-likelihood values with varying the number of train data for 1D regression tasks on RBF, Matern, Periodic, and t-noise. Here, x-axis denotes how many examples are used for training, i... x1, x2, and x5 respectively denote 51200, 102400, and 256000 train examples.
Table 2: Test results for image completion tasks on MNIST, SVHN, and CelebA. 'Context' and 'Target' respectively denote context and target log-likelihood values, and "Task' denotes the task log- log-iiklihood, All values are averaged over four seeds.
MNIST
SVHN
CelchA
Model
Context
Target
Target
Task
Contcxt
Target
Task
CNP
16678
0.0
0. 0.706_0001
2.00
2.796 oca
2.69210.01
2.09910.011
2. 2.134
NP
0.797:2..04
0.70750:004
0,714=0000
3.04550021
2.841
2,851 20.019
2.721
2.21620.012
2.246:20.013
0.752-0.029
2.069
2.946=0.02
2.957aa/us
2.897
2.329
MPNP (ours)
10.00
0.747 Loord
0.757 _005
3.220_oor
2.980_oo::
2,992 Loams
10.07
2.369
2.407 10.200
CANP
0.5 =.0.0
0168820012
0.685=0.01
3.079aa022
3.386=8.02
3,335eiu02
2.695+0.083
2.67420.011
2.64240.011
0.793
3.996
3.365
3.405
4.08610.024
2.724
2.833
BANP
1 1.32910.cm co.ce
0.75220018
0.819_00s
4.019
3.4
3.476.0mm
4.12619.00
2.764....om
2.871 10.01
MPANP (ours)
1.361=1.coo
0.79820.006 000
0.862=0008
4.1175000
3.502:0.020
3,544:2.024
4.136+0.001
2.83320.016
2.93420.200
5.2
IMAGE COMPLETION
Next we conducted 2D image completion tasks for three different datasets, i.c., MNIST, SVHN, and CelebA. For training, we uniformly sample the number of context pixels |c| {3. ..., 197) and the number of target pixels |t] E (3...., 200 |c|} from an image. For evaluation, we uniformly sample the number of context pixels |c| {3. {3..... 197} and set all the remaining pixels the tar- gets. Table clearly demonstrates that our model outperforms the baselines over all three datasets, demonstrating the effectiveness of our method for high-dimensional image data. See Appendix B for the visualizations of completed images along with the uncertainties in terms of predictive variances, and Appendix for the detailed training setup.
5.3 BAYESIAN OPTIMIZATION
Using pre-trained models with RBF kernels in Section 5.1 Infinite Training Dataset experiments, we conducted Bayesian optimization (Brochu et al., 2010) for two benchmark functions (Gramacy and Lee, 2012; Forrester al., 2008). As performance measurement, we use best simple regret, which measures the difference between the current best value and the global optimum value. Fig. 4 depicts the normalized regret and the cumulative normalized regret averaged over 100 trials of the Gramacy and Lee (2012) function. Here, we also consider a GP variant with RBF kernel, tuned by pre- training (Wang et al., 2022). It clearly demonstrates that our model shows the best performance among NPs for both the normalized regret and the cumulative normalized regret. Appendix B.4 provides the results for the Forrester et al. (2008) function and Appendix C.4 provides detailed experimental setups.
8

---

Published as a conference paper at ICLR 2023
1.0
normalized regret
1.0
10
0.8
connalized regnet
0.8
romalized regros
nemalized
0.6
cummulattvee
10*
0.6
cummuulttee
0.4
0.4
0.2
0.0
10"
0.0
100
10
20
30
40
50
50
10
20
30
40
50
10
20
30
40
50
Iteration
Iteration
Iteration
CNP
NP
BNP
MPNPP (ours)
GP
CANP
ANP
BANP
MPANP (ours)
Figure 4: Results for Bayesian optimization on Gramacy and Lee (2012) function; we measured normalized simple regret and its cumulative value for a iteration. All models pre-trained on ID regression task generated with RBF kernel (cf. Section 5.1) ) and evaluated on the benchmark function for Bayesian optimization.
Table 3: Test results for predator-prey population regression tasks on Lotka- Volterra simulated data and real data. 'Context' and 'Target' respectively denote context and target hog-likelihood values, and 'Task' denotes the task log-likeling log-likelihood. All values are averaged over four seeds.
Simulated data
Real data
Model
Context
Target
Context
Target
Task
CNP
0.327
0.035
0.181
-2.68620.22
-3.201 20.042
=..000=0.054
NP
0.112.10.63 085
0.115g0.057
0.00010.000
2.770
3.-1441o6a
2..993_0.028
BNP
0.550=c.00
0.274=0.04
0.41740.000
-2.61420.om
-3.
-2.868=0.00
MPNP (ours)
0.626 16.04
0.375_0.035
0.500
-2.621 1o.02
33.092.0.051
22.91820.061
CANP
1.615g0.02
1.03310008
4.743...119
6.41331.33
5.801
ANP
2.607.0.015
1 1.830.0.000
2.234gm.018
1.887.0007
-4. -,8488 10-88
-1.61520.158
BANP
2.654 080
1.797=0.012
2.240+a.006
2.190=20.02
-3.597=2..77
-0.741=0.100
MPANP (ours)
2.639=0.008
1.835=0.004
2.254:5.000
1.995s0.145
-5.073aa.080
-1.690=a0.00
5.4 PREDATOR-PREY MODEL
Following Lee et al. (2020), we conducted the predator-prey population regression experiments. We first trained the models using the simulation datasets which are generated from Lotka-Volteraa model (Wilkinson, 2018) with the simulation settings followed by Lee et al. (2020). Then tested on the generated simulation test dataset and real- world dataset which is called Hudson's 's Bay hare- lynx data, As mentioned in Lee et al. (2020), the real-world dataset shows different tendency from generated simulation datasets, so In Table 3, we can see the MPNPS outperform the other baselines for the test simulation datasets but underperforms in the real-world dataset compare to other baselines. This shows that model-data mismatch is an open problem for the MPNPS.
treat this experiment model-data mismatch experiments.
6 CONCLUSION
In this paper, we proposed a novel extension of NPS by taking a new approach to model the functional uncertainty for NPs. The proposed model MPNP utilizes the martingale posterior distribution (Fong et al., 2021), where the functional uncertainty is driven from the uncertainty of future data generated from the joint predictive. We present a simple architecture satisfying the theoretical requirements of the martingale posterior, and propose training scheme to properly train it. We empirically validate MPNPS on various tasks, where our method consistently outperforms the baselines.
Limitation As presented in the Predator-Prey Model experiments in Section 5.4, our method did not significantly outperform baselines under model-data mismatch. This was also higlighted in Fong et al. (2021): model-data mismatch under the martingale posterior framework remains an open problem. Our method with direct input generation also performed poorly, as we found it difficult to prevent models from generating meaningless inputs that are ignored by the decoders. We present more details on unsuccessful attempts for direct input generation in Appendix D.

---

Published as a conference paper at ICLR 2023
Societal Impacts Our work is unlikely to bring any negative societal impacts. Modeling functional uncertainty may be related to the discussion of safe Al within the community.
Reproducibility Statement We argued our experimental details in Appendix which contains used libraries and hardwares. We presented all the dataset description in Appendix C. We describes model architecture details Appendix A.
ACKNOWLEDGEMENTS
This work was partly supported by Institute of Information & communications Technology Plan- ning & Evaluation (ITTP) grant funded by the Korea government(MSIT) (No.2019-0-00075, Arti-
ficial Intelligence Graduate School Program(KAIST)
Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea govemmen((MSIT) (No.2022-
0-00713), and Institute of Information & communications Technology Planning & Evaluation (IITP) grant funded by the Korea governmen(MMSIT) (No.2021 2022-0002202 Artificial Intelligence Innovation Hub).
REFERENCES
J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450. 2016. 12
J. Bradbury, R. Frostig, P. Hawkins, M. J. Johnson, C. Leary, D. Maclaurin, G. Necula, A. Paszke, J. VanderPlas, S. Wanderman-Milne, and Q. Zhang. JAX: composable transformations of Python+NumPy programs, 2018. URL http: / thub. com/ www./oooo//ee/oooooooooooooooo goog/e/ooglleeoooooooo /axx. 16
E. Brochu, V. M. Cora, and N. De Freitas. tutorial on bayesian optimization of expensive cost functions, with application active modeling and hierarchical reinforcement learning. arXiv preprint arviv: 10/2.20// 2010.
B. De Finetti. La prévision: lois logiques, sources subjectives. Annales de l'institut Henri Poincaré, 7(1):1-68, 1937. 4
B. Efron. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics, pages 569-593. Springer, 1992.
E. Fong, C. Holmes, and S. G. Walker. arXiv:2103.1 15671, 2021. 1,3, 4,9
Martingale posterior distributions. arXiv preprint
A. Y. K. Foong, W. P. Bruinsma, J. Gordon, Y. Dubois, J. Requeima, and R. E. Turner. Meta- learning stationary stochastic process prediction with convolutional neural processes. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020. 6
A. Forrester, A. Sobester, and Keane. Engineering design via surrogate modelling: a practical guide. Wiley, 2008. 8, 15, 18, 19, 20, 21
M. Garnelo, D. Rosenbaum, C. Maddison, T. Ramalho, D. Saxton, M. Shanahan, Y. W. Teh, D. J. Rezende, and S. M. A. Eslami. Conditional neural processes. In Proceedings of The 35th International Conference on Machine Learning (ICML 2018), 2018a. 6.
M. Gamelo, J. Schwarz, D. Rosenbaum, F. Viola, D. J. Rezende, S. M. A. Eslami, and Y. W. Teh. Neural processes. ICML Workshop on Theoretical Foundations and Applications of Deep Gener- ative Models, 2018b. 1, 3, 6,
J. Gordon, W. P. Bruinsma, A. Y. K. Foong, J. Requeima, Y. Dubois, and R. Turner. Convolutional conditional neural processes. In International Conference on Learning Representations (ICLR), 2020. 6
R. B. Gramacy and H. K. Lee. Cases for the nugget in modeling computer experiments. Statistics and Computing, 22(3):7137722, 2012. 9, 19, 20, 21
10

---

Published as a conference paper at ICLR 2023
J. Heek, A. Levskaya, A. Oliver, M. Ritter, B. Rondepierre, A. Steiner, and M. Zee. Flax: A neural network library and ecosystem for JAX, 2020. URL htLh:///:/gthh com/ google/ flax. 16
M. Hessel, D. Budden, F. Viola, M. Rosca, E. Sezener, and T. Hennigan. Optax: composable gradi- ent transformation and optimisation, in jax!, 2020. URL http: http::///gthbbb//ttehhhhp /gggtutthbb/gg//gg com/ deepmind/ optax. 16
H. Kim, A. Mnih, J. Schwarz, M. Garnelo, S. M. A. Eslami, D. Rosenbaum, and V. Oriol. Attentive neural processes. In International Conference on Learning Representations (ICLR), 2018, 3, 6, 7,
D. Kingma and J. Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015. 16
A. Le, H. Kim, M. Garnelo, D. Rosenbaum, J. Schwarz, and W. Teh. Empirical evaluation of neural process objectives. In NeurIPS workshop on Bayesian Deep Learning, page 71, 2018. 1, 7, 16
Y. LeCun, I.. Bottou, Y. Bengio, and P. Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 (1:111122222211 1998. URL http: hattp://yaae htt////annnn... lecu. com/ he//eexdmmiiise
17
J. Lee, Y. Lee, J. Kim, A. Kosiorek, S. Choi, and Y. W. Teh. Set transformer: A framework for attention-based permutation-invarian neural networks. In Proceedings of The 36th International Conference on Machine Learning (ICML 2019), 2019. 4, 5, 12, 13
J. Lee, Y. Lee, J. Kim, E. Yang, S. J. Hwang, and Y. W. Teh. Bootstrapping neural processes. In Advances in Neural Information Processing Systems 33 (NeurIPS 2020), 2020. 1, 3, 6, 7, 12
M. Lee, J. Park, Jang, C. Lee, Cho, M. Shin, and Lim. Neural bootstrapping attention for neural processes. Under Review for International Conference Learning Representations (ICLR), 2022. 1,3
Z. Liu, P. Luo, X. Wang, and X. Tang. Deep learning face attributes in the wild. In Proceedings of International Conference on Computer Vision (ICCV), 2015. URL http:/ //t////mmmm //mmmmm.....////mmm//mmmmmmmmm// ie. cuhk. edu, tttpprojects//eelleeeeeeeeee /Ceeeetst/eeebbleee htt.lebbb...........m..........b 21
C. Louizos, X. Shi, K. Schutte, and M. Welling. The functional neural process. In Advances in Neural Information Processing Systems 32 (NeurIPS 2019), 2019. 1, 6
Mohamed and B. Lakshninarayanan. Learning in implicit generative models. arxiv:1610.03433, 2016. 5
Y. Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning 2011, 2011. URL http::t//ut/iaae http:///fld..staforr....oo edweedu/hooooeee wttuu//uusnumbbrrs/// 21
stanford.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, t., Kaiser, and I. Polo- sukhin. Attention all you need. Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017. 3, 4,
Z. Wang, G. E. Dahl, K. Swersky, C. Lee, Z. Mariet, Z. Nado, J. Gilmer, J. Snoek, and Z. Ghahra- mani. Pre-trained gaussian processes for bayesian optimization. arXiv preprint 2022. 8
D. Wilkinson. Stochastic modelling for systems biology. Chapman and Hall/CRC, 2018.
M. Zaheer, Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola. Deep sets.
In Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017. 3
11

---

Published as a conference paper at ICLR 2023
Sene
Pene)
Rq
X
concat
MLP
concat
Rk
R
ISAB
R
Rk
Cross attention
MLP
Z
split
concat
X
R'
R
Self attention
MLP
X
Ye
p( R' Rei Ppred)
fdec Penc)
Figure 5: Concept figure of our feature generating model applied to CANP (Kim al., 2018). Here we sample e from a simple distribution (e.g. Gaussian). We generate key feature Ri and value feature Rg for cross attention layer which corresponding to pseudo context data. We use generator as one layer ISAB (Lee et al., 2019) in our experiment.
A MODEL ARCHITECTURES
In this section, summarize the model architectures which used in experiments. Here, we only present simplified structures for each model. To see exact computation procedures for BNPS, please refer Lee al. (2020). Fig. shows our method applying CANP model (Kim al., 2018).
A.1 MODULES
Linear Layer Lin(din, doa) denotes the linear transformation of the input with dimension d., into the output with dimension dout-
Multi-Layer Perceptron MLP(n1, din, thia, domt) denotes multi-layer perceptron with the struc- ture:
MLP((m, din dnia; dout) Lin (dhia, dom) o (ReLU o Lin(chia daia)) o ReLU a Lin(din dhia);
where ReLU denotes the element-wise Rectified Linear Unit (ReL.U) activation function.
Multi Head Attention MHA( dout) (Q, K, V) denotes multi-head attention (Vaswani et al.,
2017) with headd heads which takes input as (Q, K, V) and outputs the feature with dimension dou-- The actual computation of MHA (Thead dout) (Q, K, V) can be written as follows:
(Q) Ker
split(Lin(d,, dout (Q), Thead)
(K)) c=1 split(Lin dew)) (K), Tbpaad)
(V/)
Pheaa
split(Lin(d... dous) (V), Thead)
c=1
H = concat( [softmax (aainaxx(,KKK ( / Voau) V
LN(Q' + H)
MHA MHA( (Q, K, V) = LN(O + ReLUULin(donn (O)))
where (dgy de, d denotes the dimension of Q, K, V respectively, split and concat are the splitting and concatenating in the feature dimension respectively, and LN denotes the layer normaliza- tion (Ba et al., 2016).
Sell-Attention
SA(Theuds dua)) denotes a self-attention module which simply computed as MHAATThead doms (X, X, X).
SA AA(Teeal )
Multi-head Attention Block MAB (heaad dou) denotes a multi-head attention block module (Lee al., 2019) which is simply computed MAB MAB( (Thead, drat) (X, Y) MHA (Theady drai) )X, Y. Y).
12

---

Published as a conference paper at ICLR 2023
Induced Set Attention Block ISAB (nead dou) denotes induced set attention block (Lee et al., 2019) which constructed with two stacked MAB layers. The actual computation of ISAB ISha( (Thead: amnx) (X) can be written as follows:
H = MAB (Then dowi XY, ISAB (Theadd doat) (X, MAB (Tbaad: dowr (X, H).
A.2 CNP, NP, BNP, NEURAL BOOTSTRAPPING NEURAL PROCESS (NEUBNP) AND MPNP
Encoder
The models only with deterministic encoder (CNP, BNP, MPNP) use the following struc-
ture:
Tc c
MLP(n 5, dm duy dnid 128, dout (2:),
fenc (Ze) Te.
For the MPNP, (Ze) changes into concat([res r'I) where ra is the feature of the pseudo context data generated from generator paragraph Generator. The model also with a latent encoder (NP)
1
= |c|
MLP(n4 = 5, din = dz. daia 128, doui (2;),
iEC
(Mey log 9c)
MLP(n = 2, din = d2 daid 128, dout 128 x 2)(),,
[c]
iEC
0.1 + 0.9 softplus(logs)) sc),
he = (mm,, 1T),,
Sene (7.) = [ra:he]]
where d_ dr dyy denotes the data dimension. Data dimensions vary through tasks, d2 1, dyy
for 1D regression tasks, d₂ = 2, dy for MNIST image completion task, dN = 2, d, = 3 for SVHN and CelebA image completion tasks, and d₂ dy for Lotka Volterra task.
Adaptation Layer BNP uses additional adaptation layer to combine bootstrapped representation and the base representation. This can done with a simple linear layer
ie Linid 128, daid = d₂ + 128) (F(Pre)).
(20)
Decoder structure:
All models single MLP decoder. The models except uses the following
(n, logoo) 0) MI.P(m 3, dn d, + 128, daia 128, dout = ()conna re))) σ = 0.1 + 0.9 softplus(logr)) fa re) (H,a).
and NP uses:
(H, logo) o) MLP (nu 3, d d, + 128 x 2, daid 128, dout = ()(concat((r,ra hel)) 0.1 0.9 softplus(log Sa Yes (t)) = (1, r).
Generator
MPNP use a single ISAB module as generator. The ISAB uses the following structure:
concat([es],mr Pgg
r/ ISAB (Thead = 8, dout (2,)
fgen (r.) r'c
where e; are i..dd. sampled from Gaussian distribution with dimension 128 and ngen denotes a number of pseudo context data.
13

---

Published as a conference paper at ICLR 2023
A.3 CANP, ANP, BANP AND MPANP
Encoder The models only with a deterministic encoder (CANP, BANP, Neural Bootstrapping At- tentive Neural Process (NeuBANP) and MPANP) the following structure:
Fg MLP(m) = 5, dm = dz. daid 128, dou = (x)).
Tk MLP
(Xo),
(rr))
MLP((1 = 5, din = dz, dni = 128, dout (x8)(
To SA(Phead 8, dout (r(
Te MHA MHA(Thhand (Thead = 8, dout
fene (Ze)
Te.
For the MPANP, Senc Z.) changes into
MHA(Ahad 8, dout (r, concat([rk: r4]), concat([r,,
fene Te>
where r and r are the key and value features of the pseudo context data generated from generator paragraph Generator.
ANP constructed as:
rq MLP(nn dm dz. Ani 128, doa 1x8)
Tk MLP
(Xo);
r MLP(nn = 5, dn = d2, dhid 128, dnat (X,)
Fe SA(Nbead 8, duut = 128) (r.),
TE MHA (nhead 8, doui = (r Tk, rv);
14 MLP(n1: = 2, dnn = de, chid = 128, dout = 128 x ().),
h: SA (Mhenl = 8, duot (2))
(me: log se)
h4 iEc
0.1 + 0.9 softplus(log sc), h. (m, Tn):
Senc (Z.) = [res he].
Note that r. and F3 are from the same MLP.
Adaptation Layer Like BNP, BANP also uses adaptation layer with same structure to combine
bootstrapped representations.
Decoder All models use the same decoder structure as their non-attentive counterparts.
Generator MPANP use a single ISAB module as a generator. The ISAB uses the following struc-
ture:
(oncat([e:]] Maae
(rhs r.) ISAB (Thead = 8, dout = 25, (0na((, rvl))
fgen (rk, rw) = r where are i.i.d. sampled from Gaussian distribution with dimension 256 and Rgen denotes number pscudo context data.
B ADDITIONAL EXPERIMENTS
B. ID REGRESSION
Full results for Table We provide the full test results for ID regression tasks including context, target, and task lieeliill values in Table 4.
14

---

Published as a conference paper at ICLR 2023
Table 4: Test results for ID regression tasks on RBF, Matern, Periodic, and t-noise. 'Context' and 'Target' respectively denote context and target log-likelihood values, and "Task' denotes the task oog-likelhhod. All values are averaged over four seeds.
RBF
Matern
Perindic
Model
Contest
Torger
Taok
Cemert
Target
Taok
Conteet
Target
Tosk
Conest
Target
Toak
CNP
1,08kua oz1
85152eams
1003
0o3 0.60310.00
-0.120 ozt
00729 ees
0.361
0022oom
-081611
-02601.301
1,0022a
674422 oce
-0.26 -0.26/23
40805.n res
40
0.20
-0.333ammm
4000K:3
RNP
100
10-0 ser -0:3472 40665
-06192s
43217:2110
MPSP (ours)
1.189ua
1677
11231saa
6.7961
0,305
10.771.0311
0.145
-0.329
0.061.3002
40,502en
0.000:
1,380
0.850aaam
L,00
1.300
06663a
1.019xs.xG
0.583 xo
0.83610mm
-0.415.0.15
0.574.ax0t
BANP
1,380.0
1,085u 18866
1.300
0.662
1.018.8.00
0,496....05
0.64610022
0.270.ax03
MIPANP
1,102.0
1.390 0,6M2.9.653 1,029.6.000
1.348.a
40,494.0
0.600.0.35
0.842
-0.332 0,844.3.0
Table 5: Further comparisons with baselines with increased number of parameters. 'Context' and 'Target' respectively denote context and target log-liklihood values, and 'Task' denotes the task log- likelihood. All values are averaged over four seeds.
RBF
Matern
Model
# Params
Context
Target
Task
Context
Target
Task
CNP
264 K
1.096=0.00s
0.517=0.00
0.797
1.017#0.021
0.340
0.681=0.017
NP
274 K
1.026#0.004
0.501=0.0ms
0.752
0.948+0.005
0.334
0.640=0.008
BNP
261 K
1.11510.007
0,591 _0.005
0.843.10.00 10.006
1,051 10.007
0.41610.00
0.736_0.000
MPNP (ours)
266 K
1.189#0.005
0.675=0.003
1011=000000
1.123+0.005
0.481 +00000
0.796=0.005
CANP
868 K
1.305
0.844 00006
1.035 00.
1.27810.015
0.66320.006
0.947=0.008
ANP
877 K
1.38010.000
0.858 0.002
1.09310.00
1.380
0.66810.006
1.020
BANP
885 K
1.379+0.001
0.839=0.015
1.085
0000
1.376+t00005
0.652=0.032
1.012=0.014
MPANP (ours)
877 K
1.379x0.000
0.881 0.01000000
1.102
1.380
0.692=000003
1.029
Increasing the encoder size of baselines Since the generator increases the size of the encoder in MPNPS, one can claim that the performance gain of MPNPS may come from the increased model size. To verify this, increased the hidden dimensions of the encoder of baselines and compared them with ours. The results displayed in Table 5 further clarify that ours still outperforms the baselines when the number of parameters gets in line.
B.2 HIGH-D REGRESSION
We conducted additional experiments the synthetic high-dimensional regression data (i.e., gener- ating one-dimensional y from four-dimensional x with RBF kernel). Here we used the same model structures with the ID regression task except for the input layer, and the same settings for the RBF kernel with ID regression except for 1 ~ Unif(0.5, 3.0). We fixed the base learning rate to 0.00015 for all models throughout the nigh-dimensional regression experiments.
Table 6 clearly shows our MPNPs still outperform baselines for likelioll values we measured.
B.3 IMAGE COMPLETION
MNIST We provide some completed MNIST images in Fig. 6. It shows that both MPNP and MPANP successfully fill up the remaining parts of the image for given context and capture the uncertainties as predictive variances.
CelebA We also present five examples from the CelebA dataset in Fig. 7. It shows that MPANP provides perceptually reasonable predictions even for complex three-channel images.
B.4 BAYESIAN OPTIMIZATION
We provide the results for Bayesian optimization the Forrester et al. (2008) function in Fig. 8. Our MPNPS consistently outperform baselines as discussed in Section 5.3. We also present the visual results for Bayesian optimization in Figs. 9 and 10.
15

---

Published as a conference paper at ICLR 2023
Table Test results for 4D regression tasks on RBF. 'Context' and 'Target' respectively denote context and target log- log-likelihood values, and "Task' denotes the task log-likeliine rigllikeihhood. All values are averaged over four seeds.
RBF
Model
Context
Target
Task
CNP
0.572 003
0.265:t0.00
0.410:20.00
NP
0.568 10.0000009
0.26710.004
0.40710.007
BNP
0.621 1:0.015
0.32311.000
0.467110.03
MPNP (ours)
0.820
0.441:0.004
0.633++0.004
CANP
0.957
0.585:000006
0.743++0.005
ANP
1.357 +0.06600006
0.320
0.890=10.007
BANP
1.380
0.549910000
1.013:10.002
MPANP (ours)
1.379110.000
0.645
1.046:10.002
C EXPERIMENTAL DETAILS
We attached code in supplementary material. Our codes used python libraries JAX (Bradbury et al., 2018), Flax (Heek et al., 2020) and Optax (Hessel et al., 2020). These python libraries are available under the Apache-2.0 license
We conducted all experiments on a single NVIDIA GeForce RTX 3090 GPU, except for the image completion tasks presented in Section 5.2; used 8 TPUv3 cores supported by TPU Research Cloud for the 2D image completion task. For optimization, we used Adam (Kingma and Ba, 2015) optimizer with a cosine learning schedule. Unless specified, we selected the base learning rate 10-0.0
from a grid of (5 x 10 4.50 10 4.25 x 10 44.00 ,5 10 3.75 x 10 3.50] based on validation
task log- likelihood.
C.I EVALUATION METRIC
Following Le et al. (2018). for CNP and CANP, which are deterministic models, we used the normal- ized predictive log-likelihood Σ log [r,, Ze). J. For other models, we used a approximation of the normalized predictive log-likelihood as:
logp(u/ Ze)
log
Σ ((ull,,,000) k=1
(21)
K
i=1
where 0*'s are independent samples for E [K].
C.2 ID REGRESSION
To generate tasks (Z,c), we first sample x iid. Unif( 1aif(-2,22) 2) and generate Y using each ker- -1|z=z'1? 2/2
Matern 5/2 kernel k(a, x²) 52 71 Lod 36 and periodic kernel k(2, x') s² exp 2sin (r ia r11 where all kernels use Uni.(0.1.1.O), é Uni.(). and p (n....... To generate t-noise dataset, we use Student-t with degree of freedom 2.1 to sample noise e 7 T(2.1) where y On, 0.15). Then add the noise the curves generated from RBF kemne.. We draw index set lel ~ Unif(3, 50 - 3) and n - |c| ~ Unif(3, 50 - -cl) to maintain max [2| < 50. We use a batch size of 256 for training.
nel. We use RBF kernel k(z, x') 2² exp
C.3 IMAGE COMPLETION
We use the following datasets for image completion experiments.
httpss//www.appchh. www.:ppwww.....ccccccccccceeeeee
16

---

Published as a conference paper at ICLR 2023
Image
Context
MPNP
MPNP
MPANP
MPANP
/
0
7
4
4
9
9
9
9
9
3
3
3
3
3
2
2 2
2
2
4
9
9
5
5
Figure 6: Predicted mean and standard deviation of image pixels by trained MPNPS with MNIST dataset. The first column shows the real image from test dataset. The second column shows the context dataset which given to the models. The third and the forth columns show the predicted mean and standard deviation from the MPNP respectively. The fifth and the sixth columns show the predicted mean and standard deviation from the MPANP.
MNIST
We split MNIST (LeCun et al., 1998) train dataset into train set with 50,000 samples and validation set with 10,000 samples. We use whole 10,000 samples in test dataset as test set. We
17

---

Published as a conference paper at ICLR 2023
image
context
CANP
ANP
BANP
MPANP
Figure 7: Predicted mean of image pixels by trained CANP, ANP, BANP and MPANP model. (Column 1) Here we can see the ground truth real image from the test dataset. (Column 2) The context set which given to the models. (Column 3-6) The predicted mean of image pixels by each models.
normalized regret
tormalzed regrot
0.8
cummulative normaizeed rightt
cummutatite
normelized
0.6
101
0.6
3.0
0.4
0.2
100
0.0
10
20
30
50
50
Iteration
Iterraion
iteration
iteration
CNP
BNP
MPNP (ours)
CANP
BANP
MRANP
Figure 8: Results for Bayesian optimization on Forrester et al. (2008) function.
18

---

Published as a conference paper at ICLR 2023
CNP
MPNP (ours)
0th iteration
5th iterationn
50th iteratioo
objective function
global optimum
best findings
Gramacy and Lee (2012) function
CNP
NP
BNP
MPNP (ours)
0th iteratton
5th Iterattion
Iteration
objective function
global optimum
best findings
(b) Forrester et al. (2008) function
Figure It depicts 10 solutions predicted by CNP, NP, BNP, and MPNP. (a,b) Predicted results for Gramacy and Lee (2012) function and Forrester et al. (2008) function, respectively. (Row 1) Black circles indicate the whole initial points. (Row 2) shows the 10 best solutions predicted by each models after the 5 iterations. (Row 3) It shows the 10 best solutions predicted by each models after the whole iterations.
19

---

Published as a conference paper at ICLR 2023
CANP
MPANP (ours)
0th iteration
5th iterationn
50th iteratioo
objective function
global optimum
best findings
Gramacy and Lee (2012) function
CANP
ANP
BANP
MPANP (ours)
iteratton
5th iteraation
6.6
50th Iteration
objective function
global optimum
best findings
(b) Forrester et al. (2008) function
depicts 10 solutions predicted by CANP, ANP, BANP, and MPANP. (a.b) Predicted results for Gramacy and Lee (2012) function and Forrester et al. (2008) function, respectively. (Row 1) Black circles indicate the whole initial points. (Row 2) shows the 10 best solutions predicted by each models after the 5 iterations. (Row 3) It shows the 10 best solutions predicted by each models after the whole iterations.
Figure 10:
20

---

Published as a conference paper at ICLR 2023
make 28 28 grids which both axes starting from -0.5 0.5 to indicate the coordinate of pixels, and normalize pixel values into [-0.5, 0.5]. We use a batch size of 128 for training.
SVHN We split SVHN (Netzer et al., 2011) train dataset into train set with 58,600 samples and validation set with 14,657 samples. We use whole 26,032 samples in test dataset as test set. We make 32 x 32 grids which both axes starting from -0.5 to 0.5 to indicate the coordinate of pixels, and normalize pixel values into [-0.5, 0.5). We use a batch size of 128 for training.
CelebA We use splits of CelebA (Liu et al., 2015) dataset provided (162,770 train samples, 19,867 validation samples, 19,962 test samples). We crop 32 x 32 pixels of center of images. We make 32 32 grids which both axes starting from -0.5 0.5 to indicate the coordinate of pixels, and normalize pixel values into [-0.5, 0.5). We use a batch size of 128 for training.
C.4 BAYESIAN OPTIMIZATION
We use the following benchmark functions for Bayesian optimization experiments. Throughout the experiments, we adjust the function to have the domain of [-2.0, 2.0..
Gramacy and Lee (2012) function
sin sin(10rTx) (x 1)²,
f(z)
(22)
2r
where x E [0.5, 2.5] and a global optimum is at x² 0.5486.
Forrester et al. (2008) function
f(x) = (6z 2)² sin (ln1122 (12x - 4),
(23)
where x E [0, 1] and global optimum is at x= RJ 0.7572.
D DIRECTLY GENERATING INPUT MODEL
In this section, we present our model generating pseudo contexts directly the input space. We will present two kinds of model structure, i) directly generating pseudo context pair (z. y) simultaneously by ISAB, ii) generating pseudo context data x and y, sequentially.
D.1 CONSTRUCTION
Generating pseudo context pair simultaneously, The generator of our first model which simul- taneously generating pseudo context pair (1', takes real context dataset Z, as input and outputs pseudo context dataset Z'. Here the generator is the one layer ISAB module. Then concatenate Z. and order to treat this concatenated set as context dataset. Then the encoder takes this concatenated context set as input. And the others the same with CNP CANP.
Sequentially generating pseudo context data and y/ In this model, the generator takes real context dataset Z, as input and outputs only r's of Z'. Here the generator is the one layer ISAB module with additional one linear layer. Then consider these z's as our target dataset and find the mean and variance of y' for each x' by forwarding the model with context dataset Z. and target r'. We sample o' from the Gaussian distribution with mean and variance from the prior step. We again concatenate Z. with Z' and use them as context dataset.
Training Having directly generated pseudo context set, we construct our empirical density as
N-le
(2)
1 N
(
Σ 5z. (2) + Σ 822 [x((2))
(24)
iEc
21

---

Published as a conference paper at ICLR 2023
Table Test results for ID regression tasks on RBF. 'Context' and 'Target' respectively denote
context and target log- oggliikel values, and "Task' denotes the task log-likelihood All values are averaged over four seeds.
RBF
Model
Context
Target
Task
CNP
1.096=00.02
0.515=00001
0.796=:0.020
NP
1.022.1.000
0,498 10.003
0.748 10.004
BNP
1.112
0.58810..00
0.841 1:0003
MPNP (ours)
1.189:00000
0.675=0.003
0.000
MPNP DSI(ours)
1.120=00.007
0.55 =0.006
0.822=00007
MPNP DSE(ours)
1..007
0.555=0.000
0.824
CANP
1.304
0.047:0.005
1.036:20.020
ANP
1.380.10.00
00000
1.000
BANP
1.380
0.846=2.0001
1.088
MPANP (ours)
1.379=0.000
0.881=0.00
1.102
MPANP DSI(ours)
1.380 1.0000
10.013
1.069 10.005
MPANP DSE(ours)
1.380
0.783
1.004
Given gN. we find the function parameter as
(ggN) == arg min / ((z,)))
(25)
where we simply choose 1(2, 0) := - (ogg((yloo()))) (z) In order to train the directly generating input model, which well approximate 0(NN), we should construct different objective function from Eq. 19 because we can compute the exact f f(z, (d))),zz) unlike the feature gener- ating model. First, we approximate the marginal likelihood which is,
log p PY|), Ze)
xs log
exp
Σ iE/n]
f( 0 (Ze.U (x)) -Emarg o), (26)
K
p(Z'IZe: @pred). Eq. 26 is the same training object with Eq. 16. As we mentioned in Section 3.2, we are given sufficiently well approximated 0(Z, U Z (K)) then this objective would be suffice. However only with Eq. 26, cannot train the encoder properly amortize the parameter construction process Eq. 11. To overcome this issue, we use f {(z, 0)gN (dz) as our second training objective which is,
where z10) ..... kk) i.dd.
Σ
I ((2,0)9 (K) (dz)
Σ Σ (-e(2,
Cammo((T, (27)
2EZ.UZ'(R)
Combining these two functions, our loss function for the direct MPNP is then
E, [[([(T,)) =E, [Caag (T, Φ) + Lamont (T, Φ)].
(28)
D.2 SAMPLE
In this section, we presents how the directly generating input model actually samples the pseudo context datasets.
In Fig. 11, we report generated pseudo context datasets and posterior samples from two different cases of directly generating input models for ID regression task with RBF kernel. Here we can see that the generator samples pseudo context datasets far from the real context dataset. This phe- nomenon occurs because the generator learns to generate meaningless inputs ignored by the decoder. In Fig. 12, report how two different directly generating MPANPs predict posterior samples for ID regression task with RBF kernel. Although directly generated pseudo context dataset are a bit far from context dataset, model still well capture the functional uncertainty in this case. We report
22

---

Published as a conference paper at ICLR 2023
-1
-5.0
-2.5
0.0
2.5
5.0
-5.0
-2.5
2.5
Figure 11: It shows generated pseudo context dataset direct MPANP for ID regression task with RBF kernel. The red dots are true context points sampled from GP with RBF kernel, and the black dots are generated pseudo context points. (Left) Results from simultaneously generating pseudo context pair MPANP model. (Right) Results from sequentially generating pseudo context data MPANP model.
o
-1
-1
-2
2
Figure 12: It shows posterior samples of direct MPANP for ID regression task with RBF kernel. The black dashed line is a function sampled from with RBF kernel, and the black dots are context points. We visualized decoded mean and standard deviation with colored lines and areas. (Left) Results from simultaneously generating pseudo context pair MPANP model. (Right) Results from sequentially generating pseudo context data MPANP model.
the test results for ID regression tasks on RBF for two directly generating models in Table 7. DSI and DSE indicate simultaneously generating models and sequentially generating models, respec- tively. Table 7 shows that our directly generating models still outperform CNP and CANP in the perspective of log- likeliihodd.
23


Reviews:
[{"review_id": "tns3keD_CL", "reviewer": "Reviewer_ejFi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper introduces a novel Neural Process (NP) variant which uses martingale posterior distributions to account for epistemic uncertainty. Martingale posteriors are a recent generalization of Bayesian inference proposed by Fong et al. (2021), in which valid posteriors are specified entirely in terms of the predictive distribution function (rather than relying on the traditional prior-likelihood construction). Martingale Posterior Neural Processes (MPNPs) are compared against existing NP variants on a set of different tasks, and overall outperforming the baselines.", "summary_of_the_review": "The paper provides a novel extension of NPs with strong empirical performance based on martingale posteriors and seems like a valuable contribution to the conference.\n\nTwo possible points for improvement are 1) inclusion of higher-D tasks, and 2) further analysis of model-data misspecification. ", "strength_and_weaknesses": "## Strengths\n\n- The paper explores an interesting new direction to account for epistemic uncertainty in NPs.\n- Across a range of different tasks MPNPs show competitive empirical performance over baselines.\n- The paper may open a new avenue for further improvements of NP methods.\n- Background and prior work are clearly laid out.\n\n## Weaknesses\n\n- It would be informative to see how MPNPs scale with higher dimensionality. For example, empirical comparisons on a high-D regression task complementing the 1D one.\n- The results of the Lotka-Volterra task would deserve further analysis: Why is BNP/BANP seemingly more apt at dealing with misspecification than MPNPs? My understanding is that model data-mismatch is a problem general to Bayesian inference, i.e., should also affect B(A)NP.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "kO93cXAGQO", "reviewer": "Reviewer_eSf2", "summary": "", "questions": "", "limitations": "", "rating": 10, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper applied the technique of martingale posterior distributions from Fong et al., 2021 to neural processes (Garnelo et al., 2018). The key idea is to use NP's NLL as the loss function to construct the distribution of the parameter of interest, which, provided observations, has an interpretation of posterior given some assumptions from Fong et al. 2021.", "summary_of_the_review": "I like this paper but I'm somewhat afraid that it is not going to be well received at ICLR due to its writing style. While I hope the authors could focus more on writing as a way to convey information easily (in addition to compactly), I believe this work should be accepted as it presents an interesting and solid extension of neural processes by adopting the martingale posterior interpretation of uncertainty. This introduces new opportunities in deep learning based uncertainty estimation.\n\n\n=====After rebuttal=======\n\nI've read the rebuttal and updated the score accordingly. I appreciate the clarifications and the (ongoing) effort to make this paper more readable.", "strength_and_weaknesses": "This paper is conceptually very interesting but the writing is not on par with the idea. See the section below for more details.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jk9UaJsZPI3", "reviewer": "Reviewer_apWK", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The proposed method uses the martingale posterior technique in place of Bayesian inference for the first time to construct neural processes, i.e. stochastic processes generated using neural networks. Martingale posterior approach models the predictive distribution straight ahead, by passing the need for an interim approximate Bayesian inference step. The combination of two approaches, neural processes and martingale posterior inference, makes intuitive sense. Its implementation appears to be in order and some experiment results are strong.\n", "summary_of_the_review": "This is solid work that presents a novel and interesting idea backed with a sufficient set of experimental results.", "strength_and_weaknesses": "Strengths:\n\ni) The idea is overall interesting and novel. Application of the martingale posterior idea to neural processes is a rather straightforward and almost literal but anyway useful and suitable combination of two methodologies.\n\nii) Bayesian optimization results are strong and image completion results are OK.\n\nWeaknesses:\n\ni) The consistent outperformance of BNP/BANP over MPNP/MPANP weakens the central hypothesis of the paper. Why would one expect that the martingale posterior approach to outperform Bayesian inference in that particular setup and why was it not the case?\n\nii) The comparisons appear to be against relatively old versions of NPs. I wonder how the proposed method compares against more recent versions of NPs than ANPs (2018) and BNPs (2020), for instance Evidential Turing Processes (2022).\n\nhttps://openreview.net/forum?id=84NMXTHYe-\n\niii) I find that the adaptation of the MPNP idea to CANP a bit dilutes the main message of the paper. It is after all a heavy pipeline with many components. The contribution of martingale posterior inference to performance would be more directly measurable when one uses a simple pipeline. It is also a bit misleading to have a Fig 1 which describes some architecture details that do not have much relevance to martingale posteriors. It could be shifted to the appendix and replaced by another Fig 1, maybe a plate diagram that gives a gist of the presented inference technique.\n\niv) It is great that the paper points out the limitations of the presented method, but would be even better if it also gave an educated guess on which properties of the method cause them. For instance the model-data mismatch and the decoder input ignorance. How would the authors explain these seemingly unexpected outcomes?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%