Autoregressive Conditional Neural Processes
Authors: Wessel Bruinsma
Stratis Markou
James Requeima
Andrew Y. K. Foong
Tom Andersson
Anna Vaughan
Anthony Buonomo
Scott Hosking
Richard E Turner
Venue: ICLR 2023 poster (2023)

Abstract: Conditional neural processes (CNPs; Garnelo et al., 2018a) are attractive meta-learning models which produce well-calibrated predictions and are trainable via a simple maximum likelihood procedure. Although CNPs have many advantages, they are unable to model dependencies in their predictions. Various works propose solutions to this, but these come at the cost of either requiring approximate inference or being limited to Gaussian predictions. In this work, we instead propose to change how CNPs are deployed at test time, without any modifications to the model or training procedure. Instead of making predictions independently for every target point, we autoregressively define a joint predictive distribution using the chain rule of probability, taking inspiration from the neural autoregressive density estimator (NADE) literature. We show that this simple procedure allows factorised Gaussian CNPs to model highly dependent, non-Gaussian predictive distributions. Perhaps surprisingly, in an extensive range of tasks with synthetic and real data, we show that CNPs in autoregressive (AR) mode not only significantly outperform non-AR CNPs, but are also competitive with more sophisticated models that are significantly more computationally expensive and challenging to train. This performance is remarkable given that AR CNPs are not trained to model joint dependencies. Our work provides an example of how ideas from neural distribution estimation can benefit neural processes, and motivates research into the AR deployment of other neural process models.

Published as a conference paper at ICLR 2023
AUTOREGRESSIVE CONDITIONAL NEURAL PROCESSES
Wessel P. Bruinsma - Stratis Markou 22 James Requiema"* Andrew Y. K. Foong")
Tom R. Andersson', Anna Vaughan", Anthony Buonomo", J. Scott Hosking
Richard E. Turner
'Equal contribution
Microsoft Research AI4Science, University of Cambridge, 3 BBritish Antarctic Survey, "The Alan Turing Institute
(bbriinsm,, andrewfoong @em com [em62 6, jrr41, av555, ab2707, ret26) @cam. ac uk (tomand, jask) @bas. ac.uk
ABSTRACT
Conditional neural processes (CNPs; Garnelo et al., 2018a) are attractive meta- learning models which produce well-calibrated predictions and trainable via simple maximum likelihood procedure. Although CNPs have many advantages, they are unable model dependencies in their predictions. Various works propose solutions to this, but these come at the cost of either requiring approximations or being limited to Gaussian predictions. In this work, we instead propose change how CNPs are deployed at test time, without any modifications to the model or training procedure. Instead of making predictions independently for every target point, we autoregressively define joint predictive distribution using the chain rule of probability, taking inspiration from the neural autoregressive density estimator (NADE) literature. We show that this simple procedure allows factorised Gaussian CNPs to model highly dependent, non-Gaussian predictive distributions. Perhaps surprisingly, in extensive range of tasks with synthetic and real data, show that CNPs in autoregressive (AR) mode not only significantly outperform non-AR CNPs, but are also competitive with more sophisticated models that significantly more expensive and challenging to train. This performance is remarkable since AR CNPs not trained to model joint dependencies. Our work provides example of how ideas from neural distribution estimation can benefit neural processes, motivating research into the AR deployment of other neural process models.
INTRODUCTION
Conditional neural processes (CNPs; Garnelo et al., 2018a) are a family meta-learning models which combine the flexibility of deep learning with the uncertainty awareness of probabilistic models. They are trained to produce well-calibrated predictions via a simple maximum-likelee likelihood procedure, and naturally handle fef-the-grat and missing data. making them ideally suited for tasks in climate science and healthcare. Since their introduction, attentive (ACNP; Kim et al., 2019) and convolutional (ConyCNP: Gordon et al., 2020) variants have also been proposed. Unfortunately, existing CNPs do
Figure 1: ConvCNP trained on random sawtooth functions and applied standard mode (left) our proposed autoregressive (AR) mode (right). The black crosses denote observed data points, the blue lines show model samples, and the bottom plots show the marginal predictive distributions at the locations marked by the dashed vertical lines. In standard mode, the CNP models each output with independent Gaussian (left). However, when run AR mode, same CNP can produce coherent samples and model multimodality (right).

---

Published as a conference paper at ICLR 2023
Class
Consistent
Dependencies
Non-Gaussian
Exact Training
AR CNPs (this work)
X
CNPs (Garnelo et al., 201 88)
GNPs (Markou et al., 2022)
LNPs (Gameloo et al., 2018b)
X
Table 1: Comparison of various classes of neural processes. Shows whether a class produces consistent predic- tions, models dependencies, can produce non- Gau-Gaasiinn predictions, and can be trained without approximating the objective function. For CNPs, even though the presentation by Garnelo al. (201 8a)) assumes Gaussian predictions, it is simple to relax this Gaussianity assumption; this not the case for GNPs.
not model statistical dependencies (Figure 1; left). This harms their predictive performance and makes it impossible to draw coherent function samples, which are necessary in downstream estimation tasks (Markou et al., 2022). Various approaches have been proposed to address this. Gamelo et al. (2018b) introduced the latent neural process (LNP), which uses latent variable to induce dependencies and model non-Gaussianity However, this renders the likelihood intractable, necessitating approximate inference. Another approach is the fully convolutional Gaussian neural process (FullConv GNP; Bruinsma et al., 2021), which maintains tractability at the cost of only allowing Gaussian predictions. uses a neural network to define the mean and covariance function of predictive Gaussian process (GP) that models dependencies. However, it uses much more complex architecture and is only practically applicable to problems with one-dimensional inputs, limiting its adoption compared to the more lightweight CNP. Recently, Markou al. (2022) proposed the Gaussian neural process (GNP), which is considerably simpler but sacrifices performance relative to the FuIIConGGNP.
In this paper we propose a much simpler method for modelling dependencies with neural processes that has been largely overlooked: autoregressive (AR) sampling. AR sampling requires no changes to the architecture or training procedure. Instead, we change how the CNP is deployed at test time, extracting predictive information that would ordinarily be ignored. Instead of making predictions at all target points simultaneously, we autoregressively feed samples back into the model. AR CNPs trade the fundamental property of consistency under marginalisation and permutation, which is foundational to many neural process models, for Gon-Gassiaan and correlated predictions. In Table 1 we place AR CNPs within the framework of other neural process models. Our key contributions are:
We show that CNPs used in AR mode capture rich, non- non-Gaussian predictive distributions and produce coherent samples (Figure 1). This is remarkable, since these CNPs have Gaussian likelihoods, are not trained to model joint dependencies or non- -on-Gaussiantty, and are significantly cheaper train than LNPs and FullConvGNPs (Figure 2).
We prove that, given sufficient data and model capacity, the performance of AR CNPs is least as good as that of GNPs, which explicitly model correlations in their predictions.
Viewing AR CNPs as a type of neural density estimator (Uria et al., 2016), we highlight their connections to range of existing methods in the deep generative modelling literature.
In an extensive range of Gaussian and non-Gar non-Gaussian regression tasks, we show that AR CNPs are consistently competitive with, and often significantly outperform, all other neural process models in terms of predictive log-likelihood.
We deploy AR CNPs on a range of tasks involving worl-world climate data. To handle the high-resolution data in a computationally tractable manner, we introduce a novel multi- scale architecture for ConyCNPs. We also combine AR ConvCNPs with a beta-categorical mixture likelihood, producing strong results compared to other neural processes.
Our work represents a promising first application of this procedure to the simplest class of neural processes, and motivates future work on applications of AR sampling other neural process models.
2 AUTOREGRESSIVE CONDITIONAL. NEURAL PROCESSES
Meta-learning, We first define the problem setup. Let X be compact input space and let ) be the output space. Let Dy = (X' x y) be the collection of all sets of N input-output pairs, and let D = U DN We call elements D E 'D data sets and denote D = (x,y) where x E X y E yv are the inputs and outputs respectively. In meta-learning we are given a collection of data sets (D.,. (Dm)) called mela-data set, with the individual data sets D,, called tasks (Vinyals et al., 2016). Every task Dm is split up Dm = D U D into a context set Die1 = (x/2. y!n) and targer set D (x()) yl)) Here x(c) are called the context inputs, y! the context outputs, x(I the target inputs, and ylll the target outputs. Our goal is to devise an algorithm which takes in a context set Die) and produces the best possible prediction for the target outputs y (0 given target inputs x!!)
2

---

Published as a conference paper at ICLR 2023
Ground Truth (Top) AR Predictions Left Right (Bottom)
2
X
NLL / /ddapoont
PulCCorUUP CamLNP
DunviNe CuucUPP
y0X
Densley
-1-
10P
10 Wallelock training time (min.)
10
109
y(r,)
ylxz)
yixa)
yixi)
True
Ideal CNP
AR
Figure 2: Negative log-likelihoods on non-Gaussian
Figure 3: Top: generative process: mixture model of three deterministic functions with additive Gaussian noise. Bot- tom: the four taget locations indicated by dashed lines, the panes show the true distribution and predictions by the ideal CNP and the ideal CNP applied autoregressively at
sawtonth data. Deploying the ConvCNP in AR mode
dramatically improves performance, and outperforms
state-of-the-art NPs with Gaussian (FullConvGNP)
and non Gau-Gausiian (ConvLNP) predictive distribu- tions, a fraction of the training cost.
the targets from left to right. See Appendices D.3 and E.
Neural processes. Let P be the set of all y-valued stochastic processes X. Neural processes (NPs) directly and flexibly parametrise a map xp: D - Q where Q S P and where 0 are learnable parameters. CNPs set Q to be the collection of GPs such that (x)) f (y)) = 0 for # y- GNPs let Q be the collection of continuous GPs. Latent NPs (LNPs; Garnelo et al., 2018b) let Q be collection of non-Gaussian processes by making use of latent variable. Let Px (Die) denote the finite- dineesiiin distribution of the process π (D()) evaluated at inputs x and denote its density by go( x0) ) To learn the parameters 0, NPs seek to maximise
(T) = + ΣM log go (y x D
(1)
For CNPs and GNPs, LM can be computed exactly, since do is Gaussian.', However, for LNPs, LM must be approximated (Garnelo et al., 201 8b; Foong et al., 2020), typically impacting performance.
Autoregressive CNPs. Our proposal is to take an existing CNP and run it in an autoregressive fashion, feeding predictions for earlier outputs back into the model. Inspired by the product rule, product of conditionals, modelling each conditional with CNP.
we define the joint predictive For example, in the case of three target points, qe (v3 y Díe) ) y/ D (e)))(( (0 D(e). To enable theoretical analysis of this procedure, we now proceed to up more formal notation. Suppose that गe. D + Q is an NP, and we wish to predict at some target inputs x!" given a context set O Standard NPs would output the predictive TO ((o)) which, for CNPs, would be factorised Gaussian. We propose to instead roll out the NP autoregressively. as described in Proc. 2.1.
Procedure 2.1 (Autoregressive application of neural process). For neural process ne, context set D = (x19), (()), and target inputs x let AR (Te, ( D(e) be the distribution defined as follows:
for = 1,...,
(1) P. PT (xe()) + x ()(--1))> y(() ① (0 yi-(i-1))
(I)
(2)
where a e b concatenates two vectors a and b. See Figure 7 in Appendix C for an illustration.
Since earlier samples ( feed back into later applications of ne, the whole sample y is correlated, even if no does not model dependencies between target outputs, as with CNPs. At test time, when evaluating the corresponding the density 96
(AR) of AR (To, D(e)) at y we use the formula
(AR)) (y 1 (()) EN log gg
i=1 log (v (t) (0
D @ (x (x v1
(3)
Whilst any NP can be used in AR, we focus on CNPs as they are the computationally cheapest class.
Understanding the infinite data limit. To better understand why AR CNPs successfully model dependencies, we analyse the idealised case of infinite data and model capacity. Let p(f) be the law of the gettaaenerating stochastic process, and let p(=) be the law of stochastic process representing observation noise, defined by letting exx) be vector of i.i.d. noise variables for all x. We assume
y ym (x!x) and y
(x!!) where ym((-) = fm(() (+) + E.)
(4)
(fm M= are i.i.d. draws from p(f)) and (em ( are i.i.d. draws from p(s). Define the prediction
map ny: D + P as the mapping from data set the posterior over y, Ty (D) p(y D). Then LM is a Monte Carlo approximation of the following infinitessapll objective (Foong et al., 2020):
Lxx(m) (T) -Bp Ppp(""ppp"" [KL (P 7 (D(o),, Px (D( + const.
(5)
Unless otherwise specified, assume CNPs use Gaussian likelihoods, as in Gamelo et al. (2018a). However, it is straightforward to madify them use con-Gaussian likelihoods, as we do in Section 4.4.
3

---

Published as a conference paper at ICLR 2023
Under appropriate regularity assumptions, (T) is maximised over all x when the expected KI. divergence term is zero, which occurs and only if π = Ty- In practice, NPs do not maximise oo over all n,, but (i) use a finite-sized meta-data set and (ii) restrict - P:
what we compute in practice
ideal NP
exast prodiction map
arg max C D-Q
arg max Lo D-+Q
(ii
Ty
arg max Coo (π) D-+P
(6)
M 0000
Q-P
Here TM is an NP trained on the practical objective (1), which, in the limit of infinite data, approxi- mates the so-called ideal NP Tec- The ideal NP depends on the choice of Q, iee the class of NPs under consideration, and, in turn, approximates xy- For CNPs and GNPs, using the fact that minimising KL(p, q) over q matches moments (Minka, 2001), we can readily determine and even practically compute the ideal NP for these two classes of NPs. The ideal CNP predicts a diagonal-covarianee GP whose mean function and marginal variance function match Tyi x (D) = GP k) where m(x)= = Bzy(z)) D), and k(z, z') = V[y(z) D] if x = x' and k(z, z') = O otherwise. On the other hand, the ideal GNP predicts a GP whose mean function and full covariance function match my: Tx (D) (m,, k) where m(x) = E[y(1:) D). k(z, x²) cov (y(x), ((z') D). The main result
of this subsection is that the ideal CNP, despite not modelling correlations, becomes superior to the
ideal GNP when deployed in AR mode:
Proposition 2.1 (Advantage of AR CNPs over GNPs). Assume appropriate regularity conditions on y. Let TC be the ideal CNP and let Te be the ideal GNP. Then, for all inputs x and data sets D E D,
(L)) (TC D)) & (L(PxTy(D), (D)
(7)
We provide proof in Appendix A. Intuitively, the advantage of AR CNPs comes from their ability to
model non-Gaussian dependencies. Proposition 2.1 shows that to outperform the GNP, suffices to
train CNP to model the marginals of Ty. and rely the AR procedure to induce dependencies. A
visualisation of the ideal CNP and the ideal CNP applied autoregressively can be seen in Figure 3.
Consistency and the AR design space. As shown in Table 1, AR CNPs give up the fundamental property of consistency. since the distributions {ARxx (To, D(o) : x E X N E N} are not consistent under permutation marginalisation: permuting and introducing or marginalising target points can change the distribution. This violates the conditions of the Kolmogorov extension theorem (Oksendal, 2013), preventing the distributions from defining consistent stochastic process. There is thus large design space involved when deploying AR CNPs, where choices that have no effect on the predictions of other NPs can now significantly affect performance.
One such choice is how many points to sample at a time. Sampling one at time induces dependencies between all points, but requires N forward passes. Alternatively, we could divide the N inputs in x(") into blocks of K points each, and sample each block with a single CNP forward pass. This requires N/K forward passes, with points the same block conditionally independent. If K- N. this is the standard CNP prediction; and if K = 1, we recover Procedure 2.1. This provides knob for practitioners to trade off between faster, consistent, but less expressive standard CNP predictions; and slower, less consistent, but more expressive AR predictions. In this paper, we use full AR mode with K = 1, and leave an investigation of block AR sampling to future work.
Obtaining smooth samples. Due to the lack of consistency in AR mode, the spacing chosen between target points can significantly affect performance. For example, care must be taken so the number of target points is not much greater than the size of the context sets seen during train time, to avoid confronting the model with an out-of-distribution context set size at test time. This raises the question of how to sample functions on a very fine grid. Furthermore, since CNPs do not differentiate between epistemic and aleatoric uncertainty, it is not clear how to obtain smooth, noiseless samples, that is, samples for f uncorrupted by the i.i.d. noise in (4). The following proposition shows that, for a smooth sample corrupted by additive noise, the smooth component can be approximated with the predictive mean conditioned on noisy observations:
Proposition 2.2 (Recovery of smooth samples). Let X S R be compact, and let be a stochastic process with surely continuous sample paths and sup x of (2))lrz < co. Let (En) be i.i.d. (potentially non- -on-Gaussian) random variables such that B[Eu] 0 and V(Eul < co. Consider any sequence (In)n21 C X. and let xx E X be limit point of (Im))21 If v(x²) = f(x*) + so and Un = fxx) + En are noisy observations of f, then
lim Ely(x*) /......,, v.ol = sxx) almost surely. 20000
(8)
4

---

Published as a conference paper at ICLR 2023
ConVCNP (AR)
Ground Truth
ConvCNP (AR)
Ground Truth
Figure 4: Comparison of noiseless (left) and noisy (right) samples from AR ConyCNP trained on data sampled from GP with an expoeeniatedd-quadee kernel, the ground truth noiseless samples were generated from the noisy AR samples using the procedure suggested by Proposition 2.2.
We provide proof in Appendix B. Equation (8) suggests the following two-step procedure for obtaining smooth samples from AR CNPs. Step 1: Let Xim be a number of target inputs that does not exceed the number of points seen during training. Sample Y1:a ARx... (To, D This sample includes observation noise. Step 2: Remove the noise from the sample by passing through the model once more: N D) Px To (D @ (X1:a, M1:n)). Here the predictive mean Mi forms the noiseless sample. To produce sample at arbitrarily many inputs, one may also evaluate N( i D) Px. To (
(x1:n: Vim))) ( where x1.n is arbitrary. This result of this procedure is illustrated in Figure 4, and was used to generate the noiseless samples shown in Figure (right).
Figure in Appendix C also illustrates this two-step procedure in a pictorial step-by-step fashion.
3 CONNECTIONS TO OTHER NEURAL DISTRIBUTION ESTIMATORS
Various paradigms have been developed for neural distribution estimators (NDEs): normalising flows (Dinh et al., 2015), generative adversarial networks (GANs; Goodfellow et al., 2014), variational autoencoders (VAFs; Kingma & Welling, 2014), autoregressive models (Uria et al., 2016), and diffusion models (Sohl-Dickstein al., 2015; Ho al., 2020). Figure visualises the landscape of NDEs. We argue that NPs and AR CNPs should be viewed as neural distribution estimators (NDEs) and be placed in this landscape. AR CNPs inherit the strengths model complex dependencies with tractable likelihood, but also some of their weaknesses, most notably slow test-time sampling. Slow sampling is the main draw back of AR CNPs, though
AR models, such as the ability to
Noaral promosses
Nremolising flooss
17, 18
Oeneraivo savrrriii cotmorks
Virtationdl Eot Enbertccdoo serttrr
Autorgrosstre moslas
Uncoentabble
28. 29
Counuble
9-15
30
401-2
Cons
21
52-57
Attention
22-27
48-51
Faite MLP
20
31, 32
43
Neural processes
17.
BRUNO (Kosshnooa et cl., 2020)
33. Can. VAE (Salinaus et all, 2015)
50.
DEComer (Alcom & Nguyes.
(Gumelo 201
18. Copula GNP (Markons al. 202))
34. VIVAE (Child 2020)
2u21)
GNP (Markou et al., 2012)
FlouGAN
GP VAE (Foctuio 2020)
(Yang et 2019)
3. ACNP (Kin at et., 2019)
et
52.
ComNADE (Uria al., 2016)
Normalising Bows
FallCaacCNNP (Bouinena
Com (Mariooa al., 2022
2023)
CNP Gordon al. (2020)
21.
23.
24.
22.
20. GAN (Goodiellow
WGAN (Arjoskly
DOGAN (Radiocd 201,)
SAGAN (Zhan 2019)
IaEGAAN (Chea
2017)
37. (Ganeeoo 201
42.
39.
38.
VLAB (Chen et a0,,
Cand (Foong ct al., 2020)
Al. (Kim et 2019)
(Oord
(Gulrajani 2017)
54.
56
WanNNe
PacKNNN+ (Salimans
Fae PaclNN- (Ramschandmn
2017)
PAscCNN (Oord et 20166)
2016x)
NICE (Dinh
(Miyato al..
2017)
sl., 2019)
2A. BigGAN (Brock et 2019)
Anterggreeise moddl
Net
ReaINVP (Dinh et 2017)
27. StyleGAN (Karas at. 2019)
RNADE
Kalchbermer. 2019)
10. FFOORD (Godhwold at al., 2010)
Spatial GAN (Jeicee et 2016)
44.
NADE (Uria ai., 2016)
58.
ARDM (Hongehoom 2021)
11. Glow (Kingna& Dluxival, 2018)
2020)
45.
DeepNADE (Una et al., 2014)
work)
&ResNets (Belumana
30.
PresGAN (Dieag atal. 201,) 2019)
46.
EaNADE (Unia 2014)
60.
AR ACNP this work)
15. NAF (Hunug et al., 2018)
16. TGP (Maredi et 2001)
IAF (Kingns 2016)
Res
31.
Variational autoencoders
32.
VAEs (Kingma & Welling, 2014)
IWAF. (Runcre al. 20161
49.
48.
Spanse Transfomer (Child et al..
3019)
PieeSNNIL (Chan
62. AR ComCNNP (lais work)
Grover, 2022)
(Npayen
Figure 5: Conceptual diagram showing the relationships between AR CNPs and various neural distribution estimators. The vertical axis denotes whether the model learns a distribution over finite number of random variables. countably infinite number, or an uncountably infinite number. The axis into page denotes whether the architecture is MLP-based, or uses attention or convolutions. From left to right, we show different modelling paradigms. Fruitful exchanges occur when NPs (highlighted in green) are introduced into other modelling paradigms. Our proposed AR CNPs can be viewed as introducing NPs to the AR modelling paradigm.
5

---

Published as a conference paper at ICLR 2023
it may be possible adapt techniques for speeding up AR models (Ramachandran al., 2017). One major difference between AR CNPs and existing AR models is that AR CNPs decompose the joint distribution of an uncountably infinite set of variables, allowing querying at arbitrary input locations (Section 2). Like DEformer (Alcorn & Nguyen, 2021). EoNADE (Uria et al., 2014), and XLnet (Yang et al., 2019), AR CNPs are trained to not prefer any particular order of decomposing the joint distribution into conditionals. To achieve this goal, the AR CNP shares design choices with other AR models: (i) a shared architecture is used to produce each conditional distribution, similar to WaveNet (Oord et al., 201 6a) and PixelCNN (Oord et al., 2016b); (ii) the data point index is given as input to the network as in the DEformer model (Alcorn & Nguyen, 2021); and (iii) training maximises a log- lieeeiiol including all decompositions of the joint distribution, similar to EoNADE (Uria et al., 2014) and XLnet (Yang et al., 2019).
Figure 5 also shows the connections between NPs, VAEs and normalising flows (NFs). Like VAEs, LNPs use decoders that parametrise a factorised distribution and rely on the latent variable to induce dependencies. Again, the key difference is that LNPs model a distribution over uncountable set of variables. Models like conditional BRUNO (Korshunova et al., 2020) and copula GNPs (Markou al., 2022) combine ideas from NPs and NFs, transforming a stochastic process with an invertible transformation. Finally, GAN models such as Spatial GAN (Jetchev et al., 2016) and cA (Lu et al., 2020) model countable numbers of variables, such as images of arbitrary size. Inspecting Figure 5, we see that GANs are the only class of models depicted that do not currently have an NP version: a version that models an uncountable number of variables. This suggests adversarial training of NPs as an interesting avenue for future investigation.
In recent work, Nguyen & Grover (2022) proposed the Transformer NP (TNPs), which uses a causally- masked transformer architecture with an autoregressive likelihood. In contrast, rather than proposing a new AR architecture, our work focuses on running existing CNPs in AR mode to obtain coherent samples and improved likelihoods, without modifying the architecture or training procedure. In prior work, Volpp et al. (2021) used AR sampling in order to visualise samples from CNPs. However, their work focuses on proposing a novel context aggregation mechanism for NPs, and they do not evaluate the likelihood of CNPs autoregressively or investigate any performance gains.
4 THE PERFORMANCE OF AUTOREGRESSIVE NEURAL PROCESSES
In this section we investigate the performance of AR CNPs on synthetic and real data. Across a wide range of tasks, the AR CNP is competitive with much more sophisticated approaches. Throughout, we train LNPs with both the ELBO (Garnelo et al., 2018b) and ML objective (Foong et al., 2020). Code for implementations of NPs and reproducing our experiments can be found at https: //gittub. cww/wesse wwwssws/beeeee.eee wpw.ersssssccesss....moee For all experiments, we use a random ordering of the target points in Proc. 2.1; see App. D for a justification.
4.1 SYNTHETICALLY GENERATED GAUSSIAN AND NON-GAUSSIAN DATA
Synthetic experiment setup. We evaluate an extensive collection of NP models on a wide range of Gaussian and non-Gaussian synthetic regression tasks. We consider tasks with functions drawn from (i) different GPs; (ii) non-Gaussian sawtooth process (as in Figure 1); (iii) non-Gaussian mixture task, where, with equal probability, we sample the function from one of three possible GPs or the sawtooth process. We also consider various versions of the tasks for different input and output dimension d,, dy. with dependencies across the output channels. To ensure a fair comparison, we configure the architectures to make the parameter counts comparable between all models.
Results. Table 2 highlights the best performing models on some representative tasks; for further results across all twenty synthetic tasks and further experimental details, see Appendix H. The AR procedure dramatically improves the performance of the ConvCNP, with the AR ConyCNP being the best performing model for most tasks, except the Gaussian EQ task where it performs marginally worse than the FullConyGNP. In particular, the AR ConvCNP outperforms the FullConvGNP and ConvGNP on non- Non-Gaassian tasks, in agreement with Proposition 2.1, while having a faster training time than the other convolutional models (Figure 2). For the sawtooth task, Figure 11 in Appendix H.2 illustrates that predictions by the AR ConvCNP can be multi-modal and non-Gaussian, even when us- ing a Gaussian likelihood. Finally, we note that in tasks with d, = 2, where the FullConvGNP cannot be used (as discussed in Section 1), the AR ConvCNP far outperforms all competing approaches.
4.2 SIM-TO- -IM-TO-REAL TRANSFER WITH THE LOTKA- VOLTERRA EQUATIONS
Predator-prey data. We next investigate sim-to-real transfer, where the models are trained on simulated data and tested on real data. NPs are well-suited to this setting, since large meta-data set

---

Published as a conference paper at ICLR 2023
EQ
Sawtooth
Mixture
Norm, KL to truth (1 better)
Norm. log-lik. lik. (f better)
Norm. log-lik, rig-lik. (f better)
da, dy
d2, dy =2
dz, dy =
dz, dg =2
d2, dy =1
d2, dy =2
ConyCNP
0.41 #0.01
0.41 +0.00
+.304
0.12 #0.01
-0.23+0004
-0.85 +0.01
ConvCNP (AR)
0.01 #0.00
0.03 #0.00
3.60=z.01
0.38 #0.00
0.45=0004
-0.62 20.01
ConvGNP
0.01 =0.00
0.19 +0.00
2.62+5.005
0026:
-0.24
-0.74 10.01
FullConVGNP
0.00 +0.00
2.16::0.04
-0.05
Co (ML)
0.25 #0.01
0.39 +0.00
3.06:20.0
0.31 #0.01
-0.06
-0.78 #.002
ConvLNP (ELBO)
0.06 10.00
0.79 10000
3.51
0.04 +.000
0.12
-0.92 #:001
Diagonal GP
0.40 #0.01
0.40 =0.00
Trivial
1.19 #0.00
0.79 4.000
-0.18+0.00
-0.32 #:0.0
-1.32
-1.46 4:000
Table 2: Performance of NPs training on the GP EQ task, sawtooth task, and mixture task. Diagonal GP denotes the exact GP predictive, but with correlations removed. Trivial denotes a model that predicts a Gaussian distribution with the empirical mean and standard deviation of the context outputs. Significantly best models in bold. Note that the FullConvGNP cannot be run tasks where de
Int. (S)
For. (S)
Rec. (S)
Int. (R)
For. (R)
Rec. (R)
ConvCNP
-3.47 +0.02
-4.06
40.02 -4.85
40.02
-4.17 00.04
-4.70 +0.06
-4.97 40.01
ConvCNP (AR)
-3.30 20.02
-3.47
=0.02 -3.60
20:02
-4.10
-4.27 20.03
-4.32
10.01
ConvGNP
-3.47
-3.65
10.02
-4.15
=0.05
-4.82
#0.13
-4.61
40.01
FuIIConvGNP
-3.29 20.02
-3.46
20.02
-4.16-0.04
-4.28
20.04
-4.45
Co (ML)
-3.41 +0.02
-3.84
t0.02
-4.13
+0.04
-4.45
+0.05
-4.54
ConvLNP (ELBO) -3.77 #0.02 -3.83 #0.02 -4.12
40.02
-5.45 =0.05
-5.47
40.07
-6.39 #0.00
Table Normalised liglliiels in the predator-prt prey experiments. showing interpolation (int.). forecasting (for.), and reconstruction (rec.) on simulated (S) and real (R) data. Significantly best results in hold.
can be easily generated to train them. We consider the hare-lynx data set, which is a population time series of Snowshoe hares and Canadian lynx (MacLulich, 1937). To generate simulated data, use a stochastic version of the Lotka-Volterra equations (Lotka, 1910; Volterra, 1926):
dX, a.x, dt ,Y,X, dt + ox, dw!"), dY, X, dt + 8Y,XX, dt + oY, dw (9) Under these equations, the prey population X, grows exponentially with rate a, the predator pop- ulation Y2 decays exponentially with rate y, and the predators hunt the prey. W()) and (2) are independent Brownian motions introducing noisy behaviour. These equations generate non-Gaussian data with both within-channe as well as cross-channel dependencies. We simulate the Voltera equations dense grid, and use them to generate meta data sets in three different ways. Interpo- lation: we randomly divide the data into context and target sets. Forecasting: we choose a random time, before which all data are contexts, and all future data are targets. Reconstruction: we randomly choose between the X, or Y, split the chosen series as in forecasting, and append the other series to the context. In training, for every batch, we choose one of these tasks uniformly at random.
Results. Table 3 shows the results of the best performing models. The AR ConvCNP performs best both on the simulated as well as the real data, again demonstrating that running CNPs in AR mode improves performance and can even outperform strong GNP and LNP baselines. For full experimental details and additional results see Appendix I.
4.3 ELECTROENCEPPIALOGR.. EXPERIMENTS
Electroenceepaloggran data. We next trained various NPs on real time series data consisting of electroencephalogrann (EEG) measurements (Zhang et al., 1995), following Markou et al. (2022). Each time series consists of 256 regularly spaced measurements across 7 correlated channels. For each channel, we randomly select number of the 256 points uniformly at random to be target points, and use the remaining ones context points, independently across the channels.
Results. After training, we test the models on this interpolation task and also on a reconstruction task, where we set part of a channel target and the remainder as context. In Table 4, we observe that the AR ConvCNP is competitive with the FullConvGNP, despite having significantly shorter training times and fewer parameters. Both the AR ConvCNP and the FullConvGNP outperform the ConyCNP and the ConyLNP. Full experimental detail are in Appendix
4.4 ENVIRONMENTAL MODELLING
Environmental datasets bring a range of modelling challenges. One example is fusing spatio- temporal data from disparate sources (Chang & Bai, 2018; Lahat et al., 2015), which arises in diverse
7

---

Published as a conference paper at ICLR 2023
ConvCNP
ConvCNP (AR)
ConvGNP
FullConvGNP
ConvLNP (ML)
ConvLNP (ELBO)
Int.
-1.02+0.01
0.34=oo0
-0.93+0.01
-0.35+0.01
-1.04+0.01
-1.20+0.00
Rec.
-0.07
-0.340000
-1.45=0.03
-0.5740.01
-1.53+0.02
-2.00 06
Table 4: Normalised lig-likelhhoods on the EEG experiments. Significantly best results in bold.
GAUSSIAN
BETA- BETA-CATEGORICAL
ConvGNP
ConvLNP (ML)
ConcCNP
ConvCNP (AR)
ConvLNP (ML)
ConvCNP
ConvCNP (AR)
Log-lik.
0.00+0.02
0.62+0.02
0.58+0.02
0.88+0.02
1.06+a.22
1.03=0.02
1.27
MAE (9)
13
12
13.01
13.01
12.99
13.13=0.6
13.13
Table 5: Normalised log- likkeiihoodds and mean absolute errors (MAE, in units of cloud cover %), over the
2019-2019 test period for the cloud cover task. Note that log- likelihoodss cannot be compared directly across the Gaussian and beta-categorical models. Errors indicate standard errors. Significantly best results in bold.
environmental sciences applications from climate monitoring to hydrology (Gettelman et al., 2022: Ferrer- Cid et al., 2020: Robinson et al., 2021: Lu et al., 2010; Hosseini & Kerachian, 2017). Another challenge involves estimating the probability of events of interest, such as the compound risk of both low wind speeds at an offshore wind farm and high cloud cover over a solar panel farm. To obtain robust uncertainty estimates for such events, is essential to model correlations as well as non- Gaussian marginals (such as cloud cover). Current GAN-based approaches (e.g. Ravuri et al. 2021) can capture both joint and non-Gaussian statistics, but they cannot perform data fusion or predict at arbitrary off-grid locations. The AR ConvCNP can fuse data of on-grid and off-grid modalities and make predictions at arbitrary locations while modelling arbitrary non- -au-sassian likelihoods and capturing statistical dependencies, thus achieving all the desiderata and filling a gap in the environmental modelling toolbox. Here, we assess the AR ConvCNP on two common environmental modelling tasks, namely data assimilation and statistical downscaling.
Data assimilation. Data assimilation is the task of combining observations of the Earth system to produce predictions on regular grid, called a reanalysis. Reanalyses are typically generated by fitting the trajectories of physics-basee based climate models to observations (Hersbach et al., 2020; Gettelman et al., 2022), but the potential for improving data assimilation with MI. has drawn increasing attention in recent years (Geer, 2021). To explore the AR ConvCNP's data assimilation abilities for non- Gaussian variable, we train convolutional NP models to predict simulated daily-average cloud cover fraction over Antarctica. We use reanalysis data from ECMWF ERA5 (Hersbach et al., 2020) as ground truth. Cloud cover takes values in the interval [0, 1). with observations frequently taking values of 0 or (Figure 14). We evaluate the performance of NPs using either Gaussian likelihood or a veta-categorical mixture model with three components: two discrete delta components for values of exactly 0 or 1, and beta distribution capturing continuous values in (0, 1). This provides a robust way of handling O and 1 values, unlike the existing copula GNP model (Markou et al., 2022) which can have its output constrained in (0, 1) but places zero density at the endpoints.
Data assimilation results. In Table 5 we see that the AR ConvCNP significantly outperforms competing NPs for both the Gaussian and beta-categorical likelihoods. Figure 6 shows samples drawn from the models, after observing context points on half of the space. The AR ConvCNP displays remarkable ability to extrapolate rich, non-stationary, multi-scale structure, such as sudden changes in cloud cover over the Ross Ice Shelf coastline at the bottom of the figure. By comparison, the ConvLNP and ConvGNP produce blurry, lower frequency samples. Unlike GPs, convolutional NP models have fixed receptive field induced by the CNN architecture used for the encoder, which is computationally expensive to increase. Away from the context points on the left, samples from the non-AR models will be independent of the observations, reverting some mean representation of the data (Fig. 6c-e). This highlights a further benefit of AR CNPs: successive AR applications increase the receptive field, enabling rich, conditional sample structure to extrapolate far away from observed data. See Appendix K for further commentary, sample figures, and details.
Environmental downscaling. The spatial resolutions of physics- based reanalyses are limited by their vast computational demands, making them unsuitable for capturing local and extreme events (Stocker et al., 2013; Maraun al., 2017). Statistical downscaling addresses this issue by leveraging additional information to produce fine -raiind predictions (Maraun & Widmann, 2018). Recently, NPs have been shown to outperform a large ensemble of existing climate downscaling approaches (Vaughan et al., 2022). We compare the AR ConvCNP to the MLP ConvCNP of Vaughan et al. and the MLP ConvGNP of (Markou et al., 2022) in a temperature downscaling task over Germany. In this task, the context data consist of low-resolution ERA-Interin reanalysis data and nigh-resolution topography, and the target data consist of weather station observations from the ECA&D dataset. We also consider a second setup where we reveal some station observations aid the downscaling
8

---

Published as a conference paper at ICLR 2023
Ground truth
AR ConvCNP sample ConvCNP sample
ConvLNP sample
ConvGNP sample
Cloud oover fraction
0.75
0.50
0.25
0.00
Figure 6: (a) Ground truth simulated cloud cover fraction on 25/06/2018. (b-e), Sample draws from the AR ConyCNP, ConvCNP, ConvLNP and ConvGNP with context points denoted by red dots. Context points were removed from the right hand side of the 2D space to test the models' abilities to extrapolate coherent function samples far away from observations. The ConvCNP and ConvLNP models used beta-categorical likelihood while the ConvGNP uses a low-rank Gaussian likelihood.
process. As Appendix L.2 explains, the MLP ConyCNP and MLP ConvGNP cannot be extended to include these station observations. We therefore introduce a novel multiscale architecture, which we use to run the ConvCNP in AR mode. See Appendix L for full experimental details.
Environmental downscaling results. In Table we observe that the AR ConvCNP matches the performance of the ConyGNP, which is remarkable as the latter has been previously demonstrated to outperform range of state-of-the-arr downscaling approaches (Markou et al., 2022: Vaughan et al., 2022). When additional observations from weather stations are revealed, the AR ConvCNP significantly outperforms the MLP ConvGNP in both metrics.
Downscaling
Norm. log-lik.
MAB
Down, + stations
Norm. log-lik.
MAE ("C)
ConyCNP (MLP)
-1.55 40.01
0.94 40.03
ConvCNP" (MLP)
-1.55 40.01
0.94 #0.03
ConvGNP (MLP)
-1.36
1.09 #0.09
ConvGNP" (MLP)
-1.38 40.01
1.09 =0.09
ConvCNP (AR)
-1.36 +0.01
1.04 #0.04
ConvCNP (AR)
-1.31 +0.01
0.85 400005
Table 6: Normalised log- leeliikelioodss and mean absolute errors (MAEs) in the downscaling experiments, without (leff) and with (right) assisting weather station observations. Significantly best results in bold. "Cannot use extra weather station observations.
LIMITATIONS AND CONCLUSION
We have shown that the AR procedure can be readily applied improve the performance of CNPs, producing coherent samples and dramatically improved likelihoods. Surprisingly, in an extensive range of experiments, this simple approach often outperforms more complicated methods which rely on latent variables or which explicitly model correlations. We demonstrate the effectiveness of our approach on data sets of real-world interest by applying AR CNPs on climate data fusion tasks, modelling [0, 1] -constrradd data with a beta-categorical likelihood and introducing novel multiscale architecture. Notably, AR CNPs fill a gap in the climate modelling toolbox by enabling joint, non- Gan-Gaussian predictives, which could be used to better estimate the magnitude of compound risks. We also position AR CNPs within the larger neural density estimator literature, showing the fruitfulness of combining NPs with other modelling paradigms.
More generally, AR CNPs equip the NP framework with a new knob where modelling complexity and computational expense training time can be traded for computational expense at test time. In particular, the higher quality samples and better likelihoods obtained by applying NPs autoregressively come with the additional cost of performing a forward pass for every element the target set. This can be prohibitively expensive for large target sets, and constitutes the primary practical drawback of using AR CNPs. addition, since AR CNPs do not define a consistent stochastic process, design choices for the AR procedure may affect the quality of the results. Thus practitioners need to avoid choosing target sets that lead to pathological behaviour, such as when the spatial density of the target opportunity: as
inputs is too high. However, the flexibility of this design space also presents an example, in Appendix M we show that auxiliary target points can used to further improve predictions. Finally, promising avenues for future work include applying the AR procedure to other NPs besides CNPs, and investigating the efficacy of the block sampling scheme presented Section

---

Published as a conference paper at ICLR 2023
6 REPRODUCIBILITY STATEMENT
All our experiments are carried out using either synthetic or publicly available datasets. The EEG data set is available through the UCI database." and the environmental data also publicly available through the European Climate Data Service.
We make publicly available all code necessary reproduce experiments* well instructions for downloading, preprocessing, and modelling the Antarctic cloud cover data'. Proofs for Propo- sitions 2.1 and 2.2 are given in Appendix A and Appendix B respectively. Details the model architectures and the experimental setup can be found in Appendices F to H for the synthetic datasets, Appendix the sim-to-real transfer experiments, Appendix for the EEG experiments, Appendix for the data assimilation experiment, and Appendix L for the downscaling experiment.
7 ETHICS STATEMENT
Training CNPs autoregressively improves their performance dramatically, but we do not foresee ad- verse societal impacts as result of this work. That being said, the problem of capturing the statistical trends present in a dataset must be performed with care, especially in safety critical applications, where the stakes of making incorrect and confident predictions can have severe consequences. We view the AR procedure as a useful tool, rather than a panacea, for capturing such behaviours, and hope this work encourages further research into building effective but reliable models this end.
We also note that while training CNPs is computationally cheaper than alternative NP models, AR sampling itself incurs substantial computational cost, and thus energy cost, test time. Running AR sampling on a large scale could lead to greater power demands for these models, resulting in larger carbon footprints which are undesirable. However, we believe the potential benefits for environmental modelling could outweigh this cost, while leveraging methods to make AR CNPs more computationally efficient should help alleviate this issue.
ACKNOWLEDGEMEENTS
This research was conducted while WPB and AYKF were students at the University of Cambridge. During that time, WPB was supported by the Engineering and Physical Research Council (studentship number 10436152), and AYKF supported by the Trinity Hall Studentship and the George and Lilian Schiff Foundation. SM acknowledges funding from the Vice Chancellor's s & George and Marie Vergottis scholarship and the Qualcomm Innovation Fellowship. TRA and JSH supported by Wave 1 of The UKRT Strategic Priorities Fund under the EPSRC Grant EP/W006022/1, particularly the AI for Science theme within that grant & The Alan Turing Institute. RET is supported by Google, Amazon, ARM, Improbable and EPSRC grant EP/T005386/1.
REFERENCES
Michael A. Alcorn and Anh Nguyen. The DEformer: An order-agnostic distribution estimating transformer. ICML Workshop on Invertible Neural Networks, Normalizing Flows, and Explicit Likelihood Models (INNF+), 2021.
Martin Arjovsky, Soumith Chintala, and Leon Bottou. Wasserstein generative adversarial networks. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 214-223. PMLR, 2017.
Matthew Ashman, Jonathan So, Will Tebbutt, Vincent Fortuin, Michael Pearce, and Richard E. Turner. Sparse Gaussian process variational autoencoders. arXiv preprint arXiv:2010.10177 2020.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey F. Hinton. Layer normalization. Neural Information Processing Systems Deep Learning Symposium, 2016.
data.htm
10

---

Published as a conference paper at ICLR 2023
Henri Begleiter. EEG database data set, 2022. URL UBLLittsss//arccie..ccss.
dataset s eeeeeee+aaat aataseet/eesstcttaabbbe abase.
Jens Behrmann, Will Grathwohl, Ricky T. Q. Chen, David Duvenaud, and Jör-Henrik Jacobsen. In- vertible residual networks. Proceedings of 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research. PMLR, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In Proceedings of the 7th International Conference on Learning Representations, 2019.
Wessel P. Bruinsma, James Requeima, Andrew Y. K. Foong, Jonathan Gordon, and Richard E. Turner. The Gaussian neural process. In Proceedings of the 3rd Symposium on Advances in Approximate Bayesian Inference, 2021.
Y. Burda, R. Grosse, and R. Salakhutdinov. Importance weighted autoencoders. In Advances in Neural Information Processing Systems 29. Curran Associates, Inc., 2016.
Ni-Bin Chang and Kaixu Bai. Multisensor data fusion and machine learning for environmental remote sensing. CRC Press, 2018.
Ricky T. Q. Chen, Jens Behrmann, David K. Duvenaud, and Jörn-Henrik Jacobsen. Residual flows for invertible generative modeling. Advances in Neural Information Processing Systems, 32, 2019.
X. Chen, Y. Duan, R. Houthooft, Schulman, I. Sutskever, and Abbeel. InfoGAN: Interpretable representation learning by information maximizing generative adversarial nets. In Advances in Neural Information Processing Systems 29. Curran Associates, Inc., 2016.
X. Chen, D. P. Kingma, T. Salimans, Y. Duan, P. Dhariwal, J. Schulman, I. Sutskever, and P. Abbeel. Variational lossy autoencoder. Proceedings of the 5th International Conference on Learning Representations, 2017.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. PixeISNAIL: An improved autoregressive generative model. In International Conference on Machine Learning, pp. 864-872. PMLR, 2018.
Rewon Child. Very deep VAEs generalize autoregressive models and can outperform them on images. In Proceedings of the 9th International Conference Learning Representations, 2020.
Rewon Child, Scott Gray, Alec Radford, and Uya Sutskever. Generating long sequences with sparse transformers. arXiv preprint arXiv: 10509, 2019.
Franccois Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2017.
D. P. Dee, S. M. Uppala, A. J. Simmons, P. Berrisford, P. Poli, S. Kobayashi, U. Andrae, M. A. Balmaseda, G. Balsamo, P. Bauer, Bechtold, A. C. M. Beljaars, I., van de Berg, J. Bidlot, N. Bor- mann, C. Delsol, R. Dragani, M. Fuentes, A. J. Geer, I.. Haimberger, S. B. Healy, H. Hersbach, E. V. Holm, L. Isaksen, P. Kållberg, M. Kohler, M. Matricardi, A. P. McNally, B. M. Monge-Sanz, J.JJ Morerette, B.-K. Park, C. Peubey, P. de Rosnay, C. Tavolato, J.-N. Thepaut, and F. Vitart. The ERA-interim reanalysis: Configuration and performance of the data assimilation system. Quarterly Journal of the Royal Meteorological Society, 137(656):553 597, 201 doi: 10.1002/qj.825
Adji B Dieng, Francisco JR Ruiz, David M Blei, and Michalis Titsias. Prescribed generative adversarial networks. arXiv preprint arXiv:1910.04300 2019.
L. Dinh, J. Sohl-Dickstein, and S. Bengio. Density estimation using real NVP. In Proceedings of the 5th International Conference Learning Representations, 2017.
Laurent Dinh, David Krueger, and Yoshua Bengio. NICE: Non-linear independent components estimation. In International Conference Learning Representations, 2015.
Richard Durrett. Probability: Theory and Examples. Cambridge University Press, 4 edition, 2010.

---

Published as a conference paper at ICLR 2023
Earth Resources Observation and Science Center, U.S. Geological Survey, U.S. Department of the Interior. USGS 30 are-second global elevation data, GTOPO30, 1997. URL https://doi. org/ ///g/10..0 5065 A1 2 101124--E71 EE71.
Pau Ferrer-Cid, Jose M Barcelo-Ordinas, Jorge Gareia-Vidal, Anna Ripoll, and Mar Viana. Multisen- sor data fusion calibration in IoT air pollution platforms. IEEE Internet of Things Journal, 7(4):
3124 1132, 2020.
Andrew Y. K. Foong, Wessel Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard E. Turner. Meta-learning stationary stochastic process prediction with convolutional neural processes. In Advances in Neural Information Processing Systems 33. Curran Associates, Inc., 2020.
Vincent Fortuin, Dmitry Baranchuk, Gunnar Rätsch, and Stephan Mandt. GP-VAE: Deep probabilistic time series imputation. In International conference on artificial intelligence and statistics, pp.
1651-1661. PMLR, 2020.
M. Garnelo, D. Rosenbaum, J. Maddison, T. Ramalho, Saxton, M. Shanahan, Y. Whye Teh, D. J. Rezende, and S. M. A. Eslami. Conditional neural processes. In Proceedings of 35th International Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Research. PMLR, 2018a.
M. Garnelo, Schwarz, D. Rosenbaum, F. Viola, D. J. Rezende, S. M. A. Eslami, and Y. Whye Teh. Neural processes. In Theoretical Foundations and Applications of Deep Generative Models Workshop, 35th International Conference on Machine Learning, 2018b.
A. Geer. Learning earth system models from observations: machine learning or data assimilation? Philosophical Transactions of the Royal Society A: Mathematical, Physical and Engineering Sciences, 379(2179)))))) 37902194):22200889 February 2021. Publisher: Royal Society.
Mathieu Germain, Karol Gregor, lain Murray, and Hugo Larochelle. MADE: Masked autoencoder for distribution estimation. In International conference machine learning, pp. 881-889. PMLR, 2015.
Andrew Gettelman, Alan J Geer, Richard M Forbes, Greg R Carmichael, Graham Feingold, Derek J Posselt, Graeme L Stephens, Susan C van den Heever, Adam Varble, and Paquita Zuidema. The future of earth system prediction: Advances in model-data fusion. Science Advances, 8(14): eabn3488, 2022.
J. Goodfellow, J. Pouget Abadic, M. Mirza, B. Xu, D. Warde Farley, S. Ozair, A. Courville, and Y. Bengio. Generative adversarial networks. In Advances in Neural Information Processing Systems 27, volume 27. Curran Associates, Inc., 2014.
Jonathan Gordon, Wessel P. Bruinsma, Andrew Y. K. Foong, James Requeima, Yann Dubois, and Richard E. Turner. Convolutional conditional neural processes. In Proceedings of the 8th International Conference on Learning Representations, 2020. URL https::/ oppnreeiie. net/ forum? Ld=Skey4eBYPS.
Will Grathwohl, Ricky T. Q. Chen, Jesse Bettencourt, Ilya Sutskever, and David Duvenaud. FFJORD: Free-form continuous dynamics for scalable reversible generative models. In Proceedings of the 7th International Conference on Learning Representations, 2019.
Aditya Grover, Manik Dhar, and Stefano Ermon. Flow-GAN: Combining maximum likelihood and adversarial learning in generative models. In Proceedings of the AAAI Conference Artificial Intelligence, volume 32, 2018.
Ishaan Gulrajani, Kundan Kumar, Faruk Ahmed, Adrien Ali Taiga, Francesco Visin, David Vazquez, and Aaron Courville. PixeiVAE: A latent variable model for natural images. Proceedings of the 5th International Conference Learning Representations, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2016.
12

---

Published as a conference paper at ICLR 2023
Hans Hersbach, Bill Bell, Paul Berrisford, Shoji Hirahara, András Horányi, Joaquín Muñoz-Sabater Julien Nicolas, Carole Peubey, Raluca Radu, Dinand Schepers, Adrian Simmons. Cornel Soci, Saleh Abdalla, Xavier Abellan, Gianpaolo Balsamo, Peter Bechtold, Gionata Biavati, Jean Bidlot, Massimo Bonavita, Giovanna De Chiara, Per Dahlgren, Dick Dee, Michail Diamantakis, Rossana Dragani, Johannes Flemming, Richard Forbes, Manuel Fuentes, Alan Geer, Leo Haimberger, Sean Healy, Robin J. Hogan, Elias Holm, Marta Janiskova, Sarah Keeley, Patrick Laloyaux, Philippe Lopez, Cristina Lupu, Gabor Radnoti, Patricia de Rosnay, Iryna Rozum, Freja Vamborg, Sebastien Villaume, and Jean-Noël Thépaut. The ERA5 global reanalysis. 146(730):1999 199999999,9999 2020. ISSN 1477-870X. doi: 10.1002/qj.3888
Jonathan Ho, Xi Chen, Aravind Srinivas, Yan Duan, and Pieter Abbeel. Flow++: Improving flow- based generative models with variational dequantization and architecture design. In International Conference on Machine Learning, pp. 2722-2730. PMLR, 2019.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 23:6840-6851, 2020.
Emiel Hoogeboom, Alexey A Gritsenko, Jasmijn Bastings, Ben Poole, Rianne van den Berg, and Tim Salimans, Autoregressive diffusion models. In Proceedings of the 10th International Conference on Learning Representations, 2021.
Marjan Hosseini and Reza Kerachian. A data fusion-based methodology for optimal redesign of groundwater monitoring networks. Journal of Hydrology, 552:267-282, 2017.
Chin- Bein-Wei Huang, David Krueger, Alexandre Lacoste, and Aaron Courville. Neural autoregressive flows. In International Conference on Machine Learning, pp. 2078-2087, PMLRR 2018.
Douglas R. Hundley. Introduction mathematical modelling. URL http:: whitman. edu/~hundlede / courses /M250F 03/550FFF2MM222 50. html.
Nikolay Jetchev, Urs Bergmann, and Roland Vollgraf. Texture synthesis with spatial generative adversarial networks. In Workshop on Adversarial Training of Advances, Neural Information Processing Systems 29, 2016.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
H. Kim, A. Mnih, J. Schwarz, M. Garelo,, A. Eslami, D. Rosenbaum, O. Vinyals, and Y. Whye Teh. Attentive neural processes. In Proceedings of the 7th International Conference on Learning Representations, 2019.
D. P. Kingma and J. Ba. ADAM: A method for stochastic optimization. In Proceedings of the 3rd International Conference on Learning Representations, 2015.
D. P. Kingma and M. Welling. Auto-encoding variational Bayes. In Proceedings of the 2nd Interna- tional Conference on Learning Representations, 2014.
D. Kingma, Salimans, Jozefowicz, Chen, I. Sutskever, and M. Welling. Improving varia- tional inference with inverse autoregressive flow. In Advances in Neural Information Processing Systems 29. Curran Associates, Inc., 2016.
Durk P Kingma and Prafulla Dhariwal. Glow: Generative flow with invertible 1x1 convolutions. Advances in neural information processing systems, 31, 2018
Iryna Korshunova, Yarin Gal, Arthur Gretton, and Joni Dambre. Conditional BRUNO: A neural process for exchangeable labelled data. Neurocomputing, 4 416:305-309, 2020.
Dana Lahat, Tülay Adali, and Christian Jutten. Multimodal data fusion: An overview of methods, challenges, and prospects. Proceedings of the IEEE, (4399))1449-147) 2017.
Alfred J. Lotka. Contribution to the theory of periodic reactions. The Journal of Physical Chemistry, 14(3):271-2744 1910. ISSN 0092-7325. doi: 10.102//11501111004 URL https: ado..org... 10 10.1021
13

---

Published as a conference paper at ICLR 2023
Chaochao Lu, Richard E. Turner, Yingzhen Li, and Nate Kushman. Interpreting spatially infinite generative models. In ICML Workshop on Human Interpretabiliryy Machine Learning, 2020.
Zhenyu Lu, Jungho Im, Lindi Quackenbush, and Kerry Halligan. Population estimation based on multi-sensor data fusion. International Journal of Remote Sensing, 31(21):5587-5664, 2010.
D. A. MacLulich. Fluctuations in the Numbers of the Varying Hare (Lepus Americanus). University of Toronto Press, 1937. doi: 10.3138/9781499 1487583064.
Douglas Maraun and Martin Widmann. Statistical Downscaling and Bias Correction for Climate Research. Cambridge Uiversity Press, 2018. doi:
Douglas Maraun, Martin Widmann, José M. Gutiérrez, Sven Kotlarski, Richard E. Chandler, Elke Hertig, Joanna Wibig, Radan Huth, and Renate A. I. Wilcke. VALUE: A framework to validate downscaling approaches for climate change studies. Earth's 's Future, 3(1):1-14, 2015. doi: 10.1002/2014EF00002 URL https: :ttttss://gguu /ttp///ggbb.../////// wwapppuss.onniiinnnee onlinel ibbrar wi ley.ey..oo com/ doi/ abs/10. 1 00200022//22 2201 4FF000259.
Douglas Maraun, Theodore G. Shepherd, Martin Widmann, Giuseppe Zappa, Daniel Walton, José M. Gutiérrez, Stefan Hagemann, Ingo Richter, Pedro M. M. Soares, Alex Hall, and Linda O. Mearns. Towards process- process-informed bias correction of climate change simulations. Nature Climate Change, 7(11):764-773, 2017. ISSN 1758-6798. doi: 10. 10.1038/climatt34115
Stratis Markou, James Requeima, Wessel P. Bruinsma, and Richard E. Turner. Efficient Gaussian neural processes for regression. In Workshop on Uncertainty & Robusiness in Deep Learning, 39th International Conference Machine Learning, 2021.
Stratis Markou, James Requeima, Wessel P. Bruinsma, Anna Vaughan, and Richard E. Turner. Practical conditional neural processes via tractable dependent predictions, In Proceedings of the 10th International Conference on Learning Representations, 2022.
Juan Maroñas, Oliver Hamelijnck, Jeremias Knoblauch, and Theodoros Damoulas. Transforming gaussian processes with normalizing flows. In International Conference on Artificial Intelligence and Statistics, pp. 1081-1089. PMLR, 2021.
Jacob Menick and Nal Kalchbrenner. Generating high fidelity images with subscale pixel networks and multidimenssonal upscaling. In Proceedings of the 7th International Conference Learning Representations, 2019.
Thomas P. Minka. Expectation propagation for approximate Bayesian inference. In Conference in Uncertainty in Artificial Intelligence, volume 17, pp. 362-369, Morgan Kaufmann Publishers Inc., 2001.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In Proceedings of the 6th International Conference Learning Representations, 2018.
M. Morlighem. Measures bedmachine antarctica, version 2, 2020. URL URLhttsss:/ns.ddc.orr DStta/NSDDC-075//vvrse
Tung Nguyen and Aditya Grover. Transformer neural processes: Uncertainty-aware meta learning via sequence modeling. In International Conference on Machine Learning, pp. 16569-16594. PMLR, 2022.
Augustus Odena, Vincent Dumoulin, and Chris Olah. Deconvolution and Checkerboard Artifacts. Distill, 1(10)e3, October 2016. ISSN 2476-0757. doi: (0.23915/distill.0000 URL http: // //distiill.. pubb hutistt1..pbb2211666
Bernt Oksendal. Stochastic differential equations: an introduction with applications. Springer Science & Business Media, 2013.
Aaron den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. WaveNet: A generative model for raw audio. arXiv preprint arXiv: 1609.033999999 2016a.
14

---

Published as a conference paper at ICLR 2023
Aaron van den Oord, Nal Kalchbrenner, Lasse Espeholt, Oriol Vinyals, Alex Graves, et al. Conditional image generation with PixeICNN decoders. Advances in neural information processing systems, 29, 2016b.
Aaron van den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Alec Radford, Luke Metz. and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. In Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks, 2016.
Prajit Ramachandran, Tom Le Paine, Pooya Khorrami, Mohammad Babaeizadeh, Shiyu Chang, Yang Zhang, Mark A. Hasegawa-Johnson Roy H. Camphell, and Thomas S. Huang. Fast generation for convolutional autoregressive models. In Workshop Track, International Conference on Learning Representations, 2017.
Suman Ravuri, Karel Lenc, Matthew Willson, Dmitry Kangin, Remi Lam, Piotr Mirowski, Megan Fitzsimons, Maria Athanassiadou, Sheleem Kashem, Sam Madge, Rachel Prudden, Amol Mandhane, Aidan Clark, Andrew Brock, Karen Simonyan, Raia Hadsell, Niall Robinson, Ellen Clancy, Alberto Arribas, and Shakir Mohamed. Skilful precipitation nowcasting using deep generative models of radar. Nature, 597(7878):672-677, September 2021. ISSN 1476- 4687. doi: 10. s41586 02000001---00 2. Number: 7878 Publisher: Nature Publishing Group.
URL https: / /wwww. www.wature. natura com/ aamarrriccess///
Caleb Robinson, Kolya Malkin, Nebojsa Jojie, Huijun Chen, Rongjun Qin, Changlin Xiao, Michael Schmitt, Pedram Ghamisi, Ronny Hänsch, and Naoto Yokoya. Global land-cover mapping with weak supervision: Outcome of the 2020 IEEE GRSS data fusion contest. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 14:3185-3199, 2021.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional networks for biomedical image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
Tim Salimans, Diederik Kingma, and Max Welling, Markov chain Monte Carlo and variational inference: Bridging the gap. In International Conference on Machine Learning, pp. 1218-1226.
PMLR, 2015.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. PixeICNN++: Improving the PixeICNN with discretized logistic mixture likelihood and other modifications. arXiv preprint arXiv:1701.05517, 2017.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference Machine Learning, pp. 2256-2265. PMLR, 2015.
Thomas F. Stocker, Dahe Qin, Kasn-K Plattner, Melinda M.B. Tignor, Simon K. Allen, Judith Boschung, Alexander Nauels, Yu Xia, Vincent Bex, and Pauline M. Midgley. Climate change 2013: The physical science basis. Technical report, Cambridge University Press, 2013.
A. M. G. Klein Tank, J. B. Wijngaard, G. P. Können, R. Böhm, G. Demarée, A. Gocheva, M. Mileta, S. Pashiardis, L. Hejkrlik, C. Kern-Hansen, R. Heino, Bessemoulin, G. Müller- Westermeier. M. Tzanakou, S. Szalai, T. Pálsdóttir, D. Fitzgerald, S. Rubin, M. Capaldo. M. Maugeri, A. Leitass,
A. Bukantis, R. Aberfeld, A. V. van Engelen, E. Forland, M. Mietus, F. Coelho, C. Mares, V. Razuvaev, E. Nieplova, T. Cegnar, J. Antonio López, B. Dahlström, A. Moberg, W. Kirch- hofer, A. Ceylan, O. Pachaliuk, L. V. Alexander, and P. Petrovic. Daily dataset of 20th- century surface air temperature and precipitation series for the european climate assessment. International Journal of Climatology, 22(12))1441-1453, 2002. doi: 10. 10.1002/joc..73 URL https: /trpt::/rmmtt.... rmets.
online wwt.eellbbrrry..eeee wi wiley. wwt. www.///ii//e abs 10 ab.. 10w.00022/...... 773.
Benigno Uria, lain Murray, and Hugo Larochelle. RNADE: The real-valued neural autoregressive density-estimator. Advances in Neural Information Processing Systems, 26, 2013.
15

---

Published as a conference paper at ICLR 2023
Benigno Uria, lain Murray, and Hugo Larochelle. A deep and tractable density estimator. In International Conference on Machine Learning, pp. 467-475 PMLR, 2014.
Benigno Uria, Marc-Alexandre Côté, Karol Gregor, lain Murray, and Hugo Larochelle. Neural autoregressive distribution estimation. Journal Machine Learning Research, 17(205):1-37, 2016. URL http:// fmmrr. org/ http:/appssssaeee / 7 / 6- 272 html.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems 30. Curran Associates, Inc., 2017.
A. Vaughan, W. Tebbutt, S. Hosking, and R. E. Turner. Convolutional conditional neural processes for local climate downscaling. Geoscientific Model Development, 15(1):251-268: 2022. doi: 10. 5194/gmd- 15-211--2---0022 URL https: gmd. worg htt.://ssoo..rrrrrreeeeee/////// 2022/.
Oriol Vinyals, Charles Blundell, Timothy Lillicrap. Koray Kavukcuoglu, and Daan Wierstra. Match- ing networks for one shot learning. Advances Neural Information Processing Systems 29. Curran Associates, Inc., 2016.
Michael Volpp. Fabian Flürenbrock, Lukas Grossberger, Christian Daniel, and Gerhard Neumann. Bayesian context aggregation for neural processes. In International Conference on Learning Representations, 2021. URL https: oppprevieeeeeeeeeeeeeeeeee n forum rid=ufiz2 aehFa.
V. Volterra. Variazioni fluttuazioni del bumero d' ondividui in specie animali conviventi. Memoria della Reale Accademia Nazionale dei Lincei, 2:31-113, 1926.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le.
XetNet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In International conference on machine learning, pp. 7354-7363 PMLR, 2019.
X. L. Zhang, H. Begleiter, B. Porjesz, W. Wang, and A. Litke. Event related potentials during object recognition tasks. Brain Research Bulletin, 38(6):531-535 1995.
16

---

Published as a conference paper at ICLR 2023
A PROOF OF PROPOSITION 2.1
Additional notation. If Y1 ① Y2 ~ 0x (D), then denote the distribution of y1 Y2 by Px xx T (D)). Note that Px, x2 H (D) depends on Y2. because it is the distribution of yı Y2, even though the notation does not make this dependence explicit.
The "appropriate regularity conditions". Let PA be the collection of distributions R that (a) have a density with respect to the Lebesgue measure and (b) have a covariance matrix which is strictly positive definite. Let PxG _ PN be the subcollection of distributions which are Gaussian. Then, by Corollary B.1 by Bruinsma al. (2021), for all μ E PX such that inf KL(,, cc.
arg pEP. [I, v) = -Wp))
(10)
where M()) denotes the Gaussian distribution with mean vector and covariance matrix equal to those of p.
In the proposition, by appropriate regularity conditions on we mean the assumption that, for all inputs x and D E D, ,,x in Pxx and such that inf PePla ( < co.
Assume the appropriate regularity conditions on y. We now list three technical observations.
(1) Note that Px x2 Ty (D) is the distribution of y(x1) ) D, (X2.3)) so we have the identity Pxi xa Ty (D) = Px Ty D @ (2). Therefore, for all inputs X1, inputs X2, and D E D, Pxi x2 Ty (D) is in P xil and such that inf, KL(Px, xx Ty (D), v/ < cc.
(2) The ideal CNP Te matches the means and marginal variances of the true posterior predictives (Section 2). Hence, for all E X and D E D, (D) is in P).a-
(3) The ideal GNP To matches the mean vectors and covariance matrices of the true posterior predictives (Section 2). Hence, for all inputs x and D DD PxTa (D) is in X, which means that, for all æ1 X, inputs X2, and D) DD Pz, x2 (D( is in Pl.a- In the proof, apply and use (10), we implicitly use these observations.
xf
Proposition 2.1 (Advantage of AR CNPs over GNPs). Assume appropriate regularity conditions on y. Let Te be the ideal CNP and let To be the ideal GNP. Then, for all inputs x and data sets D E D, (L (D), ARx D)) < (L(PxT,(D), (D))
Proof of Proposition 2.1. Let x some inputs and let D E be some data set. We will argue that, for all n 1...., KL(P 2n [x)(n)n) Ps, DD + 11)
x/.
< KL((P,, [x Ty((),, Pa., Xis(a-1y To (D)).
(11)
Assuming this inequality, the result follows directly from the chain rule for the KL divergence in combination with the definition of ARx (Procedure 2.1):
KL(PxT, (D), ARx (Te, D)) [KL 1x1:(=-1) Fy (D), ; Pa,, Tc( @ T
= |xx
Ey
(12)
<
|x|
E By [L ( (x1((n11))
(D), (D;,, Psn x (D)))
(13)
(D))
(14)
where the expectations are over y1:(n-1) Px (D)) To prove the inequality, note that, conditional on VI:(n-1); using (10),
arg min KL(P | Ty( = N(P... xx F (D)).
(15)
By the property of Fc that it matches the mean and marginal variance of the true posterior (Section 2),
((Pa., X1 Ty = (P 2.. ( 4 (1(o--))-
(16)
= Pa., Te(D ) (x1((=-)), V1(a-))))
(17)
Therefore,
arg min KT,(Pr, z. Xx (D v) Ps, ( (X1((n-)); Y (1))-))))-
(18)
DEP)
Noting that Pan (x (D( E P1 A.g- obtain the desired inequality.
n
17

---

Published as a conference paper at ICLR 2023
B PROOF OF PROPOSITION 2.2
Proposition 2.2 (Recovery of smooth samples). Let A' [ R be compact, and let be a stochastic process with surely continuous sample paths and supze (0>===ll[(<>>]])))))))) (=))llz < co. Let (En)n20 be i.i.d. (potentially non-Gaussian) random variables such that [[ = 0 and W[sa] < co. Consider any sequence (2n)n21 S X, and let x= E X be limit point of (In)=21. If y(x))) f(x") En and Yn = f(xx) + En are noisy observations of f. then
lim Ely(x") (((x"))l1..>) yi...... ynl = f(x") almost surely. n
(8)
Proof of Proposition 2.2. Consider the increasing filtration F, = (31, In with limit Foo
a(U Fn). Also let Tn 0E and consider the tail o-aleerra Nx=1 Tn- Let 1( (fm)) 000 be subsequence of (#n) such that Zn; - x². Let 9n Σ Li=1 yt- Since In is
function of .1,..., ,l....,,r... ,n. it is F.-measurable and therefore F measuurae..... Note that
1 n 72
On
Σ E
(19)
n
By sure continuity of f. the first term converges to f(x") surely. By the strong law of large numbers (Example 5.6.1; Durrett, 2010), the second term converges to zero tail event A E T of probability one. We conclude that fxf(=*) is (F meacuuuab.. Therefore, by almost sure convergence of b,2-bounded martingales (Theorem 5.4.5; Durrett, 2010),
lim Ely(x*)
lim B[f(z") | /i,..... Vn]
(Eleo] = 0)
(20)
= lim [[f("")|
(definition of Frr)
(21)
lim B[f(x") | [F,, T]
(o([(=")) 1 T)
(22)
lim E[Iaf("") ( Fr, TI
(P(A)= 1)
(23)
=[[Af((")|
(L²-mart. convergence)
(24)
=Af((=")
(LAf(x*) oFFo, T))
(25)
= f("",,
(P((A) 1)
(26)
where all equalities hold almost surely.
18

---

Published as a conference paper at ICLR 2023
C ILLUSTRATION OF THE AR PROCEDURE
Figure 7 depicts the AR sampling procedure (Procedure 2.1) and procedure produce smooth samples (Proposition 2.2) using the ConvCNP trained on the EQ data process from Section 4.1.
Model fit
Step 1: Draw noisy samples using AR sampling (Procedure 2.1)
Step 2: Denoise sample by passing it through the model (Proposition 2.2)
Multiple samples
Figure 7: Illustration of the AR procedure with random AR ordering and the de-noising step (Procedure 2.1 and Proposition 2.2), to produce smooth samples. Given context (black crosses), we use the CNP to get marginal predictions arbitrary input locations (first figure). choose randomly sampled input location. draw a corresponding output sample from the model's predictive (blue dots in the second plot), append this to the context set, and pass the augmented context set through the model again. We repeat this (third and fourth figures), until all function (epistemic) uncertainty has been removed and all that remains irreducible noise (aleatoric) uncertainty (fitth figure). This procedure yields noisy function samples (blue dots the sixth plot). which we pass one last time through the model to obtained denoised sample, treating the mean prediction as an approximate noiseless sample (seventh figure). Repeating this procedure yields high-quality samples from the model predictive (eighth figure).
number of times
19

---

Published as a conference paper at ICLR 2023
D NUMBER AND ORDER OF TARGET POINTS
When deploying conditional neural process (CNP) autoregressively (AR; Procedure 2.1), the number and ordering of the target points matters. In this appendix, we describe our observations of the effects of the number and ordering of the target points on the quality of the predictions. In short, our recommendation is to choose a different random ordering for every sample, and to not let the number/density of target points exceed that at training time.
D.1 EFFECTS OF THE NUMBER OF TARGET POINTS
During the AR sampling procedure, the AR CNP evaluated context sets of increasing size. Our experience is that, as long as the sizes of these context sets do not exceed the sizes seen at training time, the predictions should not be significantly affected by changes in the number of target points. However, if the AR sampling procedure evaluates the model at context sets of larger sizes than seen during training time, then that presents the model with an out-of-distributiin situation. What happens then comes down to how well the neural networks generalise. Our experience is that the predictions quickly start break down.
A notable exception of this rule of thumb are convolutionnalddee--st--- Convolutional Conditional Neural Process (ConvCNP: Gordon al., 2020). For these models, the magnitude of the density channel is what determines whether the models generalises or not. This means that it is not the total number of points that matters, but rather the density of the points. Therefore, the AR ConvCNP can be evaluated at arbitrarily many target points, long the density of these points does not significantly exceed the density of context points seen at training time. Once the density exceeds the density of the training data, the model is presented with situation, and what happens then again comes down to how well neural networks generalise.
models, such as the
distribbiin
Figure Appendix D.1 illustrates this observation. When the density target points does not exceed the training data (50 and 100 points), the predictions look calibrated. However, once the density of target points comes close or exceed the training data (200, 500, and 1000 points), bias starts creep into the predictions.
Although the number/density of points in the AR sampling procedure should not exceed that at training time, AR CNPs can still produce high-quality samples at arbitrarily many target points by following the trick outlined at the end of the two-step procedure below Proposition 2.2.
D.2 EFFECTS OF THE ORDERING OF TARGET POINTS
Our experience that, as long as the number of target points (or density) does not exceed that at training time, the ordering of the target point does not really matter. Appendix D. also demonstrates this. When the density of the target points does not exceed the training data (50 and 100 points), sampling randomly or left to right does not really matter. However, once the density of the target points comes close to or exceeds the training data (200, 500, and 1000 points), we observe a difference in performance between sampling randomly and sampling left to right. Across all numbers of target points, a random ordering seems to perform most robustly. Our recommendation is therefore to choose a different random ordering of the target points for every sample.
D.3 ANALYSIS OF AR CNPS FOR CNPs WITH GAUSSIAN MARGINALS
In this subsection, we argue that, for CNPs with Gaussian marginals, predictions in the first few AR steps might be poor, but predictions in later AR steps tend to be more accurate. Choosing a different random ordering for every sample therefore "averages out" the effects from these first few AR steps.
When evaluating a CNP with Gaussian marginals in AR mode, every conditional prediction in the AR process is Gaussian. Let us consider the process of producing an AR sample. For the first target input 1 we run the CNP forward to obtain a distribution for the corresponding target output y1. In reality, the true posterior most likely is non- non-Gaussian, which means that the prediction for the first target point may be poor. Nevertheless, context set, and run the CNP forward again. Because we now feed the earlier sample y/ through the non-linear network, the marginal predictive for the next target output y/2 (having integrated out y1)
sample this Gaussian, append the sample (æ1, Vı) to the
20

---

Published as a conference paper at ICLR 2023
50 Points Random
50 Points Left to Right
-2
-2
100 Points Random
100 Points Left to Right
200 Points Random
200 Points Left Right
2-
-2
500 Points Random
Points Left to Right
1000 Points Random
1000 Points Left to Right
2
Figure Samples and predictions for an AR ConvCNP with various numbers target points ordered randomly (left columm) and ordered left to right (right colamn). When the density of the target points does not exceed the training data (50 and 100 points), ordering the target points randomly or left to right does not matter. When the density of the target points comes close the training data exceeds it (200, 500, and 1000 points), bias creeps into predictions. The random ordering appears perform more robustly than to right. The data sampled from the EQ data process from the synthetic experiments (Section 4.1), and the trained model is also taken from the synthetic experiments. The predictions by the model are shown in solid blue and the marginals by the ground-truth FQ process are shown in dot-dashed purple.
is non-Gaussian. As we perform more AR steps, the marginal predictions of later points become increasingly non-Gaussian, increasing the model's flexibility.
We see that, for a given ordering of the target inputs, the prediction for the first target input is likely poor (because it is Gaussian), and (in the best case) the predictions become more and more accurate as we take more AR steps (because they become more and more non-Gaussian). This is exactly what is happening in Figure 3: the left prediction is Gaussian and therefore poor approximation, and more AR steps, the prediction becomes more and more
and, as go to the right and take non non-Gaum Con-Gausiian and therefore more accurate. we were to feed the target inputs right to left, then the same phenomenon would happen. The right prediction would be Gaussian and very poor
21

---

Published as a conference paper at ICLR 2023
approximation, and, as go to the left and take more AR steps, the prediction would become more
non-Gaussian and therefore more accurate.
More generally, for given ordering of the target points, the ordering will produce high quality predictions if the conditional distributions of the AR factorisation match the corresponding conditional distributions of the true posterior. Since the conditionals of the AR CNP typically Gaussian by design, this means that the ordering is "good" if the corresponding conditionals of the true posterior are close to Gaussian.
So when is a conditional of the posterior close to Gaussian? Let us assume that the true underlying process is sum of non-Gaussian process (constituting epistemic uncertainty) and independent Gaussian noise (constituting alcatoric uncertainty). Generally, a conditional will have both epistemic and alcatoric uncertainty, so a Gaussian will be bad fit. However, as we condition the conditionals of the true generative process on more and more data, the underlying function will be pinned down more and more accurately, meaning that the conditional will consist mostly of aleatoric uncertainty, which is Gaussian. Therefore, as we condition to become more and more Gaussian. This again suggests that the samples in the first few AR steps might be a poor fit (because the corresponding conditionals of the true posterior are not yet Gaussian), but that samples in later AR steps should be a better fit (because the corresponding conditionals are then close to Gaussian).
more and more data, we expect the conditionals
To summarise, an ordering of the target points is "good" if the corresponding conditionals of the true posterior are also close to Gaussian. Under the assumption that the ground-truth process is non-Gaussian process with additive Gaussian noise, conditionals tend to be close to Gaussian they are conditioned on many data points. As consequence, the earlier conditionals in the AR factorisation tend to be poor fits to the ground-truth posterior, whereas later conditionals tend to produce better fits. Choosing different random ordering for every sample therefore "averages out" the effects from the first few AR steps.
D.4 EFFECT OF THE RANDOM ORDERING ON THE SPREAD OF THE LOG-LIKELIHOOD
We have thus far argued for the benefit of using random ordering in AR, due to the robustness it provides. However, one issue with random orderings is that, since different random orderings do not in general give rise to the same predictive distribution, we may obtain different predictive log-liielhoods in practice, depending on the exact random ordering that we sample. Ideally, we would like not only the mean predictive log-likelihooc (averaged out over orderings) to be high, but also the standard deviation of the log-likelihood (due to, again, different random orderings) to be small. In other words, we would like the model to perform well regardless of the random ordering which we happen to sample.
At this point, note that if the true underlying process is Gaussian, then sufficiently well-trained AR CNP with Gaussian marginals would have a small such spread in the log-likelihooo because all conditional predictions of the model will be close to the ground truth conditional predictions. Consequently the order with which we make predictions will have a small effect on the log- likeeiiodoo resulting in small spread of predictive log-liielihood values. Consider for example the case where the conditionals of the CNP exactly match the conditionals of the true process. In this case, there will be zero variance in the predictive log- likellioooood of the process under different orderings. However, the situation is different when the ground truth is non-Gaussian. In this case, as we explained in the previous section, the conditionals of the first few target points may be highly non-Gaussian under the true process. while those of the AR CNP are Gaussian. In this case, we may get different logliikeihoods depending on the random order that we happen sample.
Figure 9 provides a quantitative illustration of the above point. In this figure, we show the standard deviation in the per datapoint predictive log-likelihood of an AR CNP (due to different random orderings) on two variants of task with sawtooth data. On the first variant, we always pass an empty context set to the model (blue), and on the other task, we pass non-empty context sets with randomly sampled number of context points, uniformly distributed between 0 and 100 (red). We observe that for empty contexts (blue), we get a relatively large standard deviation in predictive log-likeli first few target points. This likely happens because, initially, the model may randomly pick target input where the conditional of the true process is highly non- Gaussian (making a poor prediction). or it might pick target input where the true conditional is Gaussian (making a good prediction). This
for the
22

---

Published as a conference paper at ICLR 2023
Standard deviation in log-likelihood per datapoint (sawtooth data, random ordering)
0.50
Std. in log- Pg-liik per datapoint
- (x
-0-
(xl() ~ U[10, 100)
0.25
0.00
10²
102
# target points
Figure 9: Plot of the standard deviation, due to different random orderings, of the per-data-point predictive log li-lliiiihood (in nats) of an AR ConvCNP on one one-dimense dimensional sawtooth data, function of the number of target set size. For each point in the plot, have used 2 randomly sampled and fixed tasks, each of which we apply AR ConvCNP with 100 different randomly sampled orderings.
results in a larger variance in performance for the first few target points. However, more target points are introduced, the standard deviation shrinks. This because the conditionals of the true process become increasingly Gaussian, which means that matter which target input picked next, the model will approximate the true conditional accurately using a Gaussian, thereby reducing the impact of the ordering of subsequent points on the variance of the log likelihood. Further, introducing relatively modest number of initial context points (red) in a second variant of the task, substantially reduces the spread in the predictive log- logliiklissods. This is again because conditioning on context set means that the conditionals of the true process are better approximated by Gaussians, reducing the impact that different random orderings have on the spread of the log-likelihood. practice, in our experiments, we have found the variance in the log-likelihood to be near-zero for Gaussian or Gaussian-like ground truth processes, and larger, but acceptable, for non-Gaussian tasks.
23

---

Published as a conference paper at ICLR 2023
E DETAILS FOR FIGURE 3
The generative process visualised in the top panel of figure 3 is defined by the following mixture distribution:
Puree (y) (
(av((()))))
(27)
Given this mixture distribution, the (Gaussian) ideal CNP can be computed in closed form by computing the first two moments of Pune:
PyNp x) = -
(28)
where
m(x) Σ a.f((z),
(29)
( Σ a (1+ ((x)))
(30)
The updated mixture weights for the posterior distribution Pine (yl æ, D given a context set D be computed via Bayes rule and PCNP(y (y x. D(o) can be computed given the updated mixture weights. Note that in Figure 3 the prior mixture weights are 01 = dz = 0.25 and d2 = 0.5, means given by
fi(x) x² +1
(31)
f2(x) = I,
(32)
fa(x)
(33)
and the target locations are = 1,2,4, and 6. The bottom four panels of Figure 3 show ker- nel density estimates (Gaussian kernel) 30 samples drawn from the generative distribution Pirue (11, y2, U4: Ys), the ideal CNP MCNP(y1. 1/, U4, Us). and the ideal CNP applied in AR mode from left to right:
PAR CNP (11 U2, U4, yo) pCNP(y)) (PCN(()))) (PN yi 1/2 PeNP (vee 1M 92, 94).
(34)
24

---

Published as a conference paper at ICLR 2023
F DESCRIPTION OF MODELS
The architectures follow the descriptions from the respective papers they are introduced. Although these descriptions for one-dimensional inputs and outputs, the architectures are readily generalised to multidimensional inputs and ouiputs; we will explicitly mention wherever that generalisation requires extra care. All architectures use ReLU activation functions. All GNPs, in addition covariance matrix over the target points, also output heterogeneous observation noise along the marginal means; the total covariance over the target points is thus the sum of the covariance by the model and diagonal matrix formed from these observation noises.
Conditional neural process (CNP; Garnelo al., 2018a). Set the dimensionality of the encoding to K = 256. Parametrise the encoder with a teree-hideen-Bee layer multi-layer perceptron (MLP) of width 256; and parametrise the decoder with a six-hidden-laye layer MLP of width 256. For multidimensional outputs, let the decoder have width 512. For multidimensional outputs where outputs can have context points at different inputs, produce a separate encoding for every output and concatenate these into one big encoding. These encoders may or may not share parameters. In our experiments, for two-dimensional outputs, parametrise separate encoders; for gigher-dimensionll outputs, apply the same encoder.
Gaussian neural process (GNP; Markou et al., 2022). Use the same choices for K, the encoder, and the decoder the CNP. Set the rank of the kernel map to R = 64. As mentioned in the introduction, let the decoder produce one extra dimension which forms heterogeneous observation noise. For multidimensional outputs, the same caveats for the CNP apply.
Latent neural process (LNP; Garnelo et al., 2018b). The LNP builds off the CNP. Call the existing encoder the deterministic encoder. The NP adds one more encoder called the stochastic encoder. The stochastic encoder mimics the deterministic encoder, but outputs K K-dimensional vector of means and a K --dimensional vector of marginal variances. These are used to sample K -i-ieensional Gaussian latent variable (the stochastic encoding). The decoder now additionally takes in the stochastic encoding. For multidimensional outputs, the same caveats as for the CNP apply.
Attentive conditional neural process (ACNP; Kim et al., 2019). The ACNP builds off the CNP. replaces the deterministic encoder enc;: D - R with an eight-head attentive encoder enc, D x X + R (Vaswani et al., 2017). Unlike the original deterministic encoder enic, the new attentive encoder ench (at) also takes in the target input. Let D = (x(), y E D be a context set of size N and let x:!)) E X' be a target input. We now descibe the computation of enc (all) (D6)) x0)) Parametrise ox: X' + (2:32)8 and Øzy; ; X x y - (232)8 both with faree-hidden-lee layer MLPs of width 256.
Compute
the keys:
(k,,)))== 22 (x!1)
for ni 1, 1,.... N, for n I,.... N,
(35)
the values:
(Vh,,) 88 Pzy (1/10 yn
(36)
the query: (qx (qh))=1 = Ohr (0)
(37)
Then compute
e ka,))
=n==1 e (9t knnt Vhn 256
(38)
Concatenate v(9) = (v (4)
E 2556 and (qi..... qs) E 255 Let L: 255 + 255 be
a linear layer; let (rres); 256 + 2556 be one-hidden-layer MLP of width 256; and let norm; and norma be two layer normalisation layers with learned pointwise transformations (Ba al., 2016).
Then
cnc (Di), (0)) = norm (2+ + (2)) where z = norm, (v/9)) + LLq)
(39)
For multidimensional outputs, the same caveats for the CNP apply.
Attentive Gaussian neural process (AGNP). The AGNP build off the GNP. It replaces the determin- istic encoder with the same eight-head attentive deterministic encoder of the ACNP.
Attentive neural process (ALNP; Kim et al., 2019). The ALNP build off the LNP. It replaces the deterministic encoder with the same eight-head attentive deterministic encoder of the ACNP.
25

---

Published as a conference paper at ICLR 2023
Convolutional Conditional Neural Process (ConyCNP; Gordon et al., 2020). Set the discretisation to an evenly spaced grid at a certain density (the points per unit) spanning a bit more (the margin) than the most extremal context and target imputs. The points per unit and margin for every experiment. Initialise the length scales of all Gaussian kernels to twice the interpoint spacing of the discretisation. Divide the data channel by the density channel. Parametrise dec with U-Net (Ronneberger et al., 2015). Before the U-turn, let the U-Net have six convolutional layers with kernel size five, stride two, and 64 output channels; and six more such layers, but using transposed convolutions, after the U-turn. The layers after the U-turn additionally take in the outputs of the layers before the U-turn in reversed order; this is the U-net structure (Figure I; Ronneberger et al., 2015). For multidimensional outputs where outputs can have context points at different inputs, produce separate data and density channel for every output and concatenate these into one big encoding; use separate length scales for every application of ence.
specified separately
Convolutional
Gaussian neural process (ConvGNP; Markou et al., 2022). Use the same choices for the discretisation, length scales, and CNN architecture for the ConvCNP. Set the rank of the kernel map to = 64. As mentioned in the introduction, let the decoder produce one extra channel which forms heterogeneous observation noise. For multidimensional outputs, the same caveat for the ConvCNP applies.
Fully convolutional Gaussian neural process (FullConvGNP: Bruinsma et al., 2021). For the mean architecture and the kernel architecture, use the same choices for the discretisation, length scales, and CNN architecture for the ConvCNP. Implement the source channel with the identity matrix and apply the matrix transform Z + ZZT to ensure positive definiteness. Let the decoder produce one extra channel which forms heterogeneous observation noise. For multidimensional outputs, in addition to the caveat for the ConyCNP, two additional caveats apply. First, for D,- dimensional outputs, let the decoder produce D channels rather than just one. These channels should be interpreted as all covariance and cross covsrccov matrices between all outputs. Second, when applying the matrix transform Z + ZZT, these channels should first be assembled into one total covariance matrix.
Convolutional latent neural process (ConvLNP; Foong et al., 2020). The ConvLNP builds off the COnvCNP. The ConvLNP replaces the CNN architecture by two copies of this architecture placed in sequence. In between the two architectures, there is a sampling step: the first architecture outputs 32 channels, comprising 16 means and 16 marginal variances, which are used to sample 16-dimensional Gaussian latent variable; and the second architecture then takes in this sample.
Autoregressive Conditional Neural Processes (AR CNPs). The AR CNP, AR ACNP, and AR Conv CNP use the architectures described above. Rolling out an AR CNP according to Procedure 2.1 requires ordering of the target points. In all experiments, choose random ordering of the target points.
26

---

Published as a conference paper at ICLR 2023
G TRAINING, CROSS- VALIDATION,,,, AND EVALUATION PROTOCOLS
The following description applies to the synthetic experiments (Section 4.1), the predator-prey experiments (Section 4.2), the EEG experiments (Section 4.3), and the environmental downscaling experiments (Section 4.4). For the environmental data assimilation experiments, different protocol was used; we refer the reader Appendix K for full details of the environmental data assimilation experiments.
task consists of context set and target set. How precisely the context and target sets are generated is specific to experiment. To train model, we consider batches of 16 tasks at time, compute an objective function value, and update the model parameters using ADAM (Kingma & Ba, 2015). The learning rate is specified separately for every experiment. We define an epoch to consist of 2 R 16 k tasks. We typically train a model for between 100 and 1000 epochs.
For an experiment, we split up the meta-data set into training set, cross-validation set, and an evaluation set. The model is trained on the training set. During training, after every epoch, the model is cross-validated on the cross-validation 21 are fixed, which means that cross-validation always happens with exactly the same data. The cross-validation objective is confidence bound computed from the model objective. Suppose that model objective over all 12 cross-validatio tasks has empirical mean j and empirical variance 8². a higher model objective is better, then the cross-validation objective is given by j - 1.96 //v /21Z, The model with the best cross-vall valsiaaolo objective is selected and used for evaluation. Evaluation is performed with the evaluation set and also uses 2 tasks.
Cross-validation uses 2 fixed tasks. These
Conditional neural processes and Gaussian neural processes are trained, cross-validated, and evaluated with the neural process ELBO objective proposed by Gameeoo et al. (201 (2). We normalise the terms in the neural process objective by the target set sizes. Latent-variable neural processes (LNPs) are trained, cross-validated and evaluated with the ELBO objective proposed by Garnelo et al. (2018b) using five samples, also normalised by the target set size. When training LNPs with the ELBO objective, but not when cross-validating and evaluating, the context set is subsumed in the target set. Additionally, LNPs are trained, cross-validated and evaluated with the ML objective proposed by Foong et al. (2020), again normalised by the target set size. When training and cross-vale cross-validating LNPs with the MI. objective, we use twenty samples; and when evaluating, we use 512 samples. For completeness, LNPs trained with the ELBO objective are also evaluated with the ML objective using 512 samples.
To stabilise the numerics for GNPs, increase the regularisation of covariance matrices for one epoch. To encourage LNPs to fit, we fix the variance of the observation noise of the decoder to 10-4 for the first three epochs.
27

---

Published as a conference paper at ICLR 2023
H DETAILS OF SYNTHETIC EXPERIMENTS
H.1 DESCRIPTION OF EXPERIMENTS
We synthetically generate data sets by randomly sampling from five different choices for the ground- truth stochastic process f. Let the inputs be dy-dimensional. Then define the following stochastic processes:
EQ: a Gaussian process with an exponentiated quadratic (EQ) kernel:
GP(O, exp(-2iz ||x x' (B)
(40)
where & > 0 is length scale;
Matérn 31 a Gaussian process with Matérn 3 kernel:
J~ GP k( x' 121)
(41)
where k(r) = (1+ v5r + " and > is length scale;
weakly periodic: a Gaussian process with a weakly periodic kernel:
f ~ GP(00, 20
|x x
I|sin( ( (x x())l(²)
(42)
where la > 0 is length scale specifying how quickly the periodic pattern changes, 6p > 0 a length scale of the periodic pattern, and p > 0 the period; and where the application of sin is elementwise; sawtooth: a sawtooth process with random frequency, direction, and phase:
w (x, u)2 + + mod 1
(43)
where w ~ Uni is the frequency of the sawtooth wave, u ~ Unif([x Rda : ||x||2 = 13) the direction, and e Unif([O, 1J) the phase;
mixture: with equal probability, sample f from the EQ process, Matérn- process, weakly periodic process, or sawtooth process.
We will call these stochastic processes the data processes. The data processes stochastic processes with dy-dimensional inputs and one-dimensional outputs. We will turn them into processes with dy- dimensional outputs according to the following procedure: sample from the one-dimensionaalouuputt prior dy times; and, for these dy samples, take dy different linear combinations.
We choose the parameters of the data processes based on the input dimensionality dø:
C c 4
La
c.
4 =
Ω 1 4]
(44)
with c = vdy Scaling with the input dimensionality aims to roughly ensure that data with one- dimensional inputs and data with two-dimensional inputs are equally difficult. Figure 10 illustrates the sawtooth data process in all four configurations.
We will construct data sets by sampling inputs uniformly at random from X = =[-2, and then sampling outputs from one of the data processes. We will colloquially call X' the training range. For the EQ, Matérn-5 2. and weakly periodic process, but not for the sawtooth process", we also add independent Gaussian noise with variance 0.05. The numbers of context and target points are as follows. For the FQ, Matémn 5 2. and weakly periodic process, the number of context points is chosen uniformly random from {O.... 30- d and the number of targets points is fixed to 50 dg. For the sawtooth and mixture process, the number of context points is chosen uniformly at random from {0. (0....,30) 30} if dz = 1 and (0.... 75 d otherwise; and the number of targets points fixed to 100 dz. In the case of multidimensionalloourg positions of the context and target inputs for every output dimension.
data process, we separately sample the number and
For every data process and each of the four configurations, we evaluate every model in three different ways. First, we evaluate the model on data generated exactly like the training data. This task is called interpolation and abbreviated "int." in the tables of results. The interpolation task measures how well a model fits the data and is the primary measure of performance. Second, we evaluate the model on data with inputs sampled from [2, 6)ds. This task is called in-tofffipuu-distibbbttet (OOID)
"The sawtooth process is already challenging enough.
28

---

Published as a conference paper at ICLR 2023
1.0
0.5
-0.55
0.8
-0.50
0.6
fta)
-0.75
0.4
-0.5
-1.00
0.2
-1.0
-1.25
0.0
-2
0
2
-2
0
1
2
-2
0
1
2
(a) dg
1, dy
(b) de = 1, dg
2
1.0
0.5
0.8
1
0.0
0.6
-0.5 3
0,4
-1
-1.0
0.2
-1.5
-2
2
21 (c) d, 2, d,
(d) de = 2, d, =
Figure 10: Samples from the sawtooth data process with one and two-dimensional inputs (d, = and dg = 2) and one and two-dimensional outputs (d, = and dy 2)
interpolation and abbreviated "OOID" the tables results. OOID interpolation measures how well model generalises to data sampled from other regions of the input space. Third, evaluate the model on data with context inputs sampled from [- 2]d and target inputs sampled from [2, 6]., This task is called extrapolation and abbreviated "ext." in the tables of results. The extrapolation task measures how well predictions based on data in the training range generalise to other regions of the input space.
For this experiment, the learning rate 3 10-4, the margin is 0.1, and the points per unit is 64. We trained the models for 100 epochs. Due error in the cross-validation procedure, did not cross-validation, but used the model at epoch 100.
For the kernel architecture of the FuI, we reduce the points per unit and the number of channels in the U-Net by factor two. For the ConvLNP with two-dimensional inputs, we reduce the number of outputs channels in the U-Net by a factor /2; and, for training and cross-validation, we reduce the number of samples of the ELBO objective one and the number of samples for the ML objective to five.
H.2 MULTI-MODALITY OF PREDICTIONS BY AR CONVCNP
Figure 1l demonstrates molti-modality of predictions by the AR ConvCNP trained on the sawtooth process. Note that the prediction is bimodal for one and two observations, and collapses to single mode upon observing the third observation.
H.3
FULLI RESULTS
We the show the full results for all data sets and tasks in Tables 18. The AR ConvCNP consistently shows very strong performance compared to other NP models. Note that the FullConvGNP takes much longer to train than the ConvCNP (Figure 2), and cannot applied to tasks with 2-dimensional input spaces.
29

---

Published as a conference paper at ICLR 2023
Prediction at 2
1.0
Density
0.5
0.0
0.0
0.5
1.0
1.0
Density
0.5
0.0
0.0
1.0
Density
0.5
0.0
0.0
0.5
1.0
1.0
Density
0.5
0.0
0.5
1.0
0.0
Multi-modality of predictions by the AR ConvCNP, Shows four observations sampled from the sawtooth process. In the four rows, these four observations are revealed one data point time. Every row also shows kemel density estimate of the prediction 0. Filled regions central 35%-ccee regions.
Figure 11:
30

---

Published as a conference paper at ICLR 2023
Table For the Gaussian experiments, average Kullback-Leible divergences the posterior prediction map Ty with respect to the model normalised by the number of target points. Shows for one-dimensional inputs (ID; d, and two-dimensional inputs (2D; dy = 2) the performance for interpolation within the range [-2, 2) where the models were trained ("Imt.);: interpolation within the range [2, 6] which the models have never seen before ("OOID"";; and extrapolation from range [-2, 2/d= to range [2, 6) ("Fxt."). Models are ordered by interpolation performance for one-dimensional inputs. The latent variable models are trained and evaluated with the ELBO objective (E); trained and evaluated with the ML objective (M); and trained with the ELBO objective and evaluated with the ML objective (E-M). Diagonal GP refers to predictions by the ground-truth Gaussian processes without correlations. Trivial refers to predicting the empirical means and standard deviation of the test data. Errors indicate the central 35%-confieeece interval. Numbers which are significantly best (p < 0.05) are boldfaced. Numbers which are very large are marked as failed with "F". Numbers which are missing could not be
Model
Int. (ID)
OOID (ID)
Ext. (ID)
Int. (2D)
OOLD (2D)
Ext. (2D)
FullConyGNP
0.01 10.00
0.01 20.00
0.00 10.00
ConvCNP (AR)
0.03 20.00
0.03 +0.00
0.02 20.00
0.03 20.00
0.03 10.00
0.02 20000
ConvGNP
20.00
0.04 10.00
1.75
0.12 20.00
0.12 10.00
0.71 20.03
AGNP
0.10 =0.00
4.34
5.45
0.17 +0.00
0.62 40.01
0.39 +0.01
ConvLNP (E)
0.19 20.01
0.19 +0.01
+0.08
0.39 +0.01
0.39 +0.01
0.36 #0.01
ACNP (AR)
0.24 20.01
1.08 t0.02 F
0.86 +0.01
0.13 00.00
0.57 t0.01
0.40 #0.01
GNP
0.25 20.01
F
0.25 10.01
0.75 10.01
0.57 20.00
ConvLNP (M)
0.31 10.01
0.31 10.01
0.64 10.01
0.28 10.01
0.28 10.01
0.36
Diagonal GP
0.42
0.42 10002
0.84 10.01
0.29 10.01
0.29 10.01
0.40 10.01
ALNP (M)
0.43 20.01
1.03 t0.02
0.78 20.01
0.31 =0.01
0.55 #0.01
0.39 #0.01
ConvCNP
0.43 20.02
0.43 #0.02 F
0.84 #0.01 F
0.30 20.01
0.30 #0.01
0. #0.01
CNP (AR)
0.46 20.01
0.36 20.01
F
F
LNP (E)
0.51
F
4.34 40.76
0.40 #0.01
3.03
0.60 #0.01
LNP (E-M)
0.52 #0.02
2.39 40.33
#0.01
2.35 4:1.06
0.57 #0.01
ALNP (E-M)
0.53 #0.01
1.12 40.03
0.85 +0.02
0.42 #0.01
0.78 +1.72
0.41 +0.01
ACNP
0.54 =0.02
1.11 +0.02
0.84 20.01
0.34 20.01
0.57 10.01
0.40 20.01
ALNP (E)
20.01
10.06 F
1.25
0.43 10.01
1.06 13.04
0.42 20.01
LNP (M)
0.59
1.13
0.41 10.01
0.88 10.03
0.52
CNP
0.63 20.01
1..820.02
0.43 10.01
1.16 10.45
0.52 20.01
Trivial
1.08 20.01
1.08 +0.01
0.85
0.57 #0.01
0.57 +0.01
0.40 #0.00
ConvLNP (E-M)
2.01 20.11
2.01 10.11
5.95
0.44 20.01
0.44 +0.01
0.47 20.01
31

---

Published as a conference paper at ICLR 2023
Table For the non Gaus-an experiments, average log-likelihooss normalised by the number of target points.
Shows for one-dimensional inputs (ID; d: = 1) and two-dimensional inputs (2D; dg 2) the performance for interpolation within the range [-2, 2) where the models trained ("Int."); interpolation within the range [2, 6) which the models have never seen before ("OOID");; and extrapolation from the range [-2, 2) the range [2, 6jdz ("Ext."). Models ordered interpolation performance for one-dimensional inputs. The latent variable models are trained and evaluated with the ELBO objective (E); trained and evaluated with the ML objective (M); and trained with the ELBO objective and evaluated with the ML objective (E-M). Trivial refers to predicting the empirical means and standard deviation of the test data. Errors indicate the central 95%--cnnieeece interval. Numbers which significantly best (p 0.05) are boldfaced. Numbers which very large are marked failed with "F". Numbers which are missing could not run.
Model
Int. (ID)
OOID (1D)
Ext. (1D)
Int. (2D)
OOID (2D)
Ext. (2D)
ConvCNP (AR)
1.52 20.04
1.53 20.04
1.32 10.04
0.56 20.03
0.56 20.03
0.29 20.03
ConvLNP (E)
1.40
1,40 +0.05
0.82
0.06 +0.03
0.06 40.08
-0.62 #0.04
ConvLNP (M)
1.08 #0.08
40.06
-0.36 +0.08
+0.04
0,26 40.04
-0.70 #0.02
ConvGNP
0.79 +0.08
0.79 t0.06
-1.03 #0.07
0.23 #0.04
0.23 40.04
-0.79 #0.02
FullConvGNP
0.71 20.08
0.72 10.06
20.02 -0.20
ConvCNP
0.57
0.57 10.07
-0.73 10.02
0.18 20.05
0.18 10.05
-0.86
ACNP (AR)
0.07 20.03
-0.85
-0.84
-0.53 10.02
-1.52 10.10
-1.51 10.09
AGNP
0.31 20.03
-1.22 t0.07
-1.58 #0.11
10.02 -0.55
-0.79
+0.02
-0.76 #0.03
ALNP (E-M)
=0.03
0.91 t0.03
0.800 #0.04
00.67 #0.03
-1.06
#0.02
-0.70 20.03
ALNP (E)
0.35 20.03
-5.00
-3.37 20.06
00.68 #0.03
-2.99
47.64
-0.75 20.03
ALNP (M)
-0.36 10.02
-0.68
-0.68
10.02
10.02 -0.53
-0.74
10.04
-0.69 10.02
GNP
-0.38 20.02
-0.69 10.02
1.000 -0.74
-0.70 10.03
LNP (E-M) MM)
-0.43 20.02
F
3.34 +0.53
-0.66 10.02
-0.96 20.03
LNP (F)
-0.44
-0.66
F
F
ACNP
20.03
-0.83
40.03
20.05
-0.60 20.02
-1.50
-0.73 20.03
LNP (M)
-0.33 20.02
-1.28
10.05
-0.80
-0.62
-1.50
-0.75 20.03
CNP (AR)
0.65 20.02
1.14
-0.98
40.06
40.02
-1.5 40.07
-0.72 #0.03
CNP
0.68 20.02
0.79
40.04
0.73
40.03
0.69
40.92
-.05 40.08
-0.71 20.03
Trivial
0.82 20.00
0.82 +0.00
0.82 +0.00
0.82 40.00
0.82 +0.00
-0.82 20.00
ConvLNP (E-N MD
F
F
F
-0.04
-0.04
-1.47 20.67
32

---

Published as a conference paper at ICLR 2023
Table the EQ synthetic experiments
mie-immesiooal inputs, average Kullback -Libller divergences of the posterior prediction map Ty with respect to the model normalised by the number of target points. Shows for one-dimensional outputs (dy 1) and two-dinensional outputs (d, = 2) the performance for interpolation within the range [-2, 2] where the models where trained ("Int."); interpolation within the range [2, 6 which the
models have never seen before ("OOID"); and extrapolation from the range [-2, 2] to the range [2, 6] ("Ext."). Models are ordered by interpolation performance. The latent variable models are trained and evaluated with the ELBO objective (E); trained and evaluated with the ML objective (M); and trained with the ELBO objective and evaluated with the MI objective (E-M). Diagonal GP refers predictions by the ground-truth Gaussian processes without correlations. Trivial refers predicting the empirical means and standard deviation of the data. Errors indicate the central fe-codiidecce interval. Numbers which are significantly best (p 0.05) are boldfaced. Numbers which are very large are marked as failed with "F". Numbers which are missing could not be run.
EQ d, 0. dy =1
EQ d
Int.
OOID
Ext.
Int.
O0ID
Ext.
= 1, dy =2
FulIConvGNP
0.00 20.00 0.00
10.00
0.00 10.00
FullConvGNP
0.00 20.00 0.00 =0.00
0.00 10.00
ConyGNP
0.01 =0.00 0.01
=0.00
3.46
20.08
ConvGNP
0.01
10.00 0.01 10.00
1.73 10.05
ConvCNP (AR)
0.01
0.01 10.00
0.01 10.00
ConvCNP (AR)
0.01 0.01 10.00
0.01 10.00
AGNP
0.03
4.28
7.38
AGNP
0.00 7.71 40.10 7.87
ConvLNP (E)
0.06 0.06 20.00
0.11 =0.01
ACNP (AR)
0.07 1.31 #0.0
1.09 20.01
ACNP (AR)
0.07 =0.00 1.19 =0.01
0.98
=0.01
ConvLNP (E)
0.08 +0.00
0.08 +0.00
0.13 20.00
GNP
0.08 =.000
F
F
GNP
00.00
F
F
ConvLNP (M)
0.25 0.01
0.25
0.67
ConyLNP (M)
0.36 40.01 0.36 40.01
0.88 4:000
CNP (AR)
0.28 =0.00
F
ALNP (M)
0.41 10.01
40.01 F
0.99 4:0.00
ALNP (M)
0.31 =0.01
1.04 20.01
0.84 =0.01
CNP (AR)
0.42 40.00
LNP (E)
0.00 F
10.01
=0.01
Diagonal GP
0.47 0.47 #0.01
1.06 10.00
LNP (E-M)
0.37
F
1.27
20.01
ConvCNP
0.48
0.48 10.01
1.06 10.00
Diagonal GP
0.40
0.95
ACNP
0,51 40.01
1.38
1.06 40.01
ConvCNP
0.41 20.11
0.41 #0.01
0.95
#0.01
ALNP (E-M)
0.52 #0.01
1.47 40.02
1.07 40.01
ANP (E-M)
0.42 =0.01
1.18 #0.01
0.94
#0.01
ALNP (E)
0.53 +0.0
3.79
+0.00
2.84 +0.02
ANP (E)
0.44 1.32
10.01
1.25
10.01
LNP (E)
0.54 10.01
F
1.47 10.01
ACNP
0.45 0001 1.22
10.01
0.95
LNP (E M)
0.564000
F
1.42 10.01
LNP (M)
0.49 0.01
1.54
10.01
LNP (M)
0,64 10.00
F
1.52 10.00
CNP
0.54 20.01
F
1.41
20.01
CNP
0.66 #0.01
F
1.28 40.00
CNnvLNP (E-M)
0.90 20.04
0.90
=0.04
4.05
=0.00
Trivial
1.31 +0.00
1.31 40.00
1.07 +0.00
Trivial
1.19 20.00
1.19
20.00
0.96
20.00
ConvL (F-M) 2.14 #0.06
2.14 40.06
9.30 20.08
33

---

Published as a conference paper at ICLR 2023
Table For the EQ synthetic experiments
two-dimensional inputs, average Kullback-Leibe divergences of the posterior prediction map Ty with respect to the model normalised by the number of target points. Shows for one-dimensional o (dy 1) and two-dimensional outputs (d, = 2) the performance for interpolation within the range [-2, 2 where the models where trained ("Int."); interpolation within the range [2, 6) which the models have never before ("OOID"); and extrapolation from the range [-2, the range [2, 6]² ("Ext,"), Models are ordered by interpolation performance. The latent variable models are trained and evaluated with the ELBO objective (E); trained and evaluated with the ML objective (M); and trained with the ELBO objective and evaluated with the ML objective (E-M). Diagonal GP refers to predictions by the ground-truth Gaussian processes without correlations. Trivial refers to predicting the empirical means and standard deviation test data. Errors indicate central % interval. Numbers which are significantly best (p < 0.05) are boldfaced. Numbers which are very large are marked as failed with "F". Numbers which are missing could not be run.
EQ d = 2, dy
EQ d₂ = 2, dy =2
Int.
OOID
Ext.
Int.
O0ID
Ext.
CNnvCNP (AR)
0.01 10.00 0.01 10.00
0.01 10.00
ConvCNP (AR) 0.03 10.00
0.03 10.00
0.02 10.00
ConvGNP
0.08 00.00 0.08
10.00
1.92 20.02
ACNP (AR)
0.11
10.00 0.79
10.00
0.56:10.0 10.00
AGNP
0.09 10.00 0.70
0.50 10.000000
ConvGNP
0.19
0.19
0.74
ACNP (AR)
0.09 00 0..0 0.51
AGNP
0.22 00.0
0.57 10.00
GNP
0.19 00 1.01 #0.00 0.80 20.00
GNP
0.38 20.00
20.00
0.75 20.00
ConvLNP (M)
0.31 20.00 0.31
20.00 0.17 20.00
ConvNP (M)
40.00
0.52#0.00 20.00
ALNP (M)
0.34 00 0.70 10.00 0.51 20.00
Diagonal GP
0.40 10.00
0.40
10.00 0.56 10000
Diagonal GP
0.36 00.00 0.36
0.51
ConvCNP
0.41 #0.00
0.41
0.56 +0.00
CNnVCNP
0.37 00 0.37 40.00 0.51 #0.00
ANP (M)
0.42 +0.00
0.79 40.00 0.54 40.00
ACNP
0.40 00 0.72 20.00 0.51 20.00
ACNP
0.44 20.00
0.79 20 o0
0.56 10.00
ConvLNP (E)
0.41 20.00 0.41 20.00 0.46 20.00
CNP (AR)
0.52 20.00
F
CNP (AR)
0.41 20.00 0.90 20.00 0.71 20.00
NP (E-M)
0.56 00
1.95
10
0.72
LNP (E-M)
0.46
0.99 #0.01 0.65
ANP (E-M)
40.00
1.90 11.72 0.55
LNP (E)
0.48 00.00 1.04 20.01 0.67 20.00
NP (E)
0.57 #0.00
1.99
+0.03 03 0.72 :0.00
ConvLNP (E-M)
0.48 20.00 0.48 #0.00 0.59 20.01
ANP (E)
0.57 #0.00
3.51
#3.04 0.56 =0.00
ALNP (E-N) E-M)
0.49 10.01 0.72 10.00 0.51 20.00
NP (M)
0.59 10.00 1.17 10.01 0.75 10.00
ALNP (E)
0..0 0.73
0.52
CNP
0.60 10.00 3.08
0.66
LNP (M)
0.51 10.00 0.92 10.00 0.72 10.00
Trivial
0.79 10 00 0.79 10.00 0.56 10.00
CNP
0.52 00.00 0.92 20.00
0.72 20.00
ConvNP (E-M)
0.79 20.00 0.79 20.00 0.56 40.00
Trivial
0.72 00.00 0.72 20.00
0.51 20.00
ConvNP (E)
0.79 20.00 0.79 20.00 40.00
FullConvGNP
GNIIConvGNP
34

---

Published as a conference paper at ICLR 2023
Table 11: For Matérn 2 synthetic experiments with one-dimensional inputs, average Kullback-Leibler Leibler divergences of the posterior prediction map Ny with respect to the model normalised by the number of target points. Shows for one-dimensional outputs (dy 1) and wwo-dimensional outputs (dy = 2) the performance for interpolation within the range [-2, 2] where the models where trained ("Int."); interpolation within the range [2, 6] which the models have never seen before ("OOID"); and extrapolation from the range [-2, 2] to the range [2, 6] ("Ext."). Models are ordered by interpolation performance. The latent variable models are trained and evaluated with the ELBO objective (E); trained and evaluated with the ML objective (M); and trained with the ELBO objective and evaluated with the ML objective (E-M). Diagonal GP refers to predictions by the ground-truth Gaussian processes without correlations. Trivial refers predicting the empirical means standard deviation of the test data. Errors indicate the central 25%-confidence interval. Numbers which are significantly best (p < 0.05) are boldfaced. Numbers which are very large are marked as failed with "F". Numbers which are missing could not be run.
Matérn- 5
Matérn 5
Int.
OOID
Ext.
Int.
OOID
Ext.
dy 1, dy =1
d2 1, dy =2
FullConvGNP
0.00 10.00 0.00
:0.00
0.00
10.00
FuIIConvGNP
0.00 =0.00 0.00 =0.00
0.00 =0.00
ConvCNP (AR)
0.00 -0.00 0.00
=0.00
0.00
#0.00
ConvCNP (AR)
0.01 #0.00 0.01 40.00
0.01 +0.00
ConvGNP
0.01 =0.00 0.01
2.32
#0.06
ConvGNP
0.02 0.02
1.71
AGNP
0.03 =0.00
4.53
20.08
7.22
10.12
AGNP
0.04 6.14 t0.0s
7.03 60.09
ACNP (AR)
0.04 1.08
00.01
0.87
20.01
ACNP (AR)
0.05 10.00
1.18 10.01
0.96 10.01
GNP
0.09 00.00
F
F
GNP
0.13 10.00
F
F
ConvLNP (E)
0..0
0.13 10.00
0.31
ConvLNP (E)
0.16 0.16 10.00
0.29
ConvLNP (M)
0.26=0.0
0.26
00.01
0.58 #0.00
ConyLNP (M)
0.36 10.00
0.36
10.00
0.76 1:0.00
ALNP (M)
0.30=0.00
0.98
=0.01
0.78
=0.01
ALNP (M)
0.40
1.10
0.88 20.00 F
CNP (AR)
0..4
1.81
20.04
1.32 20.02
20.02
CNP (AR)
0.40
F
LNP (E)
00.00
F
1.31 20.01
Diogonal GP
0.46 0.46
0.93 20.00
LNP (E-M)
0.37
F
1.14
ConvCNP
0.46
0.93
Diagonal GP
0.40 00.01
0.40
#0.01
0.84 +0.01
ACNP
0.49 40.01
1.23 40.01
0.93 40.00
ConvCNP
0.40 .0.11
0.40
#0.01
0.84
#0.01
ALNP (E-M)
0.51 40.0
1.28
4o..n
0.99 +0.01
ALNP (E-M)
0.41 00.01 1.13
20.01
0.84 10.01
ALNP (E)
0.51 10.01
1.43
40.02
1.10 10.01
ACNP
0.42 00.1 10.01
0.84 10.01
LNP (E M)
0.500
F
1.24 10.00
ALNP (E)
1.15
0.87
#0.01
LNP (E)
1..00
F
1.79 10.01
LNP (M)
0.51 20.00
1.87 20.02
1.30
=0.01
LNP (M)
0.63 #0.00
2.33
40.02
1.23 40.00
CNP
0.54 =0.01
1.47
20.02
1.11
#0.01
CNP
0.65 +0.00
7.72 #0.00
1.23 +0.00
Trivial
1.08 20.00
1.08
20.00
0.85
20.00
Trivial
1.18 #0.00
1.18
+0.00
0.94 +0.00
ConvLNP (E M) 1.37 00.04
1.36
4.30
20.06
ConvLNP (E MM 3.07 10.06 3.06
40.06
9.83 10.09
35

---

Published as a conference paper at ICLR 2023
Table 12: For the Matérn- synthetic experiments with two-dimensional inputs, average Kullback-Leible divergences of the posterior prediction map Ny with respect the model normalised by the number of target points. Shows for onc-dimensional outputs (dy
and two-dimensional outputs (d, 2) the performance for interpolation within the range |- where the models where trained ("Int."); interpolation within the range [2, 6] which the models have never seen before ("OOID"); and extrapolation from the range [-2, 2) to the range [2, 6]2 ("Ext."). Models are ordered by interpolation performance. Diagonal GP refers to predictions by the ground-truth Gaussian processes without correlations. Trivial refers to predicting the empirical means and standard deviation of the test data. Errors indicate the central 95%-confieence interval. Numbers which are significantly best (p < 0.05) are boldfaced. Numbers which are very large are marked failed with "F". Numbers which are missing could not run.
Matérn- d,,, 22 dy =1
Matérn 5
Int.
OOID
Ext.
Int.
OOID
Ext.
d2 =2=2 2, dy =2
ConvCNP (AR)
0.01 20.00
0.01
00.00
0.00 20.00
ConvCNP (AR)
0.01 20.00 0.01 =0.00
0.01 10.00
ACNP (AR)
0.05 =0.00 0.54
0.38 20.00
ACNP (AR)
0.06 10.00 0.58 10.00
0.41 10.00
AGNP
0.08 =0.00 0.83
=0.01
0.37 #0.00
ConvGNP
0.14 0.14 0.64 +0.01
ConyGNP
0.08 =0.30 0.08
=0.00
0.60
AGNP
0.17 40.00 0.58
0.40 40.00
GNP
0.16 20.00 20.00 0.75 +0.00
GNP
0.28 #0.00 0.78 40.00 0.60 +0.00
ConvLNP (M)
0.25 0.00 0.25
10.00
0.34 00.00
ConvLNP (M)
0.29 10.00 0.29 10.00 0.38 10.00
ALNP (M)
0.26 0.00 0.51 10.00
0.37 10000
ALNP (M)
0.29 10.00 0.56 10.00 0.40 10.00
Diagonal GP
0.28 0.28
0.39
Diagonal GP
0.30 0.30 10.00 0.41
COnVCNP
0.28 0.28 10.00 0.39 00.00
ConvCNP
0.30 10.00 0.30 10.00 0.41 1:0.00
ACNP
0.29 0.00
0.39 =0.00
ACNP
0.32 20.00 0.58 10.00
0.41 20.00
CNP (AR)
0.31 =0.00 0.00
0.52 =0.00
ConvLNP (F)
0.36 20.00 0.36 40.00
0.37 20.00
ConvLNP (E)
0.32 =0.00 0.32
0.30 20.00
CNP (AR)
0.37 10-00
F
0.88 10.17
LNP (E-M)
0.34 1.07
0.69
LNP (E-M)
0.41 40.00 2.29
0.59
LNP (E)
0.35 -0.00 1.25 00.01
0.72 +0.00
LNP (E)
0.41 40.00 2.36 40.0s
0.60 40.00
ConvLNP (E-M)
0.36 -0.00 0.36
#0.00
0.43 #0.00
ALNP (E-M)
0.42 +0.00 0.61 40.00 0.41 +0.00
LNP (M)
0.37 00.0 0.75
0.51 10.00
ALNP (E)
0.42 0.61 40.00 0.41 10.00
ALNP (E-M)
0.39
10.01
0.43 10.00
LNP (M)
0.43 0.68 10.00 0.53 1:
CNP
0.39 0.67
0.54 #0.00
CNP
0.44 0.86 0.59 10.00
ALNP (E)
0.41 20.01 0.67
20.01
0.44 +0.00
ConyLNP (E-M)
0.49 #0.00 0.49 40.00
0.61 40.00
Trivial
0.55 00.00 0.55
20.00
0.39
=0.00
Trivial
0.58 +0.00 0.58 +0.00
0.11 t0.00
FullConvGNP
FullConvGNP
36

---

Published as a conference paper at ICLR 2023
Table 13: For the weakly periodic synthetic experiments with one-dimensional inputs. average Kullback-Leibbe divergences of the posterior prediction map Ny with respect to the model normalised by the number of target points. Shows for one-dimensional outputs (dy 1) and wwo-dimensional outputs (dy = 2) the performance for interpolation within the range [-2, 2] where the models where trained ("Int."); interpolation within the range [2, 6] which the models have never seen before ("OOID"); and extrapolation from the range [-2, 2] to the range [2, 6] ("Ext."). Models are ordered by interpolation performance. The latent variable models are trained and evaluated with the ELBO objective (E); trained and evaluated with the ML objective (M); and trained with the ELBO objective and evaluated with the ML objective (E-M). Diagonal GP refers to predictions by the ground-truth Gaussian processes without correlations. Trivial refers predicting the empirical means standard deviation of the test data. Errors indicate the central 95%-ccnnfidenc interval. Numbers which are significantly best (p < 0.05) are boldfaced. Numbers which are very large are marked as failed with "F". Numbers which are missing could not be run.
Weakly Periodic dy = 1, dy =1
Weakly Periodic
Int.
OOID
Ext.
Int.
OOID
Ext.
d 1, dy =2
FullConvGNP
0.02 20.00 0.02 10.00
0.00 10.00
FuNIConvGNP
0.03 20.00 0.03 =0.00
0.00 10.00
ConyCNP (AR)
0.05 =0.00
0.05
=0.00
0.04 10.00
ConvCNP (AR)
0.09 10.00 0.09 10.00
0.06 10.00
ConvGNP
0.05 10.00 0.56 10.02
ConvGNP
0.12 0.12 10.00
0.72 10.01
AGNP
1.25 1.25 =0.02
AGNP
0.20 2.17 #0.02 1.95
ConvLNP (M)
0.28
0.28 20.00 0.43 =0.00
ConvLNP (M)
0.38 #0.00 0.38 #0.00 0.54 20.00
ConvLNP (E)
2.00 0.000 0.45 =0.02
ConvLNP (E)
0.39 20.00 0.39 40.00 0.44 +0.00
Diagonal GP
0.38 =0.01 0.38 10.01 0.59 10.01
Diagonal GP
0.42 10.00 0.42 10.00 0.65 1:0.00
ConvCNP
0.40 =0.0 #0.01 0.60
ConvCNP
0,46 40.00 0.46 40.00 0.65
ALNP (M)
0,53 =0.00 0.77 #0.01
0.57 #0.01
GNP
0.50 40.00 1.02 40.01 0.76 4.000
ACNP (AR)
0.57 =0.01 20.01
0.61 =0.01
ALNP (M)
0.62 10.00 1.04 10.01 0.64 20.00
GNP
0.59 =0.01 1.31 =0.02
0.62
=0.01
ACNP (AR)
0.63 10.00 0.89
+0.01
0.66 1:0000
CNP (AR)
0.59 =0.01 2.33
20.27
1.46
20.05
CNP (AR)
0.67 10.00
2.52
10.07
1.21 10.01
LNP (E-M)
0.60 u0.01
F
4.09
LNP (E-M)
0.68 40.00
F
5.18
ALNP (E-M)
0.60 =0.01 0.78
00.01
0.59
#0.01
LNP (M)
0.69 #0.00
1.26
40.01
0.68 +0.001
LNP (M)
0.60 0.80 20.01
0.62
#0.01
LNP (E)
0.69 +0.00
F
F
LNP (E)
0.61 00001
F
9.91 10.70
ALNP (E-M)
0.70 10.00
0.85
10.01
0.64 1:0.00
ALNP (E)
0000 1.01
0.71
ACNP
0.71 10.00
0.89 10.01
0.66 10.00
ACNP
0.65 00.01 0.82 10.01 0.61 #0.01
ALNP (E)
0.72 10.00 0.93 10.01 0.72 10.00
CNP
0.67 20.01 1.45
03.0 0.68 =0.01
CNP
0.74 #0.00 1.27 40.01
0.77 20.01
Trivial
0.82 20.00 0.82
0000 0.61 #0.00
Trivial
0.89 +0.00 0.89 +0.00
0.67 40.00
NP (F-M) 1.58 =0.03
1.57
20.0
2.85 20.04
ConvL (F-M) 3.02 20.03 3.02 40.03
20.04
37

---

Published as a conference paper at ICLR 2023
Table For the weakly periodic synthetic experiments with two-dimensional inputs, average Kullback-Leibe divergences of the posterior prediction map Ny with respect to the model normalised by the number of target points. Shows for mneediiennional outputs (dy 1) and wo-diimnsiional outputs (d, = 2) the performance for interpolation within the range |-2, 2/a where the models where trained ("Int."); interpolation within the range [2, 6] which the models have never seen before ("OOID"); and extrapolation from the range [-2, 22 2 to the range [2, 6/ ("Extt"). Models are ordered by interpolation performance. The latent variable models are trained and evaluated with the ELBO objective (E); trained and evaluated with the ML objective (M); and trained with the ELBO objective and evaluated with the ML objective (E-M). Diagonal GP refers to predictions by the ground-truth Gaussian processes without correlations. Trivial refers predicting the empirical means and standard deviation of the data. Errors indicate the central "oe-conideece interval. Numbers which are significantly best (p < 0.05) are boldfaced. Numbers which are very large are marked as failed with "F". Numbers which are missing could not be run.
Weakly Periodic dy =:=2 2, dy =1
Weakly Periodic
Int.
OOID
Ext.
Int.
OOID
Ext.
d 2, dy =2
CNn (AR)
0.05 20.00 0.05 10.00
0.03 10.00
ConvCNP (AR)
0.08 20.00 0.08 :0.00
0.05 10.00
ConyGNP
0.10 =0.00 0.10
0.19 20.00
ConvGNP
0.13 10.00 0.13 10.00 0.18 10.00
ConvLNP (M)
0.18
0.21 10.00
Diagonal GP
0.20 0.20 10.00 0.28
Diagonal GP
0.19 0.19 0.27 =0.00
ConvLNP (M)
0.21 +0.00 0.21 +0.00 0.25
CNn
0.20 =0.00 0.20 0.27 =0.00
ConvCNP
0.23 0.23 20.00 0.28 20.00
ConyLNP (E)
0.22 =0.00 0.22 0.22 =0.00
AGNP
0.20 0.39 40.00 0.27 20.00
AGNP
0.23 0.00 0.35 20.00 0,26 10.00
ACNP (AR)
0,25 10.00 0.40 10.00 0.28 1:0.00
ACNP (AR)
0,23 00.00 0.37
0.26
GNP
0,25 40.00 0.38
0.25
ConvLNP (E-M)
0,23 00.00 0.23 00.00 0,29 +0.00
ConvLNP (E)
0,26 10.00 0.26 40.00 0.27 4.0.00
GNP
0.23 0.35
0.24 =0.00
CNP (AR)
0.27 3.21 10.14 0.77 20.02
CNP (AR)
0.25
0.82
0.72 =0.01
ALNP (M)
0.28 10.00 0.39 40.00 0.28 10.00
ALNP (M)
00.00
0.25
LNP (M)
0.29 0.42
0.31 40:00
LNP (M)
0.26 =0.00
=u.0a
0.33 +0.00
LNP (E-M)
0.29 40.00 1.52 40.04 0.36
LNP (E-M)
0.26 -0.00 6.28
11.0 0.44 +0.01
LNP (E)
0.30 #0.00 1.60 to.0p 0.39 40.00
LNP (E)
0.27 -0.00 9.91
+1.08 0.52 #0.01
ConvLNP (E-M)
0.30 +0.00 0.30 40.00 0.34 t0.00
ACNP
0.28 0.00 0.37
10.00 0.27 10.00
ACNP
0.30 10.00 0.40 10.00 0.29 1:0.0
CNP
0.29 00.00 0.81
10.01 0.29
CNP
0.31 10.00 0.63 10.01 0.33
ALNP (E-M)
0.31 0.00 0.39
10.00 0.26 10.00
ALNP (E M))
0.36 10.00 0.43
0.28 10.00
ALNP (E)
0.32 00.00
00
0.29 20.00
ALNP (E)
0.36 #0.00 0.44 40.00
0.29 20.00
Trivial
0.38 00.00
0.38
00.00
0.27
#0.00
Trivial
0.40 +0.00 0.40 +0.00 0.28 20.00
FulIConvGNP
FallConvGNP
38

---

Published as a conference paper at ICLR 2023
Table 15: For the sawtooth synthetic experiments with one-dimensional inputs, average log-likelihoods malised by the number target points. Shows for one-dimensional outputs (dy = 1) two-dimensional outputs (dy = 2) the performance for interpolation within the range [-2,2] where the models where trained ("Int."): interpolation within the range [2. 6] which the models have never seen before ("OOID"): and extrapola- ordered by interpolation performance. The latent variable models are trained and evaluated with the ELBO objective (E); trained and evaluated with the ML objective (M); and trained with the ELBO objective and evaluated with the ML objective (E. Trivial refers to predicting the empirical means and standard deviation of the test data. Errors indicate the central 95%-confidence interval. Numbers which significantly best (p 0.05) are boldfaced. Numbers which are very large marked failed with "F"" Numbers which missing could be run.
tion from the range [-2, the range [2, ("Ext"). Models
Sawtooth
Sawtooth
Int.
OOID
Int.
OOID
Ext.
d, dg =1
d₂ =1=1, 1, d =2
Conv (AR)
s.60
3.60
3.34
ConvCNP (AR)
2.22=0.01
1.91 2.22+t0.1
ConvLNP (E)
3.51+0.02
3.52+o.02
2.68+0.02
ConvLNP (E)
2.01
2.01=1.01
1.47+0.01
ConvLNP
3.06+0.04
3.06#t..0
0.64++0.11
ConvLNP
1.73
1.73#a..03
-0.12+0.01
ConvGNP
2.6210.05
2.6110.08
-0.04410.1
ACNP (AR)
1.01:0.01
-0.47:0.01
-0.4510.01
ConyCNP
2.38:10.4
2.37:10.01
-0.0010.01
FulIConvGNN
0.99
1.04:0.03
0.11:0.00
FullConvGNP
2.1610.04
2.1510.04
0.18110.01
ConvCNP
0.83
0.84
-0.2910.00
ALNP (E-M)
0.2740.01
-0.1840.00
-0.02
ConvGNP
0.8220.03
0.82=0.03
-0.29+0.00
ALNP (F)
0.2740.01
-15.96+0.17
-9.14120.04
GNP
0..3
F
ALNP
0.20+0.00
-0.18+0.00 F
-0.18+0.00
AINP (F-M)
-0.0620.01
-0.75=0.01
-0.35=0.00
LNP (E-M)
0.07+4.01
8.67
ALNP (E)
-0,07
-0.78=0.02
-0.35
LNP (E)
0.06+0.01
F
AGNP
00.07
-0.37
-0.48
Trivial
0.18
-0.18+0.00
-0.18+0.00
ACNP
-0.08+0.01
-0.41+0.01
-0.42+0.01
LNP
-0.1840.00
-0.1810.00
-0.1810.00
LNP (E-M)
-0.2020.00
-1.3320.00 F
CNP (AR)
-0.1810.00
-0.1810.00
-0.1810.00
LNP (E)
-0.20=0.00
F
-1.3320.00
CNP
-0.1810.00
-0.1810.00
-0.1840.00
ALNP
-0.2810.00
-0.33.0.00
-0.3310.00
GNP
0.18
0.1884..00
-0.18+0.00
CNP (AR)
0.29
-1.96=0.19
-1.43+0.06
AGNP
0.1000000
0.1846..0
-0.18+60.00
CNP
0.30
-0.51
-0.34+0.00
ACNP
0. 18
0..0000000
0.18
LNP
0.32
-0.32=0.00
-0.32+0.00
ACNP (AR)
1.18000000
08.000000
-0.1816.00
Trivial
00.0
-0.3310.00
-0.3310.00
ConvLNP (E M)
F
F
ConvLNP (E M) 10.9
-2.98
-6.74.0.04
39

---

Published as a conference paper at ICLR 2023
Table 16: For the sawtooth synthetic experiments with two-dimensional inputs, average log-likelihoods malised by the number target points. Shows for one-dimensional outputs (dy = 1) and two-dimensional outputs (d, = 2) the performance for interpolation within the range [-2,2] where the models where trained ("Int."); interpolation within the range [2, 6] which the models have never seen before ("OOID"); and extrapo- lation from the range [- 2] the range [2. 6] ("Ext."). Models ordered by interpolation performance. The latent variable models are trained and evaluated with the ELBO objective (E); trained and evaluated with the ML objective (M); and trained with the ELBO objective and evaluated with the ML objective (E (E-M). Trivial refers to predicting the empirical means and standard deviation of the test data. Errors indicate the central 95%-confidence interval. Numbers which significantly best (p 0.05) are boldfaced. Numbers which are very large marked failed with "F". Numbers which missing could be run.
Sawtooth d₂ =2, dy =1
Sawtooth
Int.
OOID
Ext.
Int.
OOID
Ext.
d, = dg =2
ConvCNP (AR)
2.59 #d.01
2.59
#0.01
2.10
#0.01
ConvCNP (AR)
0.38 +0.00
0.38 20.00
0.18 +0.00
ConvLNP (M)
2.07 =0.02
2.08
+0.02
-0.17
ConvLNP (M)
0.31 20.01
0.31 +0.01
-0.32
=0.00
ConvCNP
1.93 20.04
1.94
+0.03
-0.18
#0.00
ConvGNP
0.26 20.01
0.26 20.01
-0.33 =0.06
ConvGNP
1.90 20.04
1.91 10.03
-0.18
10.00
ConvCNP
0.12
0.12 10.01
-0.32 00.00
ConvLNP (E)
1.77
1.77 10.02
0.33
10.02
ConvLNP (E)
0.04
0.04 10.00
-0.30 =0.00
ConvLNP (E-M)
10.04
1.72 10.04
-2.30
10.87
ConvLNP (E MM)
-0.07 10.01
-0.07 10.01
-0.48
Trivial
-0.18
-0.18 10.00
-0.18
Trivial
-0.32
-0.32
-0.32 =0.00
CNP (AR)
-0.18
-0.18
10.00
-0.18
+0.00
ALNP (M)
0.32
-0.32
20.00
-0.32 =0.00
CNP
-0.18
-0.18 +0.00
-0.18
#0.00
CNP (AR)
-0.32
-0.32
20.00
-0.32 =0.00
GNP
-0.18
-0.18 +0.00
-0.18
+0.00
CNP
-..3
-0.32
40.00
-0.32 00000
LNP (M)
-0.18
-0.18
-0.18
LNP (M)
-0.32 40.00
-0.32
-0.32
AGNP
-0.18
-0.18
+0.00
-0.18
40.00
ACNP
-0.32 20.00
-0.32
+0:00
-0.32 00.00
ALNP (M)
-0.18
-0.18
10.00
-0.18
+0.00
ACNP (AR)
0.32
-0.32
10.00
-0.32 00.00
ACNP (AR)
-0.18
-0.18
10.00
-0.18
=0.00
GNP
-0.2
-0.32
10.00
-0.32 00.00
ACNP
-0.18
-0.18
-0.18 10.00
AGNP
-0.32
-0.32
-0.32
LNP (E-M)
0.19
0.86
LNP (E-M)
0.33
-0.54
-0.34
LNP (E)
-0.19 20.00
LNP (E)
0..30
-0.44
-0.34 =0.00
ALNP (E-M)
-0.20 20.01
0.118
0.18
#0.00
ALNP (E-M)
0.36
-0.33
+0.00
-0.33 =0.00
ALNP (E)
-0.20 10.01
-0.71
10.00
-0.33
1:000
ALNP (E)
-0.36 10.00
-0.33 10.00
-0.33 0.00
FullConvGNP
FuIIConvGNP
40

---

Published as a conference paper at ICLR 2023
Table 17: For the mixture synthetic experiments with one-dimensional inputs, average log-likelihoods normalised by the number of target points. Shows for one-dimensional outputs (d, 1) and two-dimensional outputs (dy = 2) the performance for interpolation within the range [- where the models where trained ("Int.);; interpolation within the range [2, 6] which the models have never seen before ("OOID"): and extrapolation ordered by interpolation performance. The evaluated with the ELBO objective (E); trained and evaluated with the ML objective (M); and trained with the ELBO objective and evaluated with the ML objective (E. Trivial refers to predicting the empirical means and standard deviation of the test data. Errors indicate the central 95%-confidence interval. Numbers which significantly best (p 0.05) are boldfaced. Numbers which are very large marked failed with "F". Numbers which missing could be run.
from the range 2- 2] the range [2. 6] ("Ext."). Models
latent variable models are trained
Mixture de =1, dy =1
Mixture
OOID
Ext.
Int.
OOID
Ext.
d, 1, dg =2
Con (AR)
0.45+t0.0
0.45
0.30
ConvLNP (E)
-0.05
0..55
0.50
ConvLNP (E)
0.12#0.04
0.12#0.04
-0.37
ConvCNP (AR)
-0.17=a.uz
-0.17+0.02
-0.29=0.02
FullConv GNP
-0.05=0.03
-0.05#t..0
-0.4940.01
FullConvGND
-0.27
-0.27+0.01
-0.63=0.01
ConvLNP
-0.06.0.03
-0.0611.03
-0.88:10.2
ConvGNP
-0.29-0.02
-0.2910.02
-2.59=0.06
ACNP (AR)
-0.1910.02
-1.3210.01
-1.3210.01
ACNP (AR)
-0.35:0.02
-1.4310.02
-1.43=0.02
ConvCNP
-0.2310.04
-0.2410.04
-1.23::0001
ConvLNP
-0.39.0.03
-0.4010.03
-1.10=0.02
CNn
-0.2420.22
-0.2340.02
-1.00=0.02
AGNP
-0.57
-3.2920.07
-3.05=0.06
AGNP
-0.41=0.02
-1.0326.03
-2.6120.66
GNP
0.60-a..
-1.7020.03
-1.67
AINP
-0.61=0.02
-1.01
-1.03#n.02
ConvCNP
-0.68
-0.6820.02
-1.39=0.01
ALNP (E-M)
00.6
-1.24...11
-1,10=0.02
ALNP
0.76
-1.1940.22
-1.19=0.02
ALNP (E)
0.67
-1.524..22
-2.25+0.03
LNP
-.89
-3.1840.05
-1.40=0.02
LNP (E-M)
-0.68=u.11
-3.05+e.04
-1.75+0.01
LNP (E-M)
-0.89
-4.43+0.07
-1.62=u.02
GNP
-0.70=0.02
-2.2210.05
-1.6220.04
ALNP (E-M)
-0.9220.2
-1.4710.03
-1.46=0.09
LNP (E)
-0.71
-3.7140.05
-2.0820.01
LNP (E)
-0.92.0.01
-4.71:0.07
-1.86=0.01
LNP
-0.7220.01
-1.4610.01
-1.2910.02
ACNP
-0.93.0.22
-1.4210.02
-1.5320.09
ACNP
0.79
-1.3140.01
-1.25+0.01
ALNP
0.95
-1.75+0.04
-1.73=0.04
CNP (AR)
-1.00=0.02
-1.15#6.02
-1.0940.02
CNP (AR)
-1.1520.2
-1.27+0.02
-1.21
CNP
-1.05
-1.1440.02
-1.15+0.02
CNP
-1.18=0.2
-1.32+0.03
-1.27=0.02
CN (E M)
-1.41
-1.4310.13
-3.4010.08
ConvLNP (E M)
-3.45.0.0r
-3.4010.77
-5.30=0.0r
Trivial
-1.3210.00
-1.3210.00
-1.32.0.00
Trivial
-1.46=0.00
-1.4610.00
-1.46=0.00
41

---

Published as a conference paper at ICLR 2023
Table 18: For the mixture synthetic experiments with two-dimensional inputs, average log-likelihoods normalised by the number target points. Shows for one-dimensional outputs (d, 1) and two-dimensional outputs (dy = 2) the performance for interpolation within the range 2_, where the models where trained ("Int."); interpolation within the range [2, 6] which the models have never seen before ("OOID"): and extrapolation from the range [-2, 2) to the range [2, 6] ("Ext.".. Models ordered by interpolation performance. The latent variable models are trained
evaluated with the ELBO objective (E); trained and evaluated with ML objective (M); and trained with the ELBO objective and evaluated with the ML objective (E (E. Trivial refers to predicting the empirical means and standard deviation of the test data. Errors indicate the central 95%-confidence interval. Numbers which are significantly best (p 0.05) are boldfaced. Numbers which are very large marked failed with "F". Numbers which missing could be run.
Mixture
Mixture
Int.
OOID
Ext.
Int.
OOID
Ext.
d₂ =2, dy =1
d, 2, dg =2
ConvCNP (AR)
0.100 #0.08
0..10
20.03
0.34
#0.03
Conv CNP (AR)
-0.62 #0.01
-0.62
20.01
-0.79 #0.01
ConvCNP
-0.49
-0.49
+0.03
-1.45
+0.02
ConvGNP
-0.74
-0.74
-1.43
=0.02
ConvGNP
-0.50 20.02
-0.50
+0.02
-1.24
t0.02
ConvLNP (M)
-0.78 #0.02
+0.09
-1.25 =0.02
-1.25=0.02
ConvLNP (M)
-0.57
-0.57
10.02
-1.07 10.02
ConvCNP
-0.85
-0.85
-1.50=0.0
ConvLNP (E)
-0.63
-0.63
10.02
-1,08 10.02
ACNP (AR)
-0.85
-4.30
-4.24
ALNP (M)
-0.73
-1.04
-1.05 10.02
ALNP (M)
-0.88
-1.41
-1.21 00.02
ConvLNP (F-M)
-0.76
-0.76
-1.37
=0.02
ConvLNP
-..92
-0.92
-1.41 =0.02
ACNP (AR)
-0.77
-1.28
-1.30
AGNP
0..3
-1.34
-0.21
=0.02
AGNP
-0.78
-1.32
-1.32
#0.02
ACNP
-0.99
-4.19
-1.27=0.0
ACNP
-0.91
-1.29
-1.17 +0.02
ConvLNP (E-M)
-1.55
-1.55
-1.73=0.0
LNP (M)
-0.91
1.44
-1.19
LNP (M)
-1.77 40.11
-4.04
-1.31
LNP (E-M)
-0.92
-1.51 40.02
-1.38
40.02
ALNP (E-M)
-1.11
-2.65
-1,23 =0.02
LNP (E)
-0.93
-1.74
10.03
-1.44
10.02
ALNP (E)
-1.12
-8.2247.61
-1.26=0.02
ALNP (E-M)
-1.00
-1.07
-1.08
#0.02
GNP
-1.20
-1.39
-1.21
ALNP (E)
-1.03
-1.08
-1.09
10.02
CNP (AR)
-1.20
-2.61 -1.32_0.0
CNP (AR)
-1.06
-1.07
-1.08
CNP
-1.20
-2.00
-1.23
GNP
-1.06
+1.09
-1.08
40.02
40.02
LNP (E-M)
-1.20
-3.23
-1.24
CNP
-1.07 20.02
-1.09
-1.10
LNP (E)
-1.20 20.2
-3.25
-1.21 =0.02
Trivial
-133 10.00
-1.32
10.00
-1.32
10.00
Trivial
-1.66
-1.46
-1.46 0.00
FullConvGNP
FullConvGNP
42

---

Published as a conference paper at ICLR 2023
1 DETAILS OF SIM-TO-REAL TRANSFER EXPERIMENTS
I.1 DESCRIPTION OF EXPERIMENT
Our goal will be to make predictions for the famous hare-lynx data set. The hare-lynx data set is a time series from 1845 to 1935 recording yearly population counts of a population of Snowshoe hares and population of Canadian lynx (MacLulich, 1937). A digital version extracted from the original graph by MacLulich (1937) is available by Hundley. Hundley, the author of this digital source, says that other authors caution that the lare-lynx data is actually a composition of multiple time series, and presents the data with that caution. We, therefore, also present the data with this caution. Figure 12a visualises the hare-lynx data set.
To make predictions for the hare-lynx data set, we use the Lotka-Volterra equations (Lotka, 1910; Volterra, 1926), also called the predator-prey equations. The mathematical model for the population counts of prey population and predator population:
equations are an idealised
prey population:
x' (t) = (xr(t) (x(())((),,
(45)
predator population:
(t) = -d((t) + (z)()))(t
(46)
These differential equations say that the prey population naturally grows exponentially with rate a, and that the predator population naturally decays exponentially with rate 8. In addition, the predators hunt the prey. The resulting additional growth in the predator population and the resulting additional decrease in the prey population are both proportional to the product of the densities. In this idealised mathematical form, the population counts converge to a smooth, noiseless limit cycle and then perfectly track this limit cycle ever after. This is unlike real-world predator-prey population counts, which exhibit noisy behaviour and imperfect cycles. We therefore consider stochastic version of the Lotka-Volterra equations, given by the following two coupled stochastic differential equations:
dX,- = ax,, dl - 3Y,X, dl + x dw,").
(47)
dy, = dt + 5Y,X, dt + OY, aw?(?)
(48)
where W(I and W(2) are two independent Brownian motions. Compared to the Lotka-Volters
equations, (47) and (48) have two additional terms, ox," dw!") and oY," dw which introduce noisy behaviour. In these terms, multiplying by X and Y," makes the noise go zero when X, and Yg become small, ensuring that X, and Y; remain positive. In addition, we multiply by a parameter to control the magnitude of the noise, and raise X, and Y, to power > 0 to control how quickly the noise grows as X, and Y, grow. Namely, X, naturally grows exponentially, so, by adding noise of magnitude proportional behaviour, we choose v to be strictly less than one. Finally, to obtain a variety of magnitudes of population counts, we multiply the realisation with scale 17.
Xr, we risk large spikes in the prey population. To moderate this
After simulating from (47) and (48) a few times, we settle on v = 1.- For the remainder of the parameters, we simply manually play around with (47) and (48), settle on parameter ranges that look reasonable, and randomly sample parameters from those intervals. Table 19 summarises the sampling distributions for all parameters of (47) and (48). Figure 12b shows four samples from the proposed stochastic model.
To generate a meta-data set, we simulate (47) and (48) on a dense grid spanning 110 years, throw away the first 10 years, and retain between 150 and 250 data points for X, and Y, The numbers of data points and the locations of the data points are sampled separately for X, and Y,. Hence, whereas the hare-lynx data is regularly spaced and the populations are always simultancously observed, our simulator generates data at arbitrary and nonsimultaneous points in time. We split these data sets into context and target sets in three different ways. To train the models, for every batch, we randomly choose one of the interpolation, forecasting or reconstruction tasks by rolling a three-sided die. We will also perform these tasks on the real hare-lynx data; in that case, for interpolation, number of target points per output be between one and fifteen. The tasks on simulated and real data are similar, but slightly differ in the number of context and target points.
let the
To deal with the positivity of population counts, we transform the marginals of all models to distributions on (0, oo) by pushing the marginals through x + xx)..
See http://ppopp e. whi tman. edu edddd/ddd//// cow.dr// M2500033 lynxxarre.... txt.
43

---

Published as a conference paper at ICLR 2023
150
Hares
Lynx
Popullatoo (x1k)
100
19100
1920
Year
(a) Illustration of the hare-lynx data
150
Lynox
100
200
60
100
20
60
80
100
60
100
(b) Four samples from the proposed stochastic version of the Lotka-Volters Volterra equations (47) and (48). The parameters of (47) and (48) are sampled according to Table 19.
Figure 12: Hare-lynx data set and proposed stochastic simulator
Parameter
Distribution
Initial condition X-
Uni 100))
Initial condition Y-
Uni 100)
Unif (0.2, 0.8))
Unif 0.08))
Y
Unif [0.8, 1.2))
Unif (n0.0 04, 0.08))
Fixed to 1/6
Unif ( 10))
Unif
7
Table Sampling distributions
parameters of the stochastic version the Votte-Voltera equations
(47) and (48). These equations are simulated on a dense grid spanning [-10, 100]. The table also shows the distribution for the initial conditions at f = 10. To not depend too heavily on these initial conditions, the simulation results on 10, 0] discarded.
For this experiment, the learning rate is 1 10-4, the margin is 1, and the points per unit is 4. We trained the models for 200 epochs.
The convolutional models layer, the stride is one instead of two. For the kernel architecture the FullConvGNP, we reduce the points per unit and the number of channels in the U-Net by factor two.
U-Net architecture with seven layers instead of six where, in the first
44

---

Published as a conference paper at ICLR 2023
Int. (S)
For. (S)
Rec. (S)
(R)
(R)
Rec. (R)
FuIIConvGNP
-3.29 =0.02
-3.46 10.02
-3.79
20.02
-4.16
-4.28
=0.04
-4.45
10.00
Conv (AR)
-3.30 =0.02
33.47
40.02 -3.00
+0.02
-4.00
+0.03
-4.27
#0.03
-4.32
#0.01
NonvNP (ML)
-3.41 t0.02
-3.84
+0.02
-4.44
=0.02
-4.13
20.04
-4.45
t0.05
-4.54 +0.01
ConyGNP
-3.47
-3.65
+0.02
-4.15
-4.21
20.05
-4.82
-4.61
+0.01
ConvCNP
10.07
-4.06
10.02
-4.85
-4.17
-4.70
-4.97
ConvNP (ELBO)
-3.77
-4.12
-5.45
-5.47
10.07
-6.39
1:005
ANP (ML)
-4.09 10.02
-4.32
10.02
-4.55
-4.31
-0.04
10.04
-4.49
10.01
ANP (ELBO-ML)
-4.22 #0.02
-4.54
+0.02
-1.80
-4.58
-4.58
+0.04
-4.68 +0.0
ACNP (AR)
20.03
16.02
-4.58
-4.40
-4.55
4.59
ANP (ELBO)
-4.32 10.03
-4.58
-0.0
-4.71 =0.5
-4.63
10.00
ACNP
-4.34 #0.02
-4.65
#0.02
-4.88
#0.02
-4.43
20.04
-4.58
40.04
-4.74
+0.00
ConvNP (E.-M.)
-6.71 +0.05
-8.44 40.11
F
-7.20
#0.31
F
Table 20: Normalised log-likelihoods in the predator-prey experiments. Shows the performance for interpolation ("Int.), forecasting ("For!"), and reconstruction ("Rec.") on simulated ("S") and real ("R") data. Models are ordered by interpolation performance on simulated data. The latent variable models are trained and evaluated with the ELBO objective (E); trained and evaluated with the ML objective (M); and trained with the ELBO objective and evaluated with the ML objective (E MM. Errors indicate the central -5%-iiideeecce interval. Numbers which are significantly best (p 0.05) are boldfaced. Numbers which are very large are marked as failed with "F", Numbers which are missing could not run.
1.2 FULL RESULTS
In Table 20, we present the full results for the sim-to-real experiments.
45

---

Published as a conference paper at ICLR 2023
10
-10
10
12
F3
10
20
20
0.2
0.1
1.0
Time
Figure Example of trial the EEG data set. Note that the signals all electrodes appear correlated, are subtly different.
J DETAILS OF ELECTROENNCEPHALOGR.APP EXPERIMENTS
We explore an electreencephalography data set collected from 122 subjects (Begleiter, 2022). There are two groups of subjects: alcoholic and control. Every subject was subjected to a single stimulus or two stimuli, and their response These measurements are in terms of trials, where a trial consists of 256 samples of the electrodes spanning second. The data sets contains up 120 trials for each subject. The data is available at https: / //aaarhiiii ics uci, edu/ml /m1 / datasatss/eeeeee described detail by Zhang et (1995). In this experiment, we focus only on seven frontal electrodes: FZ, FI, F2, F3, F4, F5, and F6. Figure 13 illustrates trial of subject, showing the samples for these seven electrodes.
measured with 64 electrodes placed
subject's scalp,
dat abase and the collection
We randomly split all subjects into three sets: an evaluation set consisting of ten subjects, cross- validation set consisting of other subjects, and a training set consisting of all remaining subjects. For each of these sets, we create a meta-data set by aggregating the trials for all subjects. We split every trial into a context and target set the same three ways as for the predator-pres experiment. First, for all seven electrodes separately, randomly designate between 50 and 256 points to be the target points and let the remainder (between and 206) be the context points. This task called interpolation and is the primary measure of performance. Additionally, randomly choose of the seven electrodes and, for that choice, split the data in two exactly like for forecasting. For all other electrodes, append all data the context set. This task is called reconstruction and measures the model's s ability to infer a signal for one electrode from the others. We train all models on the interpolation task, and evaluate the models the interpolation and reconstruction task.
For this experiment, the learning rate is 5 10-5 the margin is 0.1, and the points per unit is 256. We trained the models for 1000 epochs. For the FullConvGNP, the learning rate 2 10-4 The training run for the FullConvGNP was terminated after 84 hours, reaching epoch 127.
The convolutional models use U-Net architecture where, in the first layer, the stride is instead of two. In addition, the number of channels are adjusted as follows: the ConvCNP and ConvGNP use 128 channels, the ConvLNP uses 96 channels, and the FullConyGNP uses 64 channels. The length scales of the Gaussian kernels of the convolutional model is initialised to 0.77 1772566. To scale to seven outputs, the deep set- bet-based and attentive models reuse the same encoder for every output dimension.
46

---

Published as a conference paper at ICLR 2023
K DETAILS OF ENVIRONMENTAL DATA ASSIMILATION EXPERIMENT
In this section provide further details on the Antarctic cloud cover data assimilation experiment described in Section 4.4.
K.1 DATA CONSIDERATIONS
Data sources. Daily-averaged cloud cover reanalysis data was obtained from ERA5 (Hersbach al., 2020). An Antarctic land mask and elevation field obtained from the BedMachine dataset (Morlighem, 2020).
Figure 14 shows empireal density of the ERA5 cloud cover values calculated over the models' training period of 2000-2013. The spikes at and correspond largely to values of exactly 0 and exactly 1. This motivates the beta-categorical likelihood described in Section 4.4.
10a
Density
100
0.0
0.2
0.4
0.6
0.8
1.0
Cloud cover fraction
Figure 14: Empirical density of ERA5 cloud fraction computed over the period 2000-2013.
Data preprocessing. The cloud cover data and land/elevation auxiliary data were regridded from lat/lon Southern Hemisphere Equal Area Scalable Earth (EASE2) grid 25 resolution and cropping to size of 280 280. This centres the data on the South Pole.
Data normalisation. We normalised the data before passing to the convolutional NP models. The cloud cover and land mask data already took appropriate normalised values in [0, 1]. The elevation field was normalised from metres to values in [0, 1].
The input coordinates were normalised from metres to take values in [-1, 1].
K.2 MODEL CONSIDERATIONS
Here provide details on the training procedure and architectures for each of the convolutional NP models in the Antarctic data assimilation experiment.
Generating the training, validation, and test tasks. Following meta-leaming principles, collect data from day 7 into a task D,. Each task D, was generated by first drawing the integer number of simulated cloud cover context points No 2 2,. 5003. Letting N vary encourages the model to learn to deal with both data-sparse and data-rich scenarios. The number of target points N fixed value of 2,000.
Next, the input locations x and x were sampled uniformly at random across the entire 280 x 280 input space and the corresponding y, values were sampled without observation noise.
For the training dates, the random seed used for generating D, is updated every epoch, allowing for infinitely growing simulated training data set. In contrast, for the validation and test dates,
47

---

Published as a conference paper at ICLR 2023
the random seeds were held fixed so that metries computed over the validation and test sets not stochastic.
Training procedure. Each model was trained for 150 epochs 14 years of data from 2000-2013, An Adam optimiser was used with a learning rate of 5 x 10 and a batch size of 2. For the loss functions
negative log-liked oogllikelhood loss function for the ConvCNP and ConvGNP. For the ConvLNP we use the ELBO objective and fix the variance of the observation noise to 0.01 for the first four epochs. Validation data from 2014-2017 was used for checkpointing the model weights using the per-datapoint predictive log-likelihod.. The two year period of 201 8-2999 data was reserved for the test set.
The time taken to train each model on a Tesla V100 GPU is as follows:
ConvCNP: 25.0 hours,
ConvGNP: 27.5 hours,
ConvLNP 43.6 hours.
Architectures. For each model, the U-Net component of the encoder uses 5 x 5 convolutional kernels with the following sequence of channel numbers (d.s. 2 2 downsample layer, u.s. 2 upsample layer): 16 d=, 32 d.s, 64 d.s 128 - 64 U.S. 32 u.S. 16. We use bilinear resize operators for the upsampling layers to fix checkerboard artifacts that we encountered when using standard zero-padding upsampling (Odena et al., 2016). We use margin of 0.1 and 150 points per unit for the encoder's internal discretisation. The length scales of the Gaussian kernels for both the encoder and decoder SetConv layers are set to // 1/150 and held fixed during training. These architecture choices result in receptive field of 0.433 in normalised input space, or roughly 1.500 km in raw input space, spanning around 20% of the region in Figure in either dimension.
For the ConvGNP we use 128 basis functions for the low-rank covariance parameterisation described in Markou et al. 2021.
For the CNn we use an 8-dimensional latent variable and evaluate the MI. objective (Foong al., 2020) using 8 latent samples.
The number of learnable parameters for each model is as follows:
. ConvCNP: 618 k.
ConvGNP: 621 k, ConvLNP: 1.234 (increase due to second UNet architecture after the latent variable).
The difference in parameters from switching to a beta-categorical likelihood from a Gaussian likeli- hood is negligible.
Input data. Each model receives two context sets as input. The first contains observations of the simulated ERA5 daily-average cloud cover. The second contains a set of 6 gridded auxiliary variables. These are elevation, land mask, cos(2T x day of year/ rear/365), sin x day of year/365). I1, and Z2- The elevation and land mask auxiliary fields allow the models to predict spatial non-stationarity. For example, the convolutional filters of the model's 's encoder could learn how cloud cover around the Antarctic coastline behaves differently to the centre of the continent. The cos and sin variables inform the model at what time of year D, corresponds to, helping with learning seasonal variations in the data. The 2 and 22 inputs again help with breaking translation equivariance in the convolutional filters by informing the model where in input space the data corresponds to.
K.3 ANTARCTIC CLOUD COVER MODEL SAMPLES
Figure 15 gives a detailed breakdown of sample extrapolation ability, showing seven samples from the four Antarctic cloud cover models. The AR ConvCNP samples display remarkable structure and variation while still closely interpolating the context points. The samples also provide interesting scenarios in the gaps between the context points on the left hand side.
In contrast, the ConvCNP samples are incoherent, underestimating the probability of joint events.
The ConvLNP samples were generated by sampling from the latent variable and then computing the mean of the marginal distributions. As is visible in Figure 15, the ConvLNP displays low sample variance with respect to the latent variable. However, to the best of our knowledge we used
48

---

Published as a conference paper at ICLR 2023
faithful reproduction of the original ConvLNP model, we leave rigorous treatment of this undesirable behaviour to future work.
For all the non-AR models, the limited receptive field size leads to samples the right hand side becoming independent of the context observations on the left hand side after roughly 750 km of distance from them. This results in the models defaulting to interesting to see that all the non-AR models display similar marginal mean structure, with greater cloud cover towards the centre of the continent and lower cloud cover towards the coastline, followed by increased cloud levels over the Southern Ocean.
mean representation of the data. It
The AR samples time. The Conv model was then conditioned on these AR samples and the predictive mean was computed over the dense 280x280 target space. took 14 minutes generate these AR ConvCNP samples on Tesla V100 GPU.
drawn sparse 70x70 grid spanning the entire input space to save compute
49

---

Published as a conference paper at ICLR 2023
AR CNn
ConvCNF
ConvLNP
ConyGNP
Figure Seven samples from each model in the Antarctic cloud cover sample extrapolation task for 25/06/2018.
50

---

Published as a conference paper at ICLR 2023
L DETAILS OF CLIMATE DOWNSCALING EXPERIMENTS
L.1 DESCRIPTION OF EXPERIMENT
The MLP ConvGNP (Markou et al., 2022) can be used to successfully model dependencies between outputs in statistical downscaling task, improving log-likelihoods over the MLP ConvCNP (Vaughan et al., 2022) and enabling coherent samples. In this experiment, we demonstrate that the AR ConvCNP also be used for this purpose.
The goal of this experiment to estimate the maximum daily temperature at 589 weather stations around Germany. To estimate these temperatures, we follow Vaughan al. (2022) and use 25 coarse- grained ERA-Interim reanalysis variables (Dee et al., 2011) 1) in combination with km-resolution elevation data (Earth Resources Observation and Science Center, U.S. Geological Survey, U.S. Department of the Interior. 1997). We also consider a second setup where we reveal some of the weather station observations. These revealed weather station observations can be used by the models to aid downscaling performance.
The ERA-Interim reanalysis variables considered are tabulated in Table 21. contrast to previous downscaling work, which degrade reanalysis data to between 2 and 2.5", we opt to use the ERA- Reanalysis data at the native 0.75% resolution, consistent with the latest high-resolution climate models with horizontal resolution ranging from 0.5% to 1.0". All variables between 6º to 16 longitude and 47 to 55 latitude, covering Germany. The weather station data subselection from of the European Climate Assessment & Dataset (Tank et al., 2002) and are available at https:: /tt:://wwww /wwww ecad. eu; we use the blended data. Like for the Reanayysssss variables, take the weather stations located within the aforementioned square. The locations of the weather stations around Germany are visualised in Figure 16. The 1 km-resolution elevation data is taken from the United States Geological Survey GTOPO30 elevation data set available at https:: // /oii. org//ll////o 10. / SPOS. This provides global elevation data at 30-arc second resolution, which approximately km.
spatially subset to
Following the VALUE framework (Maraun et al., 2015), we consider all days of the years 1979-2008 and split these years into five folds. We use the first four folds (spanning 1979-2003) for training, holding out the last 1000 days for cross-valie cross-validation; and use the fifth fold (spanning 2003-2008 for evaluation.
L.2 MULTISCALE CONVOLUTIONAL ARCHITECTURE
Deploying the AR ConyCNP in this downscaling experiment comes with significant challenge. Because the elevation data has fine resolution of km, we expect that predictions by the AR ConvCNP will vary roughly also on this length scale. In the autoregressive sampling procedure (Procedure 2. I), samples from the model will be fed back into the model. Therefore, the AR ConvCNP must handle context data which varies the AR ConvCNP must roughly be a 1 km lmmrrssoltionn grid. Unfortunately, making the discretisation this fine is prohibitively expensive and imposes prohibitive memory requirements. It is this limitation that prevents us from extending the Vaughan et al. (2022)'s MIP ConyCNP and Markou et al. (2022)'s MLP ConvGNP include additional weather station observations. We must therefore innovate on the AR ConvCNP design to come up with a convolutional architecture that can handle such a fine discretisation reasonable computational expense.
km spatial scale, which means that the discretisation of
The architecture that we propose is a multiscale architecture operating on multiple spatial length scales. Let us divide the context set D₁ U Dmm J Dhe into a low-resolution component Dr. a medium-resolution component D and a high-resolution component Dzr- Let the low-resolution component Dr consist of features of the context set that vary on long spatial length scale, the medium-resolltton component Drn of features that vary on medium-range spatial length scale, and the high-resolution component Dhe of features that vary on a short spatial length scale. The central assumption of the architecture is that predictions for target points depend on precise short-length-scall details Drn nearby, but that this dependence weakens as we move away from the target point, starting to depend more on broad-stroke depend on detailed orographic information nearby, but more on general orographic shapes farther away.
length-scale components Dr. For example, predictions might
51

---

Published as a conference paper at ICLR 2023
Figure 16: Locations the 589 weather stations around Germany the downscaling experiments.
Variable
Level
Units
Surface
Maximum temperature
degrees Celsius degrees Celsius knots knots
Mean temperature
m
Northward wind
10 m
Eastward wind
10 m
Upper atmosphere
Specific humidity
850, 700, and 500 hPa
g/kg
Mean temperature Northward wind Eastward wind
850, 700, and 500 hPa
degrees Celsius
850, 700, and 500 hPa
knots knots
850, 700, and 500 hPa
Invariant
Angle of sub- scal- orography
surface
Anisotropy of sub-grid-scale orography
surface
Standard deviation of filtered -c-arii---calee orography Standard deviation of orography Geopotential Longitude
surface
surface
surface surface
J/kg degrees degrees
Latitude
surface
Temporal
Fractional position in the year transformed with t + (int(2t))ssinnm
Table 21: ERA-Interin reanalysis predictors.
Figure 17 depicts the multiscale architecture. The architecture is a cascade of three convolutional deep sets, parametrised by three CNNs; please see the caption. The low-resolution CNN handles the context data D₁ with long spatial length scale. Because these features have long spatial length scale, the CNN can get away with a low-resolution discretisation. The output of the low-resolution CNN then feeds into medium-resolutin CNN. The medium-resollttion CNN handles the context data Dmz with medium spatial length scale and has medium-resolutiin discretisation. Finally, the output the medium-resolutton CNN feeds into a high-resolution CNN. This CNN handles the context data Dr with short spatial length scale and has high-resolution discretisation.
The key to the computational efficiency of this architecture is that we construct the high-resolution discretisation only locally to the target points: small square covering 0.25% more than the most extremal target points. If the target points are confined to a small region, then the high-resolution grid will also be small, covering only 0.255 more than that region. Crucially, the high-resolution
52

---

Published as a conference paper at ICLR 2023
positioning of discretisation
0.75 (X), covering all of Germany (V)
d
resolution of discretisation
(
CNN,
density(Din)
0.1 (V), covering a medium-sizer sized square (V)
Zir data Dmr) density( (mr))
2
CNN
0.01 (V), local target points (X)
Zene data Der) density (nr)
Zhr (-) CNN
Figure 17: Multiscale architecture for the AR ConyCNP. cascade of three convolutional deep sets (Gordon al., 2020) representing jow-resolution, medium resolution, and positioning of the internal discretisation for every convolutional deep set. The context set Dm also divided into a low-resolution Dr. medium-resolution Dar. and high-resolution component Dar- The low-resolution context data Dr consists of the 25 coarse-grained ERA-Interim reanalysis variahles. The medium-resollttin Dmr and high-resolution context data Da, both consist of the station observations and the km--te aun-resolution elevation data. The functions data(D) and density (D) produce respectively the data channel and density channel for context data D; see Gordon et al. (2020). The variables Zir( .). Zmn (+ and znr (-) represent intermediate representations continuous functions. and the maps CNN,. CNN and CNNut are translation-cquivariee
nigh-resolution component. Shows the resolution
maps between functions X' Following the construction of the ConvCNP (Gordon al., 2020), these maps are all implemented with convolutional neural networks (CNN) using discretisation. For CNN,, the internal discretisation is the 0.75 rresolution grid corresponding to the 25 coarse- coarse-grained ERA- Interim reanalysis variables. For CNN the internal discretisation is a 0.1 -resolution grid spanning 5 more than the most extremal target inputs; the discretisation does not depend on the context set. For CNN the internal discretisation is 0.01 resolution grid spanning 0.25 more than the most extremal target inputs; the discretisation also does not depend on the context set.
grid will not be constructed over all of Germany, like would we were to more naively apply the ConvCNP with a high-resolution discretisation, incurring prohitive computational cost. Even though the high-resolution grid only constructed locally to the target points, the model can still capture long- long-range dependencies via the resouuttion and low-resolution grids. Namely, the medium- medium--rsoouttinn grid is a square covering 5" more than the most extremal target points, and the low-resolution grid covers all of Germany: target points must be confined to a small region. This perfectly synergises with the autoregressive sampling procedure (Procedure 2.1), because this procedure evaluates the model one target point at time. The training procedure, however, must be adapted. During training, we subsample the target points ensure that the target set is always confined small square, which is described in Appendix L.4.
Figure 17. To utilise this computational gain, the
During the autoregressive sampling procedure, the AR ConvCNP takes in earlier AR samples from the model. In the architectures of the MLP ConvCNP and MLP ConvGNP, these is no natural context data to which these samples can be appended. Therefore, in addition to the ERA-IInterm reanalysis variables and the elevation data, we also let the AR ConyCNP take in observed weather stations as context data. We will append the earlier AR samples to these weather station context data. To have the model make appropriate use of the weather station context set, must randomly divide the weather stations observations over the context and target set, which we describe in Appendix L.4. We let the low-resolution context data D₁. consist of the 25 coarse- -grrine- ERA-Interin reanalysis variables, and let the medium-resee resolution Dmn and high-reso nigh-resolution context data Drr both consist the weather station observations (and earlier AR samples) and the km-resolution elevation data. When
53

---

Published as a conference paper at ICLR 2023
the 1 resouuettttion data is fed the mediumm-rsollttin CNN, the data loses some detail, because CNN is coarser than the data; however, when is
the internal discretisation the fed to the high-resolution CNN, the data retains its detail. The same holds for the weather station observations (and earlier AR samples).
L.3
ARCHITECTURES
MLP ConyCNP and MLP ConvGNP (Vaughan et al., 2022; Markou al., 2022). The MLP ConvCNP and MIP ConvGNP are respectively ConvCNP and ConyGNP where the decoder decg fuse, decg decomposed into a convolutional architecture dedg followed by a pointwise
MLP fuses:
fuse (z( ))
( (. elevation(
(49)
In this architecture, the ERA-Interim variables are incorporated via the convolutional architecture, producing the encoding z(- .)). On the other hand, as (49) shows, the km-resolution elevation data is included via the pointwise MLP fuseg.
Parametrise dec' with seven-layer residual convolutional neural network (He et al., 2016). Every residual layer involves one depthwiss-separable convolutional filter (Chollet, 2017) with kernel size three followed by pointwise MIP. Every layer has 128 channels, and the network also outputs 128 channels. The discretisation for decg is the grid of the ERA-Intent reanalysis variables. Parametrise fuse, with strre-hidden-layer MLP of width 128.
AR CNP. The AR ConvCNP does not use the pointwise MLP fuse, to incorporate the kmn resolution elevation data. Instead, it is normal ConvCNP where the convolutional architecture is implemented by the multi-scale architecture described in Figure 17.
Parametrise CNN, with peepthwise-separdde residual convolutional neural network like in the MLP ConyCNP and MI.P ConyGNP, but use six layers instead of seven. Let CNN,, output 64 channels. The discretisation for CNN,, is the grid of the ERA-Interim reanalysis variables. Parametrise CNN with a U-Net (Ronneberger et al., 2015) using an architecture similar to what we have been using. Before the U-turn, let the U-Net have five convolutional layers with kernel size five, stride one for the first layer and stride two afterwards, 64 output channels for the first three layers and 128 output channels afterwards. After the U-turn, instead of using transposed convolutions, use regular convolutions combined with an upsampling layer using bilinear interpolation. Let CNN output 64 channels. The receptive field of CNN approximately 10 The discretisation for CNN is centred around the target points with margin 5 Parametrise CNN,r with a U-Net like for CNN but with four convolutional layers before the U-turn. The receptive field of CNN,r is approximately 0.5%. The discretisation for CNN is centred around the target points with margin 0.25*.
L.4 TRAINING DETAILS
MLP ConvCNP and MLP ConvGNP. The MLP ConvCNP and MLP ConvGNP are trained with learning rate 2.5 10-5 for 500 epochs. For the MLP ConvGNP, to encourage the covariance fit, we fix the variance of the decoder to 10 for the first ten epochs.
AR CNN. The AR ConvCNP trained with learning rate 1 10 for 500 epochs. During training and cross-vas validation, the target points are subsampled to lie in 3" x 32 square. For training. the number of target points is ensured to be least ten: and for validvaai,,,on, at least one. The size of the cross-validation set is increased ten fold.
Sampling data. For the MLP ConvCNP and MLP ConvGNP, since these architectures cannot take in weather station observations, all weather stations are used context data. For the AR ConvCNP, a data set split into context and target set by randomly selecting n points as context points and letting the remainder be target points. Specifically, the number of context points is sampled from p(n) & e 0.01n. This splitting is done after subsampling the 3" x 3" square.
54

---

Published as a conference paper at ICLR 2023
M ALTERNATE AR PROCEDURE WITH AUXILIARY DATA
We propose an additional procedure which uses autoregressive sampling with auxiliary data to generate more expressive marginal predictives. The input points of the auxiliary data chosen randomly, and then sampled autoregressively before sampling the target points. Finally, we discard the sampled values for the auxiliary data, but retain the samples for the target points. Adding auxiliary points in this way allows the model to roll out autoregressively with more steps, even the target set small (or just a single point). We describe the procedure below:
Procedure M.1 (Autoregressive application of neural process with auxiliary data). For neural process Te. context set Dis) = xx), a target input z0",, a distribution over X, a number of auxiliary data points R E N, and a number of trajectories M E N, let AuxARyo (Ta, D()), r. R, M) be the distribution defined as follows. We first autoregressively sample the auxiliary data trajectories at random locations sampled from r:
for j ..... M and 8= R.
Ft (a r,
(50)
for j = 1...,,M, M,
y faux.j) A (To. D
(51)
Next, conditioned on the auxiliary data, we sample the target point of interest make predictions. We then marginalise out the auxiliary data by averaging over the M trajectories:
y 1
Ms プラ
P200 (x(+) xlaux.)), y/ + fluux.)).
(52)
This procedure introduces three hyperparameters: the distribution from which to draw inputs r, the length of trajectories R. and the number of trajectories to sample M.
In the following experiments, we set the distribution r to be uniform over the training domain with no dependence on the context set or target point of interest: r = Uniform(Ab [b, hl). where b and h are the lower and upper bounds of the training domain, respectively. One could experiment with other choices for the distribution r. The trajectory length R is chosen between 0 and 8, and the number of trajectories M is chosen between 1 and 128.
M.1 GENERATED DATA
We create three data generating processes for experiments: mixture of functions, random sawtooth functions, and random audio-like functions. The first two experiments have multi-modal true marginals, whereas the last has heavy-tailed marginals.
Function mixture. The function mixture data are generated by choosing one of the following three functions, the first with probability of 1 the second with probability of z. and the third with probability of 1:
2 + e, У x + C, x + e,
E (O, 0.25),
(53)
[10 0.0625),
(54)
E < (10 0.25).
(55)
Sawtooth. The sawtooth data are generated from the following function:
y(æ) = (w<dr - x)] mod
(56)
We sample the frequency w [n. 5)), the direction d as either -1 or 1 with equal probability, and the shift as & (n.. 1]).
Synthetic Audio. Synthetic audio data are generated by convolving a Dirac comb" with a truncated decaying sum of sinusoids:
s(t) = se- (sin(wit) + sin(wez)) for 0 < T, otherwise,
(57)
00
f(x) = (omby(x) s(x),
(58)
y = f(x) + where e N/O00.0011)
(59)
"A trajectory length of 0 is equivalent to the standard test-time procedure.
"The Dirac comb is defined Combr(t) x 5(t kT) for given period P.
55

---

Published as a conference paper at ICLR 2023
Experiment
ConvCNP
ConvCNP (AuxAR)
Function Mixture
-0.63 = 0.20
0.46 0.15
Sawtooth
1.46 = 0.30
1.64 + 0.28
Synthctic Audio
0.12 = 0.14
0.55 + 0.10
Table 22: Using autoregressive sampling for marginal approximation improves held-out log likelihoods all experiments. Values normalized by the number of target points. Values which significantly best. (p < 0.05) are shown hold.
where
W1,42 (500, 70)), T~ (0.55 1.25)),
[n.1 0.3]).
(60)
We truncate the waves up the period length, because otherwise the convolution with the Dirac comb would lead to increasing amplitude, resulting in a non-stationary process.
M.2 TRAINING
For the function mixture experiment, no training is required because we use the analytically derived model. See Section
ideal CNP Tee
For the sawtooth and synthetic audio experiments, we train ConvCNP models. We train each model for 100 epochs using 1024 batches per epoch with batch size of 16. We discretise the encoder by evaluating 64 points per unit. We use a margin of 0.1, and a stride length of 2 for each of the layers the U-Net. Each layer has 64 channels. The receptive field size from this combination of parameters is 6.953.
During training, we sample number context points between uniformly random from (O.....75) and we sample exactly 100 target points. The context points and target points are sampled uniformly from [_, We use the Adam optimizer with a learning rate of 3 x 10-4
M.3 RESULTS
Model
ConvCNP -0.47=0.07
-0.4520.18
-0.2420.12
-0.1220.12
1.0740.18 0.74+0.10 1.27
0.36
ConvCNP (AuxAR)
0.02 +0.11
0.19
0.24
0.31+0.11
Table 23: Log likelihood values for varying context sizes using ConyCNP and ConvCNP (AuxAR) using the function mixture data generator. Column headers indicate the context set size. Log-likelihoods are shown in bold when they significantly best 0.05). Column headers context sizes. Errors indicate central 95% confidence interval.
Model
2
ConvCNP
-0.18
-0.1410.02
-0.0240.06
0.42:0.19
1.8940.25
3.06.1.13 3.54.0..8
ConvCNP (AuxAR)
-0.2110.02
10.09
0.08
0.9510.16
2.37 3.03 3.51
Table 24: Log likelihood values for varying context sizes using ConyCNP and ConvCNP (AuxAR) using the sawtooth data generator. Column headers indicate the context set size.. Log- lileliilssoods shown in bold when they are significantly best (p 0.05). Column headers are context sizes. Errors indicate central 95% confidence interval.
In Table 22, we see that using this procedure improves the held-out likeliiisssoods for all of the experiments. We can better understand the utility of this method by observing the performance for
56

---

Published as a conference paper at ICLR 2023
Model
ConvCNP
-0.91 -1.40
-0.36
ConvCNP (AuxAR)
-1.05=0.11
20.6320.23
-0.29
Table 25: Log likelihood values for varying context sizes using ConyCNP and ConvCNP (AuxAR) using the function mixture data generator. Column headers indicate the context set size. Log likelihoods are shown in bold when they significantly best 0.05). Column headers context sizes. Errors indicate central 95% confidence interval.
different context sizes in Tables and 23, For example, the sawtooth data results in Table 24 show that, for context set sizes 16, the AR method provides no benefits, The increased flexibility of this AR method is not needed in this case For context set sizes of 4 and 8, on the other hand, we see significant improvements using the AR method. Similarly, for the synthetic audio data results Table 23, we see improvements using the ConvCNP (AuxAR) for all context set sizes except 16.
Gaussian predictive models the marginal sufficiently well.
The autoregressive sampling with auxiliary data method shown here shows promise for improving modeling of processes with multi-modal and heavy-tailed marginal distributions all with no changes to the training procedure. The scenarios where this method are most useful are highly contingent upon the context size, because of its impact how well the marginals modeled by Gaussians.
57


Reviews:
[{"review_id": "zJX28T4RNJN", "reviewer": "Reviewer_WrGZ", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates an autoregressive (AR) evaluation scheme for Conditional Neural Processes (CNP) and its variants. The work is motivated by the fact that standard CNP does not model correlations between prediction targets. Incorporating these correlations is tackled in prior work by extending the *model*, which typically leads to complex training procedures. In contrast, the proposed method changes the *evaluation procedure* by defining an autoregressive, non-factorized joint distribution over targets, while leaving the model/training procedure of CNPs untouched. In a range of experiments, the paper shows that AR-CNPs show strong performance.", "summary_of_the_review": "The paper studies an interesting way of constructing expressive predictive distributions for CNP models at test time which will be of interest for the community. Both the theoretical and empirical parts are sound and well fleshed out. I have some open questions that I ask the authors to elaborate on, so I might further increase my score after the discussion period. Nevertheless, I think the paper is ready for publication at ICLR, and, thus, I recommend acceptance. \n\n\n_________________________________________\n\n\nUpdate after rebuttal: I increase my score, cf. my comment below.", "strength_and_weaknesses": "Strengths:\n\n- The paper studies an interesting and relevant problem.\n- The paper is very well written and easy to follow.\n- The experimental section is very well fleshed out.\n\nWeaknesses: there are only minor open points that I encourage the authors to elaborate on, cf. below.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "r6-cx0wHEN", "reviewer": "Reviewer_K56J", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents Autoregressive Conditional Neural Processes, a way to use conditional neural processes in an autoregressive way, by feeding newly sampled points back into the CNP and re-evaluating. The paper vouches that using CNPs in this autoregressive way gives better predictive performance, since there are no independence assumptions. Furthermore, there are no changes that need to be done to the training or architecture, since this process only affects the inference routine.\n\nThe paper notes that this process does give up the property of consistency, since the predictions can depend on the different permutations chosen during inference. Also, sampling can be done in parallel using blocks of size N, to trade off conditional dependence and speed.\n", "summary_of_the_review": "In summary, the paper proposes to augment conditional neural processes with an autoregressive inference procedure, by feeding samples back into the model. This method improves predictive performance without requiring retraining of the model.\n\nBased on my personal opinion, I think this (or similar) techniques have already been used in many recent works, so I'm not sure that it is novel enough to be of interest. However, I'm not against acceptance if other reviewers find the idea novel, since otherwise the paper is well written with reasonable experimentation.\n\n\n=============\n\nI have read the other reviews and the author's response.", "strength_and_weaknesses": "*strengths*\n- related work section is comprehensive (maybe also worth including [this](https://arxiv.org/abs/2206.03992))\n- paper is overall well written, with thoughtful discussion\n- a variety of experimental results\n\n*weaknesses*\n- the idea of unrolling autoregressively seems to be already well known at this point, in many recent works, and in some of the works already cited (e.g. esp [[1]](https://arxiv.org/abs/2207.04179) cf C.1, and also [[2]](https://arxiv.org/abs/2102.04426) [[3]](https://arxiv.org/abs/2110.02037) [[4]](https://arxiv.org/abs/2205.13554)). Therefore, I'm not sure that the ideas of this paper are novel enough to be a useful contribution to the community.\n  - e.g. [[3]](https://arxiv.org/abs/2110.02037) Section 3.1 has a more in-depth discussion about parallelizing blocks, and presents a dynamic programming method from [[5]](https://arxiv.org/abs/2106.03802) for choosing how to parallelize. It's not in the stochastic process framework, but it does seem very similar.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "JmagQb-8IVo", "reviewer": "Reviewer_AdCH", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper shows that a standard CNP (trained with the diagonal covariance assumption) can be used, at test time, to sample target values capturing their non-diagonal covariance structure. This is done by sampling one target point at a time from the CNP and feeding that predicted target point back into the context set of the CNP. This is termed as running ‘CNP in auto-regressive (AR) mode’.", "summary_of_the_review": "Several results (e.g., equation 7) rely on mathematical proofs which I think I (the reviewer) do not have the right expertise to review properly. Besides that, I think the paper is simple and clever. But due to my lack of expertise in judging the math and the possibility that there is some paper that already shows some part of what the paper shows (e.g. because it is so simple), I will exercise caution and rate it as 6 with confidence 2.", "strength_and_weaknesses": "### Pros\n\n1. Paper adopts the simplest class, CNP, to demonstrate their key idea — thus their main result seems to be free from the nuances of a complex architecture.\n2. Paper shows that CNPs in AR mode are a more flexible class than GNP.\n3. The paper clearly mentions the limitations such as the lack of consistency under marginalization and permutation. Also, care may be taken when feeding more points into the context set than shown during training. It is good for practitioners to know about these limitations and exercise caution when deploying it. \n4. Paper also proposes how one can trade off speed and expressiveness in CNPs in AR mode by sampling target points block-wise.\n5. Empirically, they show that CNP in AR mode outperforms the usual CNP that comes with diagonal covariance over target points. It also tends to do better/comparable to other baselines e.g. GNP or LNP. The fact it does so despite carrying the simplicity of CNPs seems like a good takeaway.\n\n### Weaknesses/Questions\n\n1. I do not fully understand the procedure for obtaining the smooth samples on the bottom part of page 4. \n2. Paper does not analyze how the performance is affected if we auto-regressively sample more target points than shown during training. How quickly does it get worse?\n3. Another important question not analyzed is whether TNP (a recent state-of-the-art) trained in diagonal-covariance-mode (i.e. TNP-D) would benefit similarly at test-time to what is shown in this paper for CNPs.\n4. When trying to sample $N$ target points in the AR model, we sample the points $1, 2, \\ldots, N$ sequentially. It would be good to show whether prediction error worsens going from point $1$ to point $N$ due to the accumulation of prediction errors in the context set.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster