Practical Conditional Neural Process Via Tractable Dependent Predictions
Authors: Stratis Markou
James Requeima
Wessel Bruinsma
Anna Vaughan
Richard E Turner
Venue: ICLR 2022 Poster (2022)
Keywords: conditional neural processes
neural processes
meta-learning
convolutional conditional neural processes
Gaussian neural processes

Abstract: Conditional Neural Processes (CNPs; Garnelo et al., 2018a) are meta-learning models which leverage the flexibility of deep learning to produce well-calibrated predictions and naturally handle off-the-grid and missing data. CNPs scale to large datasets and train with ease. Due to these features, CNPs appear well-suited to tasks from environmental sciences or healthcare. Unfortunately, CNPs do not produce correlated predictions, making them fundamentally inappropriate for many estimation and decision making tasks. Predicting heat waves or floods, for example, requires modelling dependencies in temperature or precipitation over time and space. Existing approaches which model output dependencies, such as Neural Processes (NPs; Garnelo et al., 2018b) or the FullConvGNP (Bruinsma et al., 2021), are either complicated to train or prohibitively expensive. What is needed is an approach which provides dependent predictions, but is simple to train and computationally tractable. In this work, we present a new class of Neural Process models that make correlated predictions and support exact maximum likelihood training that is simple and scalable. We extend the proposed models by using invertible output transformations, to capture non-Gaussian output distributions. Our models can be used in downstream estimation tasks which require dependent function samples. By accounting for output dependencies, our models show improved predictive performance on a range of experiments with synthetic and real data.

Published as a conference paper at ICLR 2022
PRACTICAL CONDITIONAL NEURAL PROCESSES VIA TRACTABLE DEPENDENT PREDICTIONS
Stratis Markou
James Requeima"
Wessel P. Bruinsma
University of Cambridge em6260 ac..ucuu uk
University of Cambridge Invenia Labs j jrr41 1 cam. ac. uk
University of Cambridge
Invenia Labs
wpb238 cam. ac. uk
Anna Vaughan
Richard E. Turner University of Cambridge ret cam ac
University of Cambridge av5550cam, ac.
ABSTRACT
Conditional Neural Processes (CNPs; Garnelo et al. 2018a) are meta-learning models which leverage the flexibility of deep learning to produce well-calibrated predictions and naturally handle off-the-grid and missing data. CNPs scale to large datasets and train with ease. Due these features, CNPs appear well-suited tasks from environmental sciences or healthcare. Unfortunately. CNPs do not pro- duce correlated predictions, making them fundamentally inappropriate for many estimation and decision making tasks. Predicting heat waves floods, for ex- ample, requires modelling dependencies in temperature or precipitation over time and space. Existing approaches which model output dependencies, such as Neu- ral Processes (NPs; Garnelo et al.) (018b) or the FullConvGNP (Bruinsma et al.) 2021), either complicated to train prohibitively expensive. What is needed is an approach which provides dependent predictions, but is simple to train and computationally tractable. In this work, we present
class of Neural Process models that make correlated predictions and support exact maximum likelihood training that is simple and scalable. We extend the proposed models by using in- vertible output transformations, to capture non- non- Gaussian output distributions. Our models can be used downstream estimation tasks which require dependent func- tion samples. By accounting for output dependencies, our models show improved predictive performance on range of experiments with synthetic and real data.
INTRODUCTION
Conditional Neural Processes Garnelo et al. 2018a are a scalable and flexible family of meta- learning models capable of producing call-calibrated uncertainty estimates. CNPs naturally han- dle off-the-grid and missing data and are trained using a simple-to-implemenn
maximum--ikellioon procedure. At test time, CNPs require significantly less computation and memory than other meta- learning approaches, such as gradient-based fine tuning (Finn et al.] [201]] Triantafillou et al.) 2019), making them ideal for resource and nower-limited applications, such mobile devices. Further, CNPs can be combined with attention (Kim et al.) 2019), or equivariant networks which account for symmetries the task hand (Gordon al. 2020 Kawano al., 2021 Holderrieth et al., 2021). achieving impressive performance on a variety of problems. Despite these favourable qualities, CNPs are severely limited by the fact that they do not model dependencies in their output (fig. I].
T
Figure 1: Unlike the ConvCNP (left) which makes independent predictions, the ConvGNP (right), introduced this work, makes dependent predictions and can be used to draw function samples (blue) which coherent. These often necessary for downstream estimation tasks.
Authors contributed equally.

---

Published as a conference paper at ICLR 2022
Limitations of CNPs: More specifically, given two target input locations I. and 2m', CNPs model their respective outputs ym and Ym independently. In this paper, we refer to such predictions as mean-field. The inability model dependencies hurts the predictive performance of CNPs and ren- ders it impossible to produce coherent function samples. Since many downstream tasks require de- pendent function samples, this excludes mean-field CNPs form range of applications. In heatwave or flood prediction for example, we need to evaluate the probability of the event that the temperature or precipitation remains above some threshold, throughout a region space and time. As illustrated by fig. [ meean-field predictions model every location independently. and may assign unreasonably low probabilities to such events. we were able draw coherent samples from the predictive, the probabilities of such events and similar useful quantities could be more reasonably estimated.
Limitations of existing models with dependencies: To address the above, follow-up work has introduced Neural Processes (NPs; Garnelo et al., 2018b Kim et al.] 2019 Foong et al., 2020), which use latent variables to model output dependencies. However, the likelihood for these models is not analytically tractable, so approximate inference is required for training (Le al. 2018 Foong ct al, 2020). Alternativell,Bruinn
et al. 202T) recently introduced a variant of the CNP called the Gaussian Neural Process, which we will refer to the FullConvGNP, which directly parametrises the covariance of a Gaussian predictive over the output variables. In this way the FullConvGNP models statistical dependencies in the output, and can be trained by an exact objective, without requiring approximations However, for DDddiensiioaa data, the architecture of the FullConvGND involves 2DD-dimensional convolutions, which can be very costly, and, for D > 1, poorly supported by most Deep Learning libraries.
-ikkliholkellooddd
Contributions: In this work (i) we introduce Gaussian Neural Processes (GNPs), a class of model which directly parametrises the covariance of a Gaussian predictive process, thereby circumventing the costly convolutions of the FuI,ConvGNP, and is applicable to digee---imensional input data. GNPs have analytic likelihoods making them substantially easier to train than their latent variable counterparts; (ii) we show that GNPs can be easily applied to multi-output regression, as well as composed with invertible marginal transformations to model non-Gaussial data; (iii) we demon- strate that modelling correlations improves performance on experiments with both Gaussian and non-Gaussian synthetic data, including a downstream estimation task that mean-field models cannot solve; (iv) we demonstrate that GNPs outperform their mean-field and latent variable counterparts on real-world electroenceehalogran (EEG) data and climate data; (v) in climate modelling, GNPs outperform a standard ensemble of widely used methods in statistical downscaling, while providing spatially coherent temperature samples which are necessary for climate impact studies.
EQ
MAT
NM
WP
CNuyGNP
Ground truth
0
Figure 2: The ConvGNP model, introduced in this work, can recover intricate predictive covariances. Columns show the posterior covariances produced by a ConvGNP, after training with synthetic data drawn from a Gaussian Process with a different covariance (exponentiated quadratic, Matern, noisy mixture or weakly periodic), and conditioned on a randomly sampled dataset.
2 CONDITIONAL & GAUSSIAN NEURAL PROCESSES
Background: We present CNPs from the viewpoint of prediction maps (Foong et al. 2020 A prediction map x is function which maps (1) a context set (x.. where x, (T Ne,N) are the inputs and y. = (Je.I....:Bb No,NN) ) the outputs and (2) set of larget inputs X (x ..... Tt. to distribution over the corresponding target outputs yt = (yr,1 Mt.
(yri Yc: xx) (y:lr),
2

---

Published as a conference paper at ICLR 2022
where r r(Xe: Ve: X²) is vector which parameterises the distribution yt. For fixed con- text Yc). using Kolmogorov's 's extension theorem (Oksendal 2013), the collection of finite- dimensional distributions (yr: Ye. Ye, for all 1........ Xt,M T M E N, defines stochas- tie process if these are consistent under (i) permutations of any entries of Yt) and (ii) marginali- sations of any entries of yz. Prediction maps include, but are not limited to, Bayesian posteriors. One
familiar example of such a map is the Bayesian Gaussian process (GP; Rasmussen 2003) posterior
(yr: Xe, Yc. xx) N (y:: m,
where m (Xe: Ye, Xx) and K k Xe: X2) are given by the usual GP posterior expressions. Another prediction map is the CNP Garnelo et al. (018a):
π
= II p(
(3)
where each p(yt.m (mm) is an independent Gaussian and I'm. r(Xe: Ye: Tt. Ft,m) is parameterised by DeepSet (Zaheer et al., (017). CNPs are permutation and marginalisation consistent and thus correspond to valid stochastic processes. However, CNPs do not respect the product rule in gen- eral (Foong et al. 2020). Nevertheless, CNPs and their variants (Gordon et al. (0200 have been demonstrated to give competitive performance and robust predictions in a variety of tasks and are a promising class of meta-learning models.
Gaussian Neural Processes: A central problem with CNP predictive distributions is that they are mean-field: eq. does not model correlations between Vt,m and Vt.m² for m * m'. However, many tasks require modelling dependencies the output variable. To remedy this, we consider parameterising a correlated multivariate Gaussian
π (yr; X,) = M)
(4)
where, instead of the expressions for the Bayesian GP posterior, we use neural networks to param- eterise the mean m = mm(x. x,) and covariance K = K (xxe X,). We refer to this class of models as Gaussian Neural Processes (GNPs). The first such model, the FullConvGNP, was intro- duced by [Bruinsma al. (2021) with promising results. Unfortunately, the FullConvGNP relies on 2 D--mmmensional convolutions for parameterising K, applying the sequence of computations
PSDD(CNNaD(h)) ② K, Et= th(r,, 51) (5t,
(5) where ① maps (x.,, Yo) 2 2D-dimensina grid h locations X (ā (T,,..., ....,,,,),.. E R2D, using a SetConv layer Gordon et al. 2020)) 2 maps h to r through a CNN with 2D-dimensions convo- lutions, followed by PSD map which ensures is positive-definite and 3 aggregates r using an RBF vh. The CNN at @ requires expensive -dmdinnnsia convolutions, which are challenging to To overcome this difficulty, we propose parameterising
(x)
[x h)
scale to higher dimensions (see appendix B).
m and K by
m; f( r). Kij k(g( r), ((a,)
where r r(xe. and g are neural networks with outputs in E and Ds, and k is an appropri- ately chosen positive-definit function. Note that, since A models posterior covariance, it cannot be stationary. The special case where K,, = I j is diagonal corresponds to mean-field CNP as presented in (Garnelo et al. (0] Equation T defines class of GNPs which, unlike the Full- ConvGNP, do not require costly convolutions. GNPs can be readily trained via the log-likelihood
0+ arg max, log Ye, Xt),
where f collects all the parameters of the neural networks f. g, and r. In this work, we consider two methods parameterise K, which we discuss next.
Linear covariance: The first method we consider is the linear covariance
K₁ = r( g(((...))
which can be seen as a distribution on their weights. This model meta-learns D, context-dependent basis functions, which approximate the true distribution of the target, given the context. By Mercer's theorem (Rasmussen, 2003). up to regularity conditions, every positive-definite function i can be decomposed as
parameters model with D, basis functions and unit Gaussian
Hzz,2/) Edzo (ba(2) (b(=))a((2))
3

---

Published as a conference paper at ICLR 2022
where (od) is a set of orthogonal basis functions. We therefore expecteq. ( to be able to recover arbitrary (sufficiently regular) GP predictives as Dg grows large. Further, the linear covariance has the attractive feature that sampling from it scales linearly with the number of target locations. A drawback is that the finite number of basis functions may limit its expressivity.
Kvv covariance: An alternative covariance which sidesteps this issue, is the kvv covariance
K k(g( (). g2,, ())(((
(10)
where k is the Exponentiated Quadratic (EQ) covariance with unit lengthscale and v is a scalar- output neural network. The e modulate the magnitude of the covariance, which would otherwise not be able to shrink near the context. Unlike linear, kvv is not limited by finite number of basis functions, but the cost of drawing samples from scales cubically in the number of target points.
Multi-output regression: Extending this approach to the multi-outpll setting where Vt. E RD, with D, > 1, can be achieved by learning functions m1...., m. and g1..... -,,, for each dimen- sion of the output variable. We can represent covariances across different target points and different target vector entries, by passing those features through either the linear or the kvv covariance
Kijab (a(t,,,,, r) go( Kijab = ((ga( r). g0
(11)
(12)
where Kijat denotes the covariance between entry a of ye.s and entry of yt..
This discussion leaves room for choosing f. 9. and r. producing different models belonging to the GNP family, of which the FullConvGNP is also a member. For example, may choose these to be DeepSets, attentive Deepsets or CNNs, giving rise to Gaussian Neu- ral Processes (GNPs), Attentive GNPs (AGNPs) or Convolutional GNPs (ConvGNPs) respectively. Particularly, in the ConvGNP, the feature function g takes the form
Neural architectures:
(13) where, crucially, are values on a D-dimensional grid at (fI...., #1), bo E R.D, and 2 uses D-dimensional rather than 2 2D-dimensional CNN. This renders the ConvGNP much cheaper than the FullConvGNP in both compute and memory, while retaining translation equivariance (see appendix for proof), making the former scalable alternative to the latter.
(Xer (x. h)
(h) g(zr,i. Ei=n (211,, nI.
3 NON-GAUSSIAN PREDICTION MAPS
For many tasks joint-Gaussian models are sufficient, However, many tasks require non-Gaussian marginal distributions instead, for example because of non-negative or heavy-tailed variables.
Gaussian Copula Neural Processes: To address this issue, draw inspiration from copula mod- els (Elidan 2013). Copulae use dependent base distribution to model correlations, and adjust its
marginals using invertible transformations better approximate the data at hand. WWllson& & Ghahra- mani (2010) and Jaimungal & Ng 2009) have extended copulae to the stochastic process setting, by using a GP as base measure, and transforming its marginals appropriately, Adapting the approach of Jaimungal & Ng we consider the following transformation to the output of a GNP
T (u,, vb), where = (v:)v.) and p(v:) = To (Vr; Xc, Ye, Xt).
(14)
where &G is the CDF of the standard Gaussian, T5d is the inverse CDF of a chosen distribution with parameters ab = (e, xt), and TG is a Gaussian prediction map. We refer to models of this form as Gaussian Copula Neural Processes (GCNPs). The log- likelihood of this model can be computed exactly using the change of variables formula (Kobyzev et al.) 2020). Since the transformation in eq. (1) is dimension-wise,, the resulting Jacobian is diagonal, and the log-likelihood takes the form
log π (yri Ye, xx) log Ye. x.) MM log e @y(y,,.))
(15)
where ((-)= = 0 (# (.. +b) v. = O(y:). Generally, can be any arbitrary invertible map such Normalising Flow (NF). However, requiring marginalisation consistency places limitations the form of O While point-wise NIs are consistent under marginalisations, it is unclear if non- marginal NFs can be used in a consistent model (appendix A.2). Last, while fully learnable marginal NFs (Durkan et al. 2019) can be used, the θ presented here was sufficient for our experiments.
4

---

Published as a conference paper at ICLR 2022
4 EXPERIMENTS
We apply the proposed models to synthetic and real data. Our experiments with synthetic data com- prise four Gaussian tasks and a non-Gaussian task. our experiments with real data, we evaluate models on electroencephalogran data as well as climate data. We train our models and the Full- ConvGNP, whenever applicable, using the maximum maximum-likelinee likelihoodd objective in eq. 7 We also train the ANP and ConvNP models discussed in Foong et al. (2020). These are latent variable models which place a distribution over r and rely on q for modelling output dependencies. Following Foong al. train the ANP and ConvNP via biased Monte Carlo estimate of the objective
0 arg max, log (Erg(rr) (yri Ye. X)]]
(16)
4.1 GAUSSIAN SYNTHETIC EXPERIMENTS
We apply the proposed models synthetic datasets generated from GPs with four different covari- ance functions. In these experiments we have access to the ground truth predictive posterior, which we can use to assess the performance of our models. For each task, we generate multiple datasets using fixed covariance function, and sub-sample these datasets into context and target sets, to which we fit the models (fig. 3). We consider tasks with one and two-dimensional inputs, the latter being a problem where the FullConvGNP cannot be feasibly applied to. Figures 4aauress/annSa 5 comparr the predictive log-likelihood of the models on these tasks, from which we observe the following trends.
GNP
AGNP
CanyGNP
PullCoevGNP
Oracle GP
100
MAT
WP
NM
Figure 3: Samples drawn from the models' predictive posteriors (green) compared to the ground truth marginals (blue), using the kvv covariance.
Matern
Nojey mixture
Woukly periodie
1.5
Log Lik
0.3
1.0
1.
0.5
0.5
0.0
0.0
-0.5
-0.5
-10-
-10
-13
-10
CP Oradle
(mesnfied))
AGNP (musnnfskk)
(mnnendield)
ANP
GP Urade ((la))
CNP (lisnr)
AGNP (Lmmu)
ConvlNNP (lumee)
CmucP
GNP (lev)
ACNP (29e)
ConvCNNP (lev)
Figure 4: Predictive log- likelihoods across datasets for the ID Gaussian tasks. The oracle GP per- formance shown in dashed black. The dashed red line marks the performance the diagonal GP oracle, where the off-diagonal covariance terms O. Error bars too small to be seen in the plots.
FQ
Matern
Naisy Mixture
Weakly Periadic
-1.0
Log Lik
0.8
000
-.2
0.4
0.0
0.4
-00
-0.27
-08
GP Oracle
Oracle (ding.)
CoavUNN (misullek))
CouvGNP (Imear)
ConnGNP (lwv)
CouvNN
Figure 5: Predictive log- likeliood performance of the models, across datasets for the 2D Gaussian experiments, where the MallConvGNP is not applicable. Error bars too small to be seen in the plots.
5

---

Published as a conference paper at ICLR 2022
Dependencies improve performance: We expected that modelling dependencies would allow the models to achieve better log- loglliikiihoos.. Indeed, for a fixed architecture, the correlated GNPs (n. M, typically outperform their mean-fiell counterparts
A) This suggests that our models can learn meaningful dependencies in practice, in some cases recovering oracle performance.
The correlated ConvGNPs (a, A often competitive with the FullConvGNP (#)) The kvv ConvGNP (+) is the only model, from those examined here, which competes with the FullConvGNP in all tasks. Unlike the latter, however, the former is scalable to higher input dimensions, and remains the best performing model in the wwo-dimensional tasks (fig. 5). For further details on the runtime and memory costs, see appendix
Correlated ConyGNPs compete with the FullConvGNP:
Correlated GNPs outperform latent variable models: Correlated GNPs typically outperform the latent-variable ANP (+) and ConvNP (x) models, which could be explained by the fact that the GNPs have a Gaussian predictive while ANP and ConvNP do not, and all tasks are Gaussian. Despite experimenting with different architectures, and even allowing for many more parameters in the ANP and ConvNP compared to the AGNP make the latent variable models competitive with correlated GNPs. We typically found the GNP family significantly easier to train than the latent variable models.
and ConvGNP (d, A) we found it difficult to
We generally observed that the kvv models (m. performed as well, and occasionally better than, their linear counterparts (I, A). To test whether the linear models were limited by the number of basis functions Dg. we experimented with various settings D, {16, 128, 512, 2048). We did not observe a performance improvement for large Dy, suggesting that the models are not limited by this factor. This is surprising because, as Dg - oo and assuming flexible enough f. g. and r, the line models should, by Mercer's 's theorem, be able to recover any (sufficiently regular) GP posterior. We leave open the possibility that the 1 linear models might be more difficult to optimise and thus struggle compete with kvv.
Kvv outperformed linear:
Predictive samples: Figure 3 shows samples drawn from the predictives, from which we qualita- tively observe that, like the FuIIConvGNP, the ConvGNP produces good quality function samples, which capture the behaviour of the underlying process. The ConvGNP is the only conditional model (other than the (NCConGG))) which produces high-quality posterior samples. The predictive log- likelihood (fig. can be used as an objective measure for sample quality.
4.2 PREDATOR- PREY EXPERIMENTS
To assess performance on a non- Gaussian synthetic task, we generate data from a Lotka-Volterna predator-prey model (Arnold, 1992). This model describes the evolution of the populations of predator and a prey species, which are related via a stochastic non-linear difference equation. These time series non-Gaussian, since neither of the populations can fall below zero. To encode this prior knowledge, we use marginal transformations, enforcing non-neen negativvtt in the output variable.
We generate synthetic data following the method of Gordon et al. (2020) (see appendix D We train and evaluate the ConyGNP, ConvNP and FullConvGNP models. We also train ConvGCNT with an exponential CDF transformation a (u) 1 w/vh 1b (xxx. Ye: xt). The marginals of these ConvGCNPs explicitly encode the prior knowledge that the populations are non non-negative quantities. Figure G shows the model fits on a fixed predator time series. also shows the predictive log-like oigliikelhoood on test data and the liglliill on a downstream estimation task, explained below.
Dependencies improve performance: Similarly to the Gaussian tasks, we observe that modelling improves the predictive log- over mean field models performs better than the linear covariance
output dependencies (A.
Further, the kvv covariance
We observe that exponential marginal transformations improve per- formance in the correlated models, as the ConyGCNP models (+.+) typically exhibit better per- formance than their ConvGNP counterparts (a, A). Further, the ConvGCNPs produce non-negative posterior samples, which are arguably more plausible than the samples produced the other models.
Marginal transformations:
Comparison with the ConvNP and FullConvGNP: The ConyGNP with kvv covariance (4) is competitive with the FullConvGNP (s), while adding marginal transformations can further improve the model's performance (+). Both models outperform the ConvNP (x) by considerable margin.
Correlated GNPs and downstream estimation: We assessed the performance of the models on downstream estimation task, illustrated in fig. Given context set, we use the trained models

---

Published as a conference paper at ICLR 2022
ConvGNP (meanfield)
ConvGCNP (meanfield)
0.0
Threshold estimator log lik
Predictive log lik
ConvGNP (linear)
ConvGCNP (lincar)
-03
252
-044
ConvGNP (kvv)
ConyGCNP (kvv)
-0.5
-0.6-
2x
-0.7
1.2-
ConvNP
FullConvGNP
CamCNP (suastiol))
ClustiNP (kev)
252
CanGCNNP (mssuniid))
OawGCNPP Jooy)
30
ComGNN (near)
ConoNP FalCkmnGNP
ComCCNN (Inear)
The predator modelling task. Model fits (left) where black and red crosses show the context and target sets of dataset, the blue regions show marginals, and the green lines are samples from the predictive. The dashed line marks y O. Error bars for per-datapoint predictive logllikels and threshold estimation task log- log-likelshoods too small to be seen in the plots.
Figure 6:
ConvCGNP (meanfield)
ConvCGNP (kvv)
2-
10
20
30
40
20
30
T
Figure 7: Illustration of the failure mode of mean-field models in the threshold estimation task. The context and target are shown in black and red crosses, and the black line shows the threshold for this context set. In each plot, three samples from the predictive are shown in orange, green and blue.
to estimate the probability that the population will exceed the maximum population observed in the context, by factor of least 1.1. Mean-field models make independent predictions, and thus the proportion of function samples which exceed the threshold is unreasonably large. By contrast, correlated model avoids this failure mode as their function samples are coherent. Figure shows that correlated models (A, this threshold estimation task, with the latter being marginally worse than a random prediction.
a) are substantially more accurate than mean-field models (a.
4.3 ELECTROENCEEPHAOOGR.. EXPERIMENTS
We applied the proposed and competing models on a real electroenceehallgram (EEG) dataset. This dataset comprises of 7632 multivariate time series, collected by placing electrodes different sub- jects' scalps (Zhang et al. 1995). Each time series exhibits correlations across channels, as the levels of activity different regions of the subjects' brain correlated. It is plausible that there also ex- ist patterns which are shared between different time series from a single subject, as well as across subjects, making this ideal task for multi-output meta-learning model such the ConyGNP. We showcase the ability of the ConvGNP to perform correlated multi-output predictions and handle missing data. We train the models on time series with 7 channels, where we occlude randomly cho- sen windows of of the channels, using the occluded values as targets. Table [ shows the models' performance on held-out test data, and fig. shows an example of a ConvGNI To demonstrate the benefits of meta-learning, the table includes multi-output GP baseline (MOGP, Bruinsma et al.) 2020), which is trained from scratch for every task, without a meta-learning component (see ap- pendix Observe that this baseline is significantly outperformed by all meta-models.
Correlated models improve predictive log-likelihood: We observe that the correlated ConyGNPs show considerable improvement in terms of lig-liiel,oodd compared to the mean-field ConvGNP as well as the ConvNP. This improvement in log- likeeioldoood suggests that the correlated ConvGNPs leam to represent meaningful correlations in the target outputs.
ConyGNPs can model multivariate and partially observed data: Figure g shows that the kvv ConvGNP is able to infer unobserved values from the EEG data of held-out test example. The model produces both calibrated marginals as well as plausible function samples, and represents not only temporal, but also cross-channel correlations.

---

Published as a conference paper at ICLR 2022
CONvGNP 1inear
CONvNP
MOGP
MEAN-FIELD
kvv
LOG LIK.
-5.27 = 0.01
-1.39 + 0.01
- 0.00
-3.96 0.01
- 0.42
Table 1: Per-datapoint log-likelihood on the held-out EEG test set.
10
15
0.4
0.3
0.5
0.4
0.6
x'
Figure 8: Fit of the ConvGNP (kvv) on the EEG data. Each pane shows one of the three channels with unobserved targets (red crosses). All other data (black crosses), including the remaining four channels are observed. Marginals are shown in green, and two samples are shown in blue and pink.
4.4 TEMPERATURE DOWNSCALING FOR ENVIRONMENTAL MODELLING
Lastly, we apply the models a environmental modelling task. In climate modelling, future pro- jections are obtained by simulating the atmospheric equations of motion on a spatio-temporal grid. Unfortunately, computational constraints typically limit spatial resolution to around 100-200km (Eyring et al. 2016), which insufficient to resolve extreme events and produce local projections Allen et al. 2016 Maraun et al. 2017). To address this issuc, so-called statistical downscaling is routinely applied. A mapping from low-resolution simulations to historical data is learnt, and is then applied to future simulations produce high-resolution predictions (Maraun & Widmann 2018).
While numerous data-driven approaches to statistical downscaling exist, they are often limited to making predictions on fixed set of points which historical observations are available (Sachin- dra et al. 2018 Vandal et al.] 2017 2018: 2019] [Bhardwaj et al.) 2018 Misra er al., 2018: White al. 2019 Pan al. 2019 Baño-Medina al. 2020 Liu al. 2020 Recently, Vaughan et al.
[2021 have applied ConvCNPs to temperature downscaling, enabling predictions at arbitrary loca- tions. Though the ConyCNP outperforms an ensemble of existing methods, it unable generate
coherent samples, limiting its practical applicability.
We modify the model of Vaughan et (2021) which maps a low-resolution grid of reanalysis data together with local orographic information, to set of features used to param- eterise Gaussian predictive. We use reanalysis and historical station data throughout Europe, to set up three experiments, where we train on data from the years 1979-2002, and make predictions held out test set from the years 2003-2008, emulating realistic prediction tasks using future climate simulations. The first experiment uses training set of 86 specific stations in Europe. following a standardised experimental protocol known as the VALUE framework (Maraun et al. 2015), for which extensive baselines available. Note that train and test the models the same stations, but at different time periods. The second is a larger experiment consisting stations across Europe and the 86 VALUE stations as held out test The third experiment 713 training and 250 held out test stations, all located in Germany, which has densest station cover- age in Europe. This experiment highlights the benefits of correlated models, since the stations are near one another and are thus highly correlated, For further details on our setup, see appendix Sample
Experimental setup:
training set of 3043
Beeanlviis
45
Train stations
5F
27
10
12
12
LP
14"
WF
12
10
14
Figure 9: Illustration of sampled temperature fields. After training low-res. simulations (top left) and observed data (bottom left), the models are conditioned on future low-res. simulations, to make predictions. The remaining columns show the predicted mean and three samples from the predictive.
8

---

Published as a conference paper at ICLR 2022
Correlations improve performance: Across all three experiments, modelling correlations im- proves the predictive conclude that the correlated models have learnt to model meaningful statistical output dependen- cies. The log-likelihood improvement is greatest for the Germany experiment, where the stations are near one another and thus more strongly correlated. In the VALUE experiment, the correlated Con- vGNPs improve on the ConvNP, as well as on the mean-field ConvGNP, which has itself been shown Vaughan et al. 2021) to outperform an ensemble of widely used statistical downscaling methods.
(table 2). Since small differences are observed in the MAE,
Sampling coherent temperature fields:
Correlated ConyGNP models can be used to sample co- herent temperature fields, something that mean-field models cannot perform. Figure shows that the sampled fields are not only coherent, but also exhibit non- con-trivial correlation structure.
EUROPE (VALUB)
EUROPE (ALL)
GERMANY
CONVGNP (MEAN-FIELD)
-188.12 /106
-176.93 1,41
-663.90 0.94
CONVGNP (LINEAR)
-131.58 / 1.03
-153.23 / 1.33
-224.74 / 0.92
CONVGNP (KVV)
-131.47/ 1.02
-157.25
1.30
-009.06 / 1.00
CONVNP VALUE BASELINE (MAE)
-144.14 1.08
-163.11 1.38
-252.94 / 1.13
1.32
Table 2: Predictive log-likelihood and MAE for the three temperature prediction experiments.
5 RELATED WORK
Conditional and Latent Neural Processes: Our work proposes novel members of the CNP family (Garneoo et al., 2018a Gordon et al.) 2020), which capture output dependencies, and also lend themselves to multi-output and non-Gaussian predictions. Similar approaches which model output correlations include the FullConvGNP model Bruinsma et al. (202T) which, however, cannot be feasibly scaled beyond onc-dimensional input data, due to its use of 2 2D-dimenniinall convolutions. The latent Neural Process family (Garnelo et al. 2018b Kim et al. 2019 Foong et al. (020) is another model class that capture output dependencies. Unfortunately, these models involve latent variables which make the log- liglliiel analytically intractable and complicate training (Le) al. 2018). Petersen al. (202T) recently introduced a model similar to the ConvNP, which uses GP to represent the latent variable of ConvNP-like model, however this too suffers from the same training challenges, and has a computational cost that scales cubically with the context size.
Deep Kernels: The GNP covariances we presented bear similarities to Deep Kernels (DKs; Patac- chiola al. 2019; Calandra et al.) [016)] Wilson et al. (0055). DKs also use neural networks the context of GPs but, unlike GNPs which define a GP predictive, DKs define a GP prior, which is conditioned on data using Bayes' rule. Thus, the computational cost of DKs at test-time is cubic in the context points, whereas that of GNPs is linear, enabling the latter to scale to larger datasets.
Our ConVCGNP models can be regarded as meta-learning versions of the Cop- ula Processes of Wilson & Ghahramani] (2010) and [Jamungal & Ng (2009). These transform the marginals of a GP, defining a non-Gaussian prior process, and perform inference over it, which requires compute which scales cubically with the size of the context. By contrast, ConvGCNPs directly parameterise a predictive, making their test-time complexity linear in the context.
Copula Processes:
6 CONCLUSION
In this work, we present a tractable method for modelling statistical dependencies in the output of CNP. We propose parameterising the covariance of a predictive Gaussian by passing context- dependent feature vectors through positive definite covariance functions. The resulting GNP mod- els. account for output correlations, but unlike existing methods, they can be applied to data with higher-dimension inputs while maintaining an analytically tractable log- ligllikel,hood, which makes them especially easy to train. GNPs can be extended to multi-output regression, and also com- bined with invertible marginal transformations to model non-Gaussian data. We demonstrate that modelling correlations improves predictive performance over mean-field models on Gaussian and non-Gaussian synthetic data, including a downstream estimation task that mean-field models cannot solve. Our models also show improved performance over their mean-field and latent counterparts on real EEG and climate tasks. In statistical temperature downscaling, our models outperform stan- dard ensemble of widely used methods in statistical downscaling, while providing spatially coherent temperature samples. This renders our models suitable for application climate impact studies.
9

---

Published as a conference paper at ICLR 2022
7 ETHICS STATEMENT
We believe that modelling statistical output dependencies with GNPs will enable members of the Neural Process family to be used part of larger estimation pipelines, for example, but not lim- ited to, climate modelling pipelines, to improve human decision making. However, since the ef- fectiveness of meta-learning models depends on the amount and quality of training data available (Bronskill 2020), we acknowledge that relying on over- or under-estimated uncertainties produced by GNPs, may cause adverse outcomes. Training procedures which do safeguard against these pitfalls may result in mis-estimation of statistical dependencies which may have serious practical effects, such as bias and discrimination. This concern however is not unique the GNP family, or to meta-learning, but in virtually every uncertainty-aware probabilistic model. We believe that acknowledging and investigating these issues is useful for producing reliable probabilistic models, to improve our decision making processes.
8 REPRODUCIBILITY
All our experiments are carried out using either synthetic or publicly available datasets. We provide precise details for how to generate the data used for the synthetic experiments in appendices The EEG data is available via the UCI dataset website (link .. The climate modelling data is also available to the public, through the Copernicus European Climate Data Store (link
We also provide the details of our models, such as architectural choices, and the training schemes used in each experiment in appendices[Cioor We publish the complete repository containing all our code, including the models, training scripts, pretrained models and Jupyter notebooks which will produce all plots in this paper (linkF).
ACKNOWLEDGEMENTS
We thank David Duvenaud for helpful comments draft. Wessel P. Bruinsma supported by the Engineering and Physical Research Council (studentship number 10436152). Anna Vaughan acknowledges the UKRI Centre for Doctoral Training the Application of Artificial Intelligence to the study of Environmental Risks (AI4ER), led by the University of Cambridge and British Antarctic Survey, and studentship funding from Google DeepMind. Richard E. Turner is supported by Google, Amazon, ARM, Improbable and EPSRC grant EP/T005386/1.
REFERENCES
SK Allen, Boschung, A Nauels, Y Xia, Bex, and PM Midgley. Climate change 2013: the physical science basis. contribution of working group i the fifth assessment report of the inter- governmental panel on climate change. 2016.
Vladimir I. Amold. Ordinary Differential Equations. Springer- Berlin Heidelberg, 1992.
Jorge Luis MedinM, Rodrigo Gareía Manzanas, José Manuel Gutiérrez Llorente, al. Config- uration and intercomparison of deep learning neural models for statistical downscaling. 2020.
Amit Bhardwaj. Vasubandhu Misra, Akhilesh Mishra, Adrienne Wootten, Ryan Boyles, JH Bowden, and Adam J Terando. Downscaling future climate change projections over puerto rico using a non-hydrostatic atmospheric model. Climatic Change, (33(11)) 2018.
John Bronskill. Data and computation efficient meta-learning. PhD thesis, University of Cambridge, 2020.
Wessel Bruinsma, Eric Perim, William Tebbutt, Scott Hosking, Arno Solin, and Richard Turner. Scalable exact inference in multi-output gaussian processes. In International Conference on Ma- chine Learning, pp. 1190-1201. PMLR, 2020.
10

---

Published as a conference paper at ICLR 2022
Wessel P. Bruinsma, James Requeima, Andrew Y. K. Foong, Jonathan Gordon, and Richard E. Turner. The gaussian neural process, 2021.
Roberto Calandra, Jan Peters, Carl Edward Rasmussen, and Mare Peter Deisenroth. Manifold gaus- sian processes for regression, 2016.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1251-1258, 2017.
Dick Dee, S M? Uppala, AJ Simmons, Paul Berrisford, P Poli, S Kobayashi, U Andrae, MA Bal- maseda, G Balsamo, d Bauer, et al. The cra-interim reanalysis: Configuration and performance of the data assimilation system. Quarterly Journal of the royal meteorological society, 137(656): 553-597, 2011.
Conor Durkan, Artur Bekasov, lain Murray, and George Papamakarios. Neural spline flows, 2019.
Gal Elidan. Copulas in machine learning. In Piotr Jaworski, Fabrizio Durante, and Wolfgang Karl Härdle (eds.), Copulae in Mathematical and Quantitative Finance, pp. 39-60, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg.
Veronika Eyring. Sandrine Bony, Gerald Meehl, Catherine A Senior, Bjom Stevens, Ronald Stouffer, and Karl E Taylor. Overview of the coupled model intercomparison project phase (cmip6) experimental design and organization. Geoscientific Model Development, 955: 1958, 2016.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. International Conference on Machine Learning, pp. 1126-1135. PMLR, 2017.
Andrew Y. K. Foong, Wessel P. Bruinsma, Jonathan Gordon, Yann Dubois, James Requeima, and Richard Turner. Meta-learning stationary stochastic process prediction with convolutional neural processes, 2020.
Marta Garnelo, Dan Rosenbaum, Chris Maddison, Tiago Ramalho, David Saxton, Murray Shana- han, Yee Whye Teh, Danilo J. Rezende, and S. M. Ali Eslami. Conditional neural processes. CoRR, 0abs//807.0166 2018a.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola. Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural processes. CoRR, Abbs/187700162,, 2018b.
Daniel T Gillespie. Exact stochastic simulation of coupled chemical reactions. The journal of physical chemistry, 81(25):2340-2361 1977.
Jonathan Gordon, Wessel P. Bruinsma, Andrew Y. K. Foong, James Requeima, Yann Dubois, and Richard E. Turner. Convolutional conditional neural processes, 2020.
Peter Holderrieth, Michael Hutchinson, and Yee Whye Tch. Equivariant learning of stochastic fields: Gaussian processes and steerable conditional neural processes, 2021.
Sebastian Jaimungal and Eddie KH Ng. Kernel-based copula processes. Joint European Con- ference on Machine Learning and Knowledge Discovery in Databases, pp. 628 643. Springer, 2009.
Makoto Kawano, Wataru Kumagai, Akiyoshi Sannai, Yusuke Iwasawa, and Yutaka Matsuo. Group equivariant conditional neural processes. CoRR, abs/2102.08759 2021.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, S. M. Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. CoRR, abs/1901.05761, 2019.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6 1412 6980, 2014.
11

---

Published as a conference paper at ICLR 2022
AMG Klein Tank, JB Wijngaard, GP Können, Reinhart Böhm, Gaston Demarée, A Gocheva, M Mileta, S Pashiardis, L Hejkrlik, C Kemm-Hasenn et al. Daily dataset of 20th-century sur- face air temperature and precipitation series for the european climate assessment. International Journal of Climatology: A Journal of the Royal Meteorological Society, 22(12):1441-1453 2002.
Ivan Kobyzev, Simon Prince, and Marcus Brubaker. Normalizing flows: An introduction and review of current methods. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Tuan Anh Lc. Hyunjik Kim, Marta Garnelo, Dan Rosenbaum, Jonathan Schwarz, and Yee Whye Tch. Empirical evaluation of neural process objectives. In NeurlPS workshop Bayesian Deep Learning, 2018.
Yumin Liu, Auroop R Ganguly, and Jennifer Dy. Climate downscaling using ynet: deep convo- lutional network with skip connections and fusion. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pp. 3145-3153, 2020.
Douglas Maraun and Martin Widmann. Statistical downscaling and bias correction for climate research. Cambridge University Press, 2018.
Douglas Maraun, Martin Widmann, José M Gutiérrez, Sven Kotlarski, Richard E Chandler, Elke Hertig, Joanna Wibig, Radan Huth, and Renate AI Wilcke. Value: A framework to validate downscaling approaches for climate change studies. Earth's 's Future, 3(1):1-14, 2015.
Douglas Maraun, Theodore G Shepherd, Martin Widmann, Giuseppe Zappa, Daniel Walton, José M Gutiérrez, Stefan Hagemann, Ingo Richter, Pedro MM Soares, Alex Hall, et al. Towards process- informed bias correction of climate change simulations. Nature Climate Change, 7(11):764-773, 2017.
Saptarshi Misra, Sudeshna Sarkar, and Pabitra Mitra. Statistical downscaling of precipitation using long short-term memory recurrent neural networks. Theoretical and applied climatology, 134(3): 1179-1196, 2018
Bernt Oksendal. Stochastic differential equations: an introduction with applications. Springer Science & Business Media, 2013.
Baoxiang Pan, Kuolin Hsu, Amir AghaKouchak, and Soroosh Sorooshian. Improving precipitation estimation using convolutional neural network. Water Resources Research, 55(3):2301-2311, 2019.
Massimiliano Patacchiola, Jack Turner, Elliot J. Crowley, and Amos J. Storkey. Deep kernel transfer in gaussian processes for few-shot learning. CoRR, abs/1910.05199, 2019. URL http://
arxiv. 1910. 05199
Jens Petersen, Gregor Köhler, David Zimmerer, Fabian Isensee, Paul F. Jäger, and Klaus H. Maier- Hein. Gp-convenp: Better generalization for convolutional conditional neural processes time series data. CoRR, abs/2106.04967, 2021. WRRL/ttts://arxxt /2tg 04967
Carl Edward Rasmussen. Gaussian processes in machine learning. In Summer school on machine learning, pp. 63-71. Springer, 2003.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In International Conference Medical image computing and computer- assisted intervention, pp. 234-241. Springer, 2015.
DA Sachindra, Khandakar Ahmed, Md Mamunur Rashid, S Shahid, and BJC Perera. Statistical downscaling of precipitation using machine learning techniques. Atmospheric research. 212: 240-258, 2018.
David M Theobald, Dylan Harrison-Atlas, William B Monahan, and Christine M Albano. Ecologically- maps of landforms and physiographic diversity for climate adaptation plan- ning. PLoS One, 10 10(12):0043361, 2015.
12

---

Published as a conference paper at ICLR 2022
Eleni Triantafillou, Tyler Zhu, Vincent Dumoulin, Pascal Lamblin, Utku Evei, Kelvin Xu, Ross Goroshin, Carles Gelada, Kevin Swersky, Pitrre-Antoine Manzagol, et al. Meta-dataset: dataset of datasets for learning learn from few examples. arXiv preprint arXtv:1903.03090 2019.
Thomas Vandal, Evan Kodra, Sangram Ganguly, Andrew Michaelis, Ramakrishna Nemani, and Auroop R Ganguly. Deepsd: Generating high resolution climate change projections through single image super-resolution. In Proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining, pp. 1663-1672, 2017.
Thomas Vandal, Evan Kodra, Jennifer Dy, Sangram Ganguly, Ramakrishna Nemani, and Auroop R Ganguly. Quantifying uncertainty in discrete-contimuous and skewed data with bayesian deep learning. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2377-2386, 2018.
Thomas Vandal, Evan Kodra, and Auroop R Ganguly. Intercomparison of machine learning methods for statistical downscaling: The case of daily and extreme precipitation. Theoretical and Applied Climatology, 137(1-2):557-5700 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is you need. CoRR, abs/1706.03762, 2017.
URL WRLLhttp:/ /tte:://arae //rxiv org/ abs/1706. 03762
Anna Vaughan, Will Tebbutt, J Scott Hosking, and Richard E Turner. Convolutional conditional neural processes for local climate downscaling. Geoscientific Model Development Discussions, pp. 1-25, 2021.
BL White, A Singh, and A Albert. Downscaling numerical weather models with gans. In AGU Fall Meeting Abstracts, volume 2019, pp. (CC3DD13557) 2019.
Andrew G Wilson and Zoubin Ghahramani. Copula processes. Advances in Neural Information Processing Systems, 23:2460-2468, 2010.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric P. Xing. Deep kernel learning, 2015.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabás Póczos, Ruslan Salakhutdinov, and Alexander J. Smola. Deep sets. CoRR, abs/1703.00614, 2017.
Xiao Lei Zhang, Henri Begleiter, Bernice Porjesz, Wenyu Wang, and Ann Litke. Event related potentials during object recognition tasks, Brain research bulletin, 38(6):331-538, 1995.
13

---

Published as a conference paper at ICLR 2022
A ADDITIONAL THEORETICAL CONSIDERATIONS
In this appendix we provide some additional discussion theoretical aspects of our models. In ap- pendix A.I we provide a proof for the translation equivariance of the ConvGNP, which follows from the fact that the ConvCNP is translation equivariant. In appendix A.2 we provide some additional discussion on composing arbitrary invertible maps such as Normalising Flows (NFs) with neural processes.
A. TRANSLATION EQUIVARTANCE OF THE CONVGNP
In this section of the appendix we provide proof for the translation equivariance of the ConvGNP model. The mean function of ConvGNP has the same form as ConvCNP model (Gordon et al. 2020) and thus translation equivariant. It therefore remains to show that the covariance function of the ConvGNP is also translation equivariant.
As explained in the main text, the covariance the ConvGNP computed as follows. First, a feature function g is applied to the context set and target inputs. This feature function consists of the following sequence of computations
(xe, Yo)
(x, h) D CNND-NN(
O 2 g(arir))
E ( Fr.)) nh
(17)
where step ! maps (Xc, to a D-dimensional grid of values h with corresponding locations at (....... LL), 34 E R.D using a SetConv layer (Gordon et al. 2020p, step ? maps h through a CNN with D-dimensional convolutions, and 3 aggregates using an RBF v. The grid locations & are set using context and target inputs (see Gordon al., 2020 according to
x = gr srrid(mmin Emar), where T'iin = min,(x,, ),), Tmaz = max x
(18)
Lastly, the features outputted by g are passed through a positive definite function 1 to produce the entries of the covariance
Ki = k(g(2,, r).
(19)
To show the translation equivariance of the covariance function, it suffices to show that if trans- lation is applied to both the context and target inputs, then the resulting covariance matrix remains unchanged. In particular, consider applying a translation u to the context and target inputs
x (o ... ,'.N) (Nr.,1 1 + u, u,..., Fe, + u) x = (o (2 x (x1,1 + u...., FrMM + u)
(20)
(21)
and applying y to the translated inputs, according to
(x
hh r'r CNND(h')
2 g(a(,,))) Σ
(22)
1=1
where the grid locations are now X' = ( H1), T1 IR where by substituting x x² into eq. (18) we have
x
(23)
Now, noting that h h, by the translation equivariance of the SetConv layer (Gordon et al. 2020
and r' r, by the equivariance the CNN, and substituting these together with eqs. 20
and 23 into eq. (22) obtain
glol r')
Σ (ol
Σ (aas + ur, + u) r =
Σ d (115, (F
where we have used the stationarity of the RBF v. This shows that is invariant to translations of the context and target inputs. The covariance function of the ConvGNP is therefore equivariant, as required.
14

---

Published as a conference paper at ICLR 2022
A.2 NORMALISING FLOWS AND GENERAL INVERTIBLE MAPS
Composing Normalising Flows with Neural Processes: In section of the main text we explained that general arbitrary maps could be composed with neural processes in an attempt to model joint dependencies and non- Gan-ssssi marginals. However, requiring model that is consistent under marginalisation places theoretical limitations on the form of θ which we can use.
Suppose we have Neural Process with prediction map π. ee Me: xt), which we wish to compose with an invertible map θ Thus, draw a sample from the predictive distribution, we first draw
Theoretical limitations:
(we,1...., Mt, ~ TI (14; the Mor Fz),
(24)
and then apply the marginal transformation
(yt,1...., Mr. (ur,1...... Mr
(25)
The role of O could be either modelling the marginals of the data, representing joint dependencies in the output variable, or both. Now, e must be able to handle variable-length tuples as arguments, to handle arbitrary target points. Thus, should be able query θ with any number M of inputs. In addition, the model should be consistent under marginalisations: making a joint prediction for target variables yt,1 and 1,.2. and then marginalising over 2, should be equivalent to making a prediction over 1 directly. Put concretely, marginalisation consistency requires that the following equality in distribution holds
O(u,, 2r,2): 4 (u,,)).
(26)
For a careful choice of θ and distribution of (u,,1, Mx.2) this equality can true, but general will not be true. If we want to construct an invertible θ such that it gives consistent model for all underlying Neural Processes, then that requires the the stronger condition that
( 1, 41,2); Ouu,.)) almost surely.
(27)
From this condition, see that Q can only be marginal transformation because
(ur,1, 2e,2) (O(u,,) 1), (u,,)))
(28)
and similarly for other M. Though for a given Neural Process there may exist appropriate θ which satisfies marginalisation consistency, constraining problem, which we found to be beyond the scope of this paper.
achieve this challenging research
Marginal maps cannot model joint dependencies: We now note that if i mean-field and θ is marginal transformation, then it follows that the y variables are independent. Therefore, it is not possible to compose a mean-field CNP with an NF to model joint dependencies in this way. Instead, must rely π for modelling dependencies, and θ for learning the marginals.
B COMPUTATION TIME AND MEMORY COMPARISON
Forward pass for one-dimensional models: In this section we provide quantitative measurements for the computational and memory costs of the models. Table the runtime and memory foot- print of the GNP, AGNP, ConyGNP and FullConyGNP models applied data with one-dimensional inputs, D = 1. In particular, we measure the runtime and memory required to perform single for- ward pass through the neural architecture of each model that was used for the one-dimensional Gaussian tasks, on an NVIDIA GeForce RTX 2080 Ti GPU.
The ConyGNP is cheaper than the FullConyGNP: In table we see that the FullConvGNP model requires factor of two more runtime and a factor of five larger memory footprint for this example. should be noted that the performance difference between the ConyGNP and FullConvGNP is less pronounced in the one-dimensional setting, compared to the higher-dimensional settings for two reasons. First, both the runtime and memory footprint of the mullConvGNP increase much quicker than those of the ConvGNP, due to the fact that the FullConvGNP uses 2D 2D-dimensional convolu- tions. Second, the parallelisation of operations in the GPU may wash out some of the differences in runtime in favour of the FullConvGNP. More specifically, in cases where some of the GPU workers are idle, i.c. the GPU is not at its parallelisation limit, additional computations can be carried out at small overhead by using these idle workers. Some of the additional computations required by the
15

---

Published as a conference paper at ICLR 2022
GNP
AGNP
CONVGNP
FULL.CNNVGN
RUNTIME (x 1 SEC) MEMORY (MB)
0.58 0.00
1.62 0.00
1.74 0.00
4.59 + 0.01
0.476
2.971
0.231
1.23
Table 3: Computational and memory costs for one-dimensional models, during a forward using batch size of one, that is, a single task was passed to each model.
FullConvGNP come at reduced runtime cost to what we would expect if the parallclisation limit of the GPU was reached. Since we expect this limit to be reached for larger D, we also expect the performance difference be more pronounced for higher dimensions, for this reason.
Forward pass through a CNN: To further highlight the how the runtime and memory costs scale with the convolution dimension Des we measured the runtime and memory footprint with Da 1, and 3. In particular we used depth-wise separable CNN (Chollet 2017) similar to that used in Bruinsma et al. (202I). consisting of twelve hidden convolutional layers in D. = 1, 3 dimensions, each with a kernel size of 5, measuring the runtime and memory cost of forward pass through the network. Thus for each dimension, the CNN is applied to a tensor with to 2. Figure shows that the runtime and memory required by the CNN increase exponentially with De. Extrapolating Dr 4, observe that the runtime and memory costs become extremely large. making the FullConyGNP difficult to apply, even to data in FullConvGNP requires D. 2D dimensional convolutions. For
CNN
Dr dimensions, each with size of N 128, plus additional channel dimension which
2 dimensions, since the the FullConvGNFP would require De = dimensional convolutions, which is would be significantly above the runtime and memory we can afford with existing GPUs.
Runtime cost
GPU log-runtime
10a
10a
(log sec)
10a
10
10-2
10-3
Memory footprint
Log Log-mennn memory
(log Bytes)
10a
107
109
10a
2
4
Convolution dimension Dr
Figure 10: Scaling of the runtime cost and memory footprint of CNN a function of the convolu- tion dimension. See text for discussion. Error bars have been included for the runtime, but are too small be seen in this plot.
C GAUSSIAN SYNTHETIC EXPERIMENTS
Data generation process: Each synthetic task consists collection of datasets sampled from the same generative process. To generate each of these datasets, we first determine the number of context and target points. We use a random number between 3 and 50 context points and a fixed number of 100 target points. For each dataset sample the inputs of both the context and target points, that
16

---

Published as a conference paper at ICLR 2022
is Xe; X2 uniformly at random in the region 1- 2] for the ID tasks and in |- x [-2, 2] for the 2D tasks. We then sample the corresponding outputs Ye. Y as follows.
Exponentiated Quadratic (EQ): We sample Ye from GP with EQ covariance
KEo (z, x') = of exp (-2
(T 20 x 1 z2)
with parameters (o f) = (1.00, 1.00).
Matern 5/2: We sample ye, from a GP with a covariance
ka x') = o 1 L 382
2
exp (-7);
(29)
where |ar - n/I, with parameters (o () (1.00, 1.00).
Noisy mixture: We sample ye, Yt from a GP which is a sum of two EQ kernels
(z, x²) k 2') + (z,,
with the following parameters (o (1) = (1.00, 1.00) and (o f2) = 01200.
Weakly periodic:
We sample Ve.Yt from GP which is the product of an EQ and a periodic
covariance
kwp(Ir, x²) kEq(Z, 2') exp (-2
( - x'1/p)
2 2 (π| ( -
with EQ parameters (o,, kp) (1.00, 1.00) and periodic parameters (p, LEQ) (0.25, 1.00).
Lastly, for all tasks we add iid Gaussian noise with zero mean and variance of = 0.05². This noise level was not given to the models, which in every case learned noise level from the data. We generate training data for 100 epochs, each consisting of 1024 iterations, each of which contains different tasks. For testing, we use single epoch of 1024 iterations, each of 8 different tasks.
Neural architectures for ID tasks (GNP, AGNP, ANP): For the GNP model we use a fully con- nected neural network consisting of an encoder with six hidden layers of 128 units each, mean aggregation layer and a decoder with a single hidden layer, also with 128 units. For the AGNP model, we use the same architecture for the GNP model, except the aggregation layer consists of a dot product self-attention layer Vaswwani et al. 2017). For the ANP model, we follow the same architecture which used in Kim et al. (2019), which uses deterministic and stochastic path in the encoder. The deterministic path consists of the same encoder used in the GNP and AGNP mod- els, while the stochastic path consists of fully connected network with two hidden layers of 128 units each. The outputs of the deterministic and the stochastic paths are concatenated and passed through the same decoder architecture as for the GNP and the AGNP.
Neural architectures for 1D Gaussian tasks (ConyGNP, ConvNP): We use the same architecture for all the ConvGNP models. This consists of a SetConv layer (Gordon et al. 2020), followed by a pointwise linear transformation with nonlinearity, CNN with UNet architecture (Ronneberger jet al.) 2015), another linear transformation, and lastly a SetConv layer for producing the features required by the mean-field, linear kvv covariance. The first SetConv maps the context set to a discretised grid with 64 points per unit, producing two channels at each point on the grid, referred to as the data and the density channels in Gordon et al. (2020). The pointwise linear transformation maps the two features outputted by the first SetConv to eight features which are fed to the UNet network. The UNet itself consists of six regular convolution layers with kernel size of and stride of 2, and channel sizes
(Cin: Cout) (8, 8), (8, 16), (16, 16), (16, 32), (32,64),
followed by six layers of transpose convolutions, again with a kernel size of 5, a stride of 2 and channel sizes
(Ci, Cow) (64, 32), (64, 32), (64, 16), (32, 16), 8), (16,8).
Note the numbers of channels in the transpose convolutions are of the above dimensions because of the additional connections of the UNet architecture. For the ConvNP model, similar
17

---

Published as a conference paper at ICLR 2022
architecture as for the ConvGNP, except the model contains two stacked UNet networks. The first SetConv of the ConvNP uses the same discretisation of 64 points per unit, as the ConvGNP. Then, the first UNet of the ConvNP is the same the UNet of the ConvGNP, except it has twice the number of output channels. Half of these are used as the mean and the other half as the log-variance of 64 independent Gaussian variables, for each position in the convolution grid. This is followed by another UNet with the same architecture as the ConvGNP UNet, followed by a linear transformation and SetConv layer. The SetConv maps the outputs of the second UNet and target input a mean and log- variance.
Neural architecture for 1D tasks (FullCon+GNP); For the GuIIConvGNP follow Bruinsmal et al. (2021), who use a one-dimensional ConvCNP- for the predictive mean and another two- dimensional convolutional architecture for the predictive covariance. For the precise algorithm of the FuIIConvGNP, we refer the reader to Appendix E.2 of BBuuinsma et al.) Here we give the spe- cific details of the CNNs used in our implementation. Unlike Bruinsma et al., we use UNet-style architectures for both these CNNs, as opposed to depthwise-separabe CNNs, because we find the former to train much quicker both in terms of wall-clock time as well as number of epochs. For the mean parametrisation, we use the same architecture as for the ConvGNP, except this outputs single feature representing the predictive mean. For the covariance architecture we use a discretisa- tion of 30 points per unit use UNet consisting of six layer of two-dimensional convolutions, with a kernel size 5 x 5, a stride of 2 and the same numbers of channels that are used in the mean parameterisation, namely
(Cm, Coui) (8, 8), (8, 16), (16, 16), (16, 32), (32, 64),
followed by six layers of transpose convolutions, also with a kernel size of 5 x and stride of 2, and channel sizes
(Cins Coun) = (64, 32), (64, 32), (64, 16), (32, 16), (32, 8), 8).
Neural architectures for 2D tasks (ConvGNE ConyNP): We use the same UNet architectures for the 2D tasks as those for the ID tasks, except the convolutions are now two-dimensional and the discretisation resolution of the SetConv layer set to 32 points per unit.
General notes: We use the same number of features in the output layer of the GNP, AGNP and ConvGNP models. The mean-field covariance uses two features, one for the mean and one for the marginal variance of the predictive, while the linear and kvv models both use the same number of D = 512 features for computing the predictive covariance. We use a learnable homoscedastic noise variable for all models, and ReLU activation functions for all hidden layers. We optimise all models with Adam (ingmma & Ba 2014), using a learning rate of 10 4, For the ConvNP we use 10 latent samples to evaluate the loss during training, and 512 samples during testing. We do not use any weight regularisation.
D PREDATOR-PREY SYNTHETIC EXPERIMENTS
Data generation process: We broadly follow the method specified in Appendix C.4 of Gordon et al. 2020) for generating data from the Lotka-Volterra Volterra model, which uses the algorithm specified in Gillespie (1977). For each time series, we first sample the predator birth and death parameters 0,, 0g and the 0g, from the distributions
0 ~ 1.00 x 10-2 x Uniform (1 - e, 1 + c),
(30)
0 ~ 5.00 x 10- Uniform( - + c),
(31)
0g 5.00 10
Uniform(1
+c,
(32)
04 1.00 10-2 Uniform(1 (, c))
(33)
where e = 0.1. We choose these parameters following Gordon et al. (2020), because they result in
plausible oscillatory as well as transient behaviours of the predator and prey populations. We then initialise the predator and prey populations at X = 50 and Y = 100 respectively at time / = 0. and perform sequence of discrete steps. At each step one of the following events occur:
1. We sample At Exponential (R-1), where R 0, XY 02X + 0,Y O.XY
18

---

Published as a conference paper at ICLR 2022
2. We sample one of the following events:
(a) A predator is born with probability 0₃ 0,XY/R, R. increasing X by 1.
(b) A predator dies with probability 02 R, decreasing X by 1.
(c) A prey is born with probability 8gY/E, increasing Y by 1.
(d) A prey dies with probability 0,XY R, decreasing Y by 1.
3. Increment t + At and repeat until 100 is reached, when 10 events occur.
Lastly, we linearly interpolate the predator and prey time series and randomly choose context and target points from the range [0. I]. We choose a random number of between and 50 context and a fixed number of 100 target points, to produce a dataset. The linear interpolation is performed because otherwise the algorithm yields many more context and target points at regions where R is large, however we would like the input locations of the data to be independent of the event rate R. Lastly we scale the target outputs by a factor of 102 and add positive constant of 10-2 it. The reason for this last positive constant is that strictly speaking the marginal transformations used in the ConvGCNP models are not differentiable at O, and we circumvent this pathology by adding this constant. We generate training data for 100 epochs, each consisting of 1024 iterations, each of which contains 16 different time series. For testing, we use a single epoch of 1024 iterations, each of 16 different time series.
Neural architectures for the predator-prey tasks (ConyGNP, ConvNP, FulIConvGNP); We use the same model architectures for the ConvGNP, ConvNP and FullConvGNP in the predator-prey tasks as those which we used for the ID synethtic Gaussian experiments, except we modify the discretisation resolution used by the various SetConv layers. More specifically, for the SetConvs of the ConvGNP, the ConvNP and that used in the mean parameterisation of the FullConvGNP, we use a discretisation of 16 points per time unit. For the SetConv used in the covariance architecture of the MalCConyGNP we use 8 points per unit.
Neural architectures for the ConyGCNPs: For the ConvGCNPs we use identical architectures as for their ConvGNP counterparts, except we also compose the model with a marginal transformation. More specifically, we use a marginal transformation 4 A (+))))) +)), where P is the CDF of the exponential distribution
( = 11 e//0
where (+)) = (xx: is an additional feature outputted by the ConvGNP. To avoid numerical instabilities during training, limit v to the range [1, oo), by passing the corresponding raw feature outputted by the ConvGNP through a SoftPlus function and adding to the result.
General notes: We the same number of features in the output layer of the ConvGNP models.
The mean-field covariance uses two features, one for the mean and one for the marginal variance of the predictive, while the linear and kvv models both use the same number of D = 32 features for computing the predictive covariance. We use a learnable heteroscedastic noise variable for all models, and ReLU activation functions for all hidden layers. We optimise all models with Adam, using a learning rate of 5 x 10-1 For the ConvNP we use 16 latent samples to evaluate the loss during training, and 512 samples during testing. We do not use any weight regularisation.
E ELECTROENCEEPALOOGRAN EXPERIMENTS
Details on datasets: For the EEG experiments we use the publicly available EEG dataset, lavailable on the UCI Datasets]] website. We preprocess the data to remove invalid time series, leaving us with 7632 time series across 106 subjects. We pool all 106 subjects together (both control and alcoholic subjects) and sample 86 of the subjects for training. 10 for validation and 10 for testing. Each time series consists of 256 equispaced measurements over 64 EBG channels, from which we keep the seven channels with names FZ, F1, F2, F3, F4, F5. F6, following Bruinsma et al. (2020).
Details on datasets for training the meta-models (ConyGNP, ConyNP): We train each of our meta-models for 1000 epochs, each consisting of 256 iterations, at each of which the model is presented with a batch 16 different tasks. To generate each task, we first select a window size W between 1 and 50. We then choose window of size W from the 256-long time series, and set the
https://kdd. css. usi www..i.ccc//aaaaaaeeeeeeeee
dat..daaaa...... html
19

---

Published as a conference paper at ICLR 2022
channels FZZ, Fl, F2 within this window context points. All other channels in this window, as well as all seven channels outside this window are used as context points.
Details on datasets for testing the models (ConyGNP, ConvNP, MOGP): During the testing phase of the meta- metaa-moeess, and the training phase of the MOGP, we use the same procedure for generating datasets as that used during the training phase of the meta-models,, except a constant W = 50. We evaluate the models on the held-out test set. For the meta-models, we use I testing epoch of 256 iterations, each of which consists of presenting 16 tasks to the model. The MOGP model is trained and tested on individual time series without a meta-learning component, and is used as a non meta-learnt baseline illustrate the benefits of mntaa-eerring. We use 500 randomly sampled tasks, selected in the same way as the datasets used to test the meta-models. For each task, we fit the hyperparameters and the mixing matrix parameters on the context set of each dataset individually, and report the normalised predictive log-likelihood on the target set.
set the window size to
Neural architectures for the EEG tasks (ConvGNP, ConvNP): these EEG experiments we use an on-the-grid ConvCNP architecture for the ConvGNP model, as specified in Goroodnn et al. (2020). More specifically, an on-the grid architecture does not require an input SetConv architecture, but instead uses convolutional layer with positive-constrained weights. Following this convolution, linear pointwise transformation is applied, followed by a UNet CNN, with six convolution and six transpose convolution layers. All layers use a kernel size of 5 and stride of 2. The numbers of channels for the six convolution layers are
(Cans Cous) ) = (16, 32), (32, 64), (64, 128), (128, 256), (256, 512), (512, 1024)
while for the transpose convolutions, the channels are
(Cin: Cout) (1024, 512), (1024, 256), (512, 128), (256, 64), (128, 32), 324,
followed by a last linear transformation mapping the output of CNN the dimensionality pected by the ConvGNP model. An output SetConv is used to interpolate the features to off-the-grid locations for purposes such as sampling. For the ConvNP, we follow the same approach for the synthetic experiments, and stack two UNets with a latent variable layer in between. These are then followed by SetConv for making off-the-grid predictions.
General notes for the meta-model GNP, ConyNP): We optimise both the ConvGNPs and the ConvNP using Adam and a learning rate of x 10-4 4, We use early stopping, selecting the model snapshot with the best performance on the validation set, through all of the 1000 training epochs, as the final model. In no case did we observe overfitting to the training set, even though we used no weight decay. In our experiments, we used Dg 512 features for the linear and kvv models.
Details of the MOGP model: To demonstrate the benefits of the meta-learning approach, we also compare with a baseline multi-output Gaussian process (MOGP) model which, for every task sep- arately. is trained from scratch, without a meta-learning component. To accelerate the training of many MOGP models, we use the approach by Bruinsma et al. (2020); an implementation is openly available on GitHub (link() The MOGP models have p = outputs and m = 3 latent processes with EQ kernels with initial length scales 10-2 All noises are initialised to 10-2 The mixing matrix and all hyperparameters are optimised using scipy's implementation of L-BFGS-B
F ENVIRONMENTAL EXPERIMENTS
F.1 EXPERIMENTAL DESIGN
Experiments are conducted within the VALUE framework (Maraun et al. 2015) to facilitate com- parison to existing downscaling methods. VALUE provides suite of experiments for temperature and precipitation downscaling in an idealised framework mapping from two-degree resolution grid- ded reanalysis observations to station observations for daily maximum temperature at 86 locations around Europe. Models are trained on data from 1979-2003 and evaluated on data from 2003-2008. This mapping is then able to be applied to low-resofution climate model output to generate high resolution future projections for downstream tasks. Formally, we predict temperature y at target (longitude, latitude) location x given low resolution predictors at two degrees resolution and oro- graphic data at We consider three experiments:
httpp:///g http://hhhhubccm///////e////////
20

---

Published as a conference paper at ICLR 2022
1. Europe value only: models are trained context data from ERA-Interin reanalysis from 1979-2002 and the 86 VALUE stations, and evaluated at these same locations for 2003- 2008. The purpose of this experiment is to exactly reproduce the VALUE experiment protocol to facilitate comparison to state of the art baselines.
2. Europe all: In practice, key limitation of current downscaling models is that they can only make predictions at a discrete set of locations determined at training time. Neural process models offer significant advantage the posterior stochastic process can be queried at an arbitrary location at inference time, regardless of the availability of training data. To evaluate how well the GNP models perform making predictions at new locations in the validation period we design a second experiment where models are trained on context data from ERA-Interimt reanalysis from 1979-2002 and 3043 target stations around Europe, and evaluated on context data from 2003-2008 at the 86 VALUE stations.
3. Germany only: Germany has the highest density of stations any country in Europe. As this is the scenario where modelling correlations between targets is most likely to improve performance we conduct a final experiment limiting the domain to Germany. Training is ERA-Interime reanalysis context data from 1979-2002 and station data from 689 stations around Germany, with evaluation from 2003-2008 on 250 held out stations.
Locations of training and test stations for each experiment are shown in fig. 16
Experiment
Experiment 2
Experiment 3
Thain
Test
Figure 11: Locations of the training (top) and test (bottom) target locations for experiment (left), experiment (centre) and experiment 3 (right).
F.2
DATA
Consistent with the VALUE protocol, we use the following context and target data:
Context data Context data are taken from ERA-Interim reanalysis (Dee et al. 2011) from 1979- 2008 with daily temporal resolution and spatial resolution interpolated to two degrees using bilinear interpolation. We consider 25 variables:
Surface: maximum temperature, mean temperature, northward and eastward wind.
Upper atmosphere (850/700/500 hPa): specific humidity, temperature, northward and east- ward wind.
21

---

Published as a conference paper at ICLR 2022
Invariant: angle of sub-grid scale orography, anisotropy of sub-grid scale orography, stan- dard deviation filtered subgrid orography, standard deviation of orography, geopotential, longitude, latitude and day of year transformed to (cos(t), sin(t).
To account for sub-grid-scae topography, the context set also includes topographic variables at each target location. These include the true elevation, difference between reanalysis gridscale and true elevation and m'PII (Theobald et al. 2015).
Target data Target data are taken from weather station observations from 3129 weather stations around Europe reported part of the European Climate Assessment Dataset (Klein Tank al.) 2002). This dataset includes daily observations of maximum temperature from 1979-2008.
F.3 NEURAL ARCHITECTURES AND TRAINING
For the architectures we follow (Vaughan et al. 2021). The encoder consists of a CNN with residual blocks, each consisting of two layers of depth-separable convolutions (Chollet 2017) with kernel size of and 128 channels followed by ReL.U activations. The encoder followed by SetConv layer with RBF kemne.. Finally, a MLP is used to update predictions given the clevation of the target locations. This MLP consist of four hidden layers each with 64 units and ReLU activations. For the ConvGNP-1 near and ConvGNP-kve models the covariance. For the ConvNP we use 32 channels for the latent function and 24 samples to calculate the neural process maximum likelihood training time. Each model trained for 500 epochs.
Dy 128 features for computing
F.4 ADDITIONAL SAMPLES
This section shows additional examples similar to fig. comparing the ConvCNP and ConvGNP samples for different days.
ERA Beomalesis
Predicted
Sumple
Sumple 2
Sample
32"
10
Train statines
-1
12
14"
10
Figure 12: As for Figure 9, but for 12/01/2003.
ERA-1 Rocachoie
Predicsed y
Somple
Sample
Sample
10"
500
47
Train stations
52"
632
Figure 13: As for Figure 9, but for 24/02/2003.
22

---

Published as a conference paper at ICLR 2022
Romnlysis
Prediced j
Sample
Samale
Sumple
top-ank
Train stations
-
147
30
12
30
12
14'
11
12"
14
14
Figure 14: As for Figure but for 06/03/2004.
ERA- Hesmalois
Precected
Somple
Sample
Sampoe
CapcUN lnsonnilll
34"
22"
Thin stations
Can (ann)
Figure 15: As for Figure 9, but for 23/01/2006.
Prodicted p
Sample
Sample
Train stations
11F
12
12
147
35
12
T1P
12"
1.
34"
Figure As for Figure 9, but for 04/04/2007.
23


Reviews:
[{"review_id": "rfqo51TyZy4", "reviewer": "Reviewer_g5Qp", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors introduce a relatively straightforward extension for Gaussian neural processes in which both mean and (half of the) covariance functions are specified as neural networks, and the covariance function is either explicitly calculated as an inner product in (7) or as a squared exponential covariance function modulated in magnitude by an auxiliary neural network and calculated using the outputs of a neural network rather than the input data itself (x_t, x_c, y_c). Their multi-output and non-Gaussian strategy follows the modulated kernel and Gaussian copula formulation, respectively. Comprehensive experiments on both artificial and real data demonstrate the advantages of the proposed model over the related, but more computationally expensive, fullConvGNP model. Moreover, on real data, the proposed model outperforms both mean field, convNP and MOGP approximations.\n\nSomething that is not discussed in the paper is the setting of the length scale of the squared exponential kernel.\n\nConsidering that one of the motivations of the proposed approach is how prohibitive existing approaches are, having estimates of computational cost and/or runtime experiments could be a welcome addition to the paper.", "summary_of_the_paper": "The authors present a class of neural process models that are able to produce correlated predictions while amenable to exact, simple and scalable maximum likelihood optimization supporting multiple outputs. By using invertible transformations (gaussian copula), the model is able to capture non-Gaussian output distributions. Experiments with artificial and real data (EEG and climate), highlight the predictive ability of the proposed model.", "summary_of_the_review": "The proposed approach though technically simple relative to existing literature in neural and Gaussian process literature it is well motivated, technically sound and with comprehensive experimental results that support the improved predictive performance claimed by the authors.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YymvgNmDOg-", "reviewer": "Reviewer_WnVi", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nWhen I saw 'Tractable Dependent Predictions' in the title, I assumed that a normalizing flow was being used to capture the joint distribution. This is a modern, flexible family of density estimators for which computing the likelihood is tractable. I was disappointed when I found that these weren't used in the paper. The copula model is very close to a full normalizing flow model. Can you explain what would be necessary to extend your method to a full normalizing flow?\n\nI found the discussion of fullconvgp inadequate, as it seems like the most relevant baseline for capturing these dependencies. It would be very helpful if you included an equation or two. I shouldn't need to dig into the literature to understand the difference between your method and prior work. What exactly is the difference between fullconvgp and convgnp?\n\nI was disappointed that there were no error bars. What sources of variance are you averaging over?\n\nCan you make the analysis of fig 3 more quantitative? Right now, the assertion that the samples are better is too qualitative. Can you check, for example, a q-q plot at a few x axis locations?\n\nI'm curious how your model behaves when the data truly has diagonal covariance. I expect that there are off-diagonal artifacts due to overfitting. Can you run a quick experiment to check this?\n\n\n\n", "summary_of_the_paper": "There is a long line of recent interesting work on neural processes, a scalable and more flexible alternative to GPs for performing prediction at a set of test points (x1, ..., xm) given a conditioning set ((x, y)_1, ..., (x, y)_n). This mapping is learned via meta-learning. \n\nThis paper addresses a core issue of the popular conditional neural process: the predictions at each test point are conditionally independent given the conditioning set. This is an inappropriate modeling assumption for many real-world datasets. In response, the authors propose to go beyond a non-diagonal Gaussian to describe the joint distribution. For example, they use some structured Gaussian covariances (linear, kvv) and also a Gaussian copula model. \n\nThey demonstrate both qualitatively and quantitatively that modeling these dependencies improves the performance of the model on a variety of datasets spanning application domains.\n\n\n", "summary_of_the_review": "I feel that the contribution is fairly incremental and I'm disappointed that they did not consider full normalizing flow models. However, I found the exposition engaging and the experiments thorough.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VMFGTxdBOZT", "reviewer": "Reviewer_Z7Bg", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths \n1. The proposed methods are easy to understand and implement. \n2.  The way of modeling dependencies in the input (or the output) through a covariance function looks simple but indeed improves the performance.\n3. Using Copulae, the proposed methods can handle non-Gaussian distributions\n\nWeaknesses \n1. I think the novelty of the proposed methods is not striking. The proposed method looks like a direct implementation of a Gaussian process with deep kernel learning in the meta learning setting (through the form of NP): just including the global representation vector \\mathbb{r} into the mean and the covariance functions for a GP.    \n\n2. The descriptions (in Introduction) related the reference [Bruinsma et al., 2021] looks a little bit confusing. First, the name “Gaussian Neural Process” (and the concept of modeling a Gaussian predictive using neural networks) is already introduced in the reference paper. Second, it sounds that the proposed methods are belonging to the same class as the FullConvGNP [Bruinsma et al., 2021] is (due to the same class name “Gaussian Neural Process”). Are the proposed methods also translation equivariant as the FullConvGNP is? If this is true, it is required to include proofs. If not, I think the introduction section should be carefully revised to clearly differentiate the proposed methods from the FullConvGNP [Bruinsma et al., 2021].      \n", "summary_of_the_paper": "The manuscript proposes variants of Neural process (NP), which can model correlation in the input (and in the output for multiouput regression). The main idea is to directly parameterize the mean and the covariance functions of a Gaussian predictive via neural networks. The authors also propose to use Copulae to handle non-Gaussian marginal distributions.  ", "summary_of_the_review": "I like that fact that the authors include multiple data sets that can show the effectiveness of modeling dependencies in the input (or the output). However, I think the novelty of the proposed methods is not that impressive. \n\nMinor comments\n\nIn the end of page 1: “CNPs model their respective outputs y_m and y_m’ independently, that is y_m \\prep y_m’”. It would be better if the mathematical definition of the symbol “\\prep” was included in the manuscript. \n\nThe current version of the manuscript is not printable (on Windows 10). I tried to print the manuscript out from Adobe reader and Chrome several times but failed to do it. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a class of neural processes that lifts the limitations of conditional neural processes (CNPs) and produces dependent/correlated outputs but that, as CNPs, is inherently scalable and it is easy to train via maximum likelihood. The proposed model is extended to multi-output regression and to capture non-Gaussian output distributions. Results are presented on synthetic data, an electroencephalogram dataset and on a climate modeling problem. The paper parameterizes the prediction map as a Gaussian, where the mean and covariance are determined using neural networks. Non-Gaussian prediction maps are obtained using copulas. 

Technically speaking, the reviewers found the approach to be incremental and only marginally significant and I agree with them. Issues such as estimates of computational cost, using fixed lengthscales for the covariances and relationships/using normalizing flows have been addressed by the authors satisfactorily. Empirically, the contribution of the paper is somewhat significant, as it provides similar flexibility to other more computationally expensive processes and more general assumptions than conditional neural processes.
Decision: Accept (Poster)