Convolutional Conditional Neural Processes
Authors: Jonathan Gordon
Wessel P. Bruinsma
Andrew Y. K. Foong
James Requeima
Yann Dubois
Richard E. Turner
Venue:  (2020)
Keywords: Neural Processes
Deep Sets
Translation Equivariance

Abstract: We introduce the Convolutional Conditional Neural Process (ConvCNP), a new member of the Neural Process family that models translation equivariance in the data. Translation equivariance is an important inductive bias for many learning problems including time series modelling, spatial data, and images. The model embeds data sets into an infinite-dimensional function space, as opposed to finite-dimensional vector spaces. To formalize this notion, we extend the theory of neural representations of sets to include functional representations, and demonstrate that any translation-equivariant embedding can be represented using a convolutional deep-set. We evaluate ConvCNPs in several settings, demonstrating that they achieve state-of-the-art performance compared to existing NPs. We demonstrate that building in translation equivariance enables zero-shot generalization to challenging, out-of-domain tasks.

Published as a conference paper at ICLR 2020
CONVOLUTIONAL CONDITIONAL NEURAL PROCESSES
Jonathan Gordon
Wessel P. Bruinsma"
Andrew Y. K. Foong
University of Cambridge
University of Cambridge
University of Cambridge yk cam. ac.uk
5g8010cam. aeam.acc.u ac.uk
Invenia Labs wpb2 3 cam. ac. uk
James Requeima
Yann Dubois
Richard E. Turner
University of Cambridge Invenia Labs jrr410 cam. ac. uk
University of Cambridge
University of Cambridge Microsoft Research re e 2 6 cam ac uk
yannduboi s 60 gmaal. com
ABSTRACT
We introduce the Convolutional Conditional Neural Process (CONVCNP), a new member of the Neural Process family that models translation equivariance in the data. Translation equivariance is important inductive bias for many learning problems including time series modelling, spatial data, and images. The model embeds data sets into infiniteddimensionall function space opposed to finite- dimensional vector space. To formalize this notion, we extend the theory of neural representations of sets to include functional representations, and demonstrate that any vranslatio-eeqiiaaiinn embedding can be represented using a convolutional deep set. We evaluate CONVCNPs in several settings, demonstrating that they achieve ttate-of-ute oitte-of-the-arr performance compared to existing NPs. We demonstrate that building in translation equivariance enables zero-shot generalization to chal- lenging, out-of-domain tasks.
INTRODUCTION
rich class of models that define conditional distribution p(y| xx 2, 0) over output variables y given input variables æ, parameters 0, and a set of observed data points in a context sel Z {m Im JMM 1m=1- key component of NPs is the embedding of context sets Z into a representation space through an encoder Z + (Z)), which is achieved using DEEPSETS function approximator (Zaheer et al., (017)) This simple model specification allows NPs to be used for (i) meta-learning (Thrun & Prati 2012 Schmidhuber 1987). since predictions can be generated on the fly from new context sets test time; and (ii) multi-task or transfer learning (Requeima et al. 2019), since they provide a natural way of sharing information between data sets. Moreover, conditional NPs (CNPs: Garnelo et al. (018a),, a deterministic variant of NPs, can be trained in particularly simple way with maximum likelihood learning of the parameters O, which mimics how the system used at test time, leading to strong performance (Gordon et al. 2019
Neural Processes (NPs: Garnelo et al. 2018b
Natural application areas of NPs include time series, spatial data, and images with missing values. Consequently, such domains have been used extensively to benchmark current NPs (Garnelo et al.. 2018a Kim al. 2019). Often, ideal solutions to prediction problems in such domains should be translation equivariant: if the data are translated in time or space, then the predictions should be translated correspondingly (Kondor & Trivedi 2018 Cohen & Welling, 2016). This relates to the notion of stationarity. As such, NPs would ideally have translation equivariance built directly into the modelling assumptions as inductive bias. Unfortunately, current NP models must learn this structure from the data set instead, which is sample and parameter inefficient as well as impacting the ability of the models to generalize.
The goal of this paper is to build translation equivariance into NPs. Famously, convolutional neural networks (CNNs) added translation equivariance to standard multilayer perceptrons (LeCun et al.. 1998 Cohen & Welling 2016). However, it is not straightforward to generalize NPs in an analogous way: (i) CNNs require data to live "on the grid" (e.g. image pixels form a regularly spaced grid), while many of the above domains have data that live "off the grid" (e.g. time series data may be observed irregularly at any time & R). (ii) NPs operate on partially observed context sets whereas
Authors contributed equally. Complete description of author contributions Appendix

---

Published as a conference paper at ICLR 2020
CNNs typically do not. (iii) NPs rely embedding sets into tnite-dimennsonal vector space for which the notion of equivariance with respect to input translations is not natural, as we detail in Section 3] In this work, we introduce the CONVCNP, new member of the NP family that accounts for translation equivariance [ This is achieved by extending the theory of learning on sets to include functional representations, which in turn be used to express any translation-equivarian NP model. Our key contributions can be summarized as follows.
(i) We provide a representation theorem for translation translation-cquiiaaiaat functions sets, extending a key result of Zaheer et al. (017)) to functional embeddings, including sets of varying size.
(ii) We extend the NP family of models to include translation equivariance.
(iii) We evaluate the CONVCNP and demonstrate that it exhibits excellent performance on several synthetic and real-world benchmarks.
BACKGROUND AND FORMAL PROBLEM STATEMENT
In this section we introduce the notation and precisely define the problem this paper addresses.
Notation. In the following, let X' = Rd and y C Rdd (with y compact) be the spaces of inputs and outputs respectively. To ease notation, we often assume scalar outputs Y [ R. Define Zas (X x > as the collection of M onput-output pairs, Z U Zm as the collection of at most M pairs, and Z = U 2m as the collection of finitely many pairs. Since we will consider permutation -inaariaivint functions on Z (defined later in Property I), we may refer to elements of Z as sets data sets. Furthermore, we will use the notation [1] (1).
Conditional Neural Processes (CNPs). CNPs model predictive distributions as p(y|æ, 2) P(y] (x, 2), 0), where & defined as composition p o E of an encoder E: Z + R' map- ping into the embedding space R' and a decoder p: R - C'','' 3). Here E E' is a vector representation of the set 7, and Cx is the space of continuous, bounded functions X - y endowed with the supremum norm. While NPs (Garnelo et al.) (018b) employ latent variables to indirectly specify predictive distributions, in this work we focus on CNP models which do not.
As noted by Lee et al. 2019);; Blody & Teh 2019), since E is function on sets, the form of / in CNPs tightly relates to the growing literature on learning and representing functions on sets (Zaheer et al., 2017 Qi et al. 2017a Wagstaff et al. 2019). Central to this body of work is the notion that, because the elements of a set have no order, functions on sets are naturally permutation invariant. Hence, to view functions on Z functions sets, we require such functions to be permutation invariant, This notion is formalized in Property 1
Property 1 (S,, innariint and S-invariant functions). Let S, be the group of permutations of symbols for n E N. A function & on Z,, is called S,,iinariant if
(Z,,) Zz) for all π E S., and Zn E Zn, where the application of to Zn is defined TZn ((Ix(1), Vr(1),..., (Iz(x) Vr(n)))- A function & on 2 is called S-invariant if the restrictions $ z,, are S,, -,,iivaaiant for all n.
Zaheer et (0177) demonstrate that any continuous -ivvirri function f: Zx + R has sum-decomposition WWagstaff et al. 2019) i.e. a representation of the form f(Z) = f())=p(E=;e 2(2)) for appropriate P and Φ (though this could only be shown for fxed-sized sets). This is indeed the form employed by the NP family for the encoder that embeds sets into latent representation.
Translation equivariance. The focus of this work is on models that are translation equivariant: the input locations of the data are translated by amount then the predictions should be translated correspondingly. Translation equivariance for functions operating on sets is formalized in Property
Property 2 (Translation equivariant mappings on sets). Let H. be an appropriate space of functions on X, and define T and T" as follows:
T X'xx 2 2,
T,Z ((0) + T, v1), (Im T, ym
T' X x H
T=M()) h(x -)).
Source code available at http://pp////ggtt.....o////////e convenn
2

---

Published as a conference paper at ICLR 2020
Then mapping Φ: 2 - H. is called translation equivariant if ( Tz)(Z) for all X and Z E Z.
Having formalized the problem, we now describe how construct CNPs that translation equivariant.
CONVOLUTIONAL DEEP SETS
We are interested translation equivariance (Property 2) with respect to translations on X. The NP family encoder maps sets Z equivariance with respect to input translations in X' is not well defined. For example, a function f on X' can be translated by T E Xt: f(- 7). However, for a vector æ E Rd which can be seen as a function [d] + R. (i) = I₁, the translation æ( T) is not well-defined. To overcome this to map into a function space H. containing functions on X. Since functions in H map from X, our notion of translation equivariance (Property 2) is now also well defined for (Z). As we demonstrate below, every translation-en raansation-equiivriaan function on sets has representation in terms of a specific functional embedding.
embedding in a vector space R4 for which the notion of
issue, we enrich the encoder F: 2
Definition (Functional mappings on sets and functional representations of sets), Call a map E: 2 + H functional mapping on sets if it maps from sets 2 to an appropriate space of functions H. Furthermore, call ( ) the functional representation of the Z.
Considering functional representations of sets leads to the key result of this work, which can be summarized as follows. For Z' C appropriate, a continuous function 4: Z' + Cb(X satisfies Properties aadd 2 if and only if it has representation of the form
+ (Z) p (F(Z)),
F(Z) = (( - x) E H.
for continuous and translation-equivane that p is a map between function spaces. We also remark that continuity of + is not in the usual sense; return this below.
p: H Cx(X))), and appropriate @ and wh. Note
Equation (1) defines the encoder used by our proposed model, the CONVCNP. In Section 3.1 we present theoretical results in more detail. In particular, Theorem eestablishes equivalence between any function satisfying Properties [] and the representational form in Equation (1) In doing so, provide extension of the key result Zaheer al. (2017) 7 functional representations and show that it can naturally be extended to handle varying-size sets. The practical implementation of CONvCNPs the design of p. o, and 1b- - is informed by our results in /Section 3.1 (as well as the proofs, provided in Anpppeiix A), and is discussed for domains of interest in Section 4
sets,
3.1 REPRESENTATIONS OF TRANSLATION EQUIVARIANT FUNCTIONS ON SETS
In this section we establish the theoretical foundation of the CONYCNP, We begin by stating definition that is used in our main result. We denote [m] m......).
Definition 2 (Multiplicityy. A collection Z' is said have multiplicity K if, for every set Z E Z', every æ occurs at most K times:
mult z' := sup {sup {i [m]] x H 31 Fm (R,, D1 E z') = K.
number every
For example, in the case of real-world data like time series and images, we often observe only one (possibly nulti-dimensional) observation per input location, which corresponds multiplicity one. We are now ready to state our key theorem.
Theorem 1. Consider an appropriate collection Z'.M ZeM with multiplicity K. Then function Φ: Z'AM - C)( is continuous permutation invariant (Property 1), and translation
For every m E [M], Z'SM M O Zm must topologically closed and closed under permutations and translations.
For every [M]. the restriction + 2/ 72m is continuous.
3

---

Published as a conference paper at ICLR 2020
Coutest sel Ze = (atn hnnm
Functioonal representation Evaluate at discrelization (4:)
Apply CNN and prediet
plus 12 Ze)
x h00) - Σ Dt Rnd (density channel)
600
E an V(
- 20 Σ Ve -xA)
fulte) (efolte) Pater -4)
(a)
require: p (CNN, 1/)), th, and density Y
require: p CNN and CONVØ
require: context (In, Un) ) target (#rm) ) begin
require: image context Mc, and target mask M.
begin
lower, upper + ((a,) ( U (Z (t.) + uniform uniform__grddllwee grid(lower, upper; ?)
2
I/ We discretize the pixel locations.
3
Ze + M. 0
// Extract context set.
h t En h + h( /h!
Un] witt x.n)
4
h + CONVo (Mc, Ze] 7
h (1.)) + (l:C) /h
s
(fa(t.), fo
CN,((,,, ho)
ft + Mr @ CNN(())
pm + Ex (olt)) -4)
om - E (t))) -t,
[S) (C(2222)))
return (J4m: m m=
return (H,
10 end
end
(b)
(c)
Figure I: (a) Illustration of the CONv forward pass in the off-the--grid case and pseudo-code for (b) off-the-grid and (c) on-the-grid data. The function pos: IR - (0, oo) is used to enforce positivity.
equivariant (Property 2) and only if it has representation of the form
((Z) = (E(()), E((x1,))).
v.)...., (Im, 3mm) = En o()))(( -xx)
for some continuous and translation translation-equivarian
H. + C,(X,)) and some continuous Φ: y - I. K+1 and :p: X R. where H. an appropriate space of functions that includes the image of E. We call function & of the above form CONvDEEPSE
The proof of Theorem is provided in Appendix A We here discuss several key points from the proof that have practical implications and provide insights for the design of CONyCNPs: (i) For the construction of p and E, set flexible positive-definit kernel associated with Repro- ducing Kernel Hilbert Space (RKHS; Aronszajn (1950)) which results in desirable properties for F. (ii) Using the work by Zaheer et al. (2017). we set ((y) (y y ... )² ) to the powers of up order K. Theorem requires o to be powerful function approximator of continuous, innslaaioo-equivariin maps between functions. In Seccionn 4 we discuss how these theoretical results inform our implementations of CONVCNPs,
Theorem extends the result of Zaheer et al. (217)) discussed in [Section 2|by embedding set into an infinie-ddimensionll space the RKHS instead of a finite dintessoinn space. Beyond allowing the model to exhibit translation equivariance, the RKHS formalism allows us to naturally deal with finite sets of varying sizes, which turns to be challenging with finite-dimensional embeddings. Furthermore, our formalism requires p(y) (y² y ,.....,,,. ²) to expand up to order no more than the multiplicity of the sets K; if K is bounded, then our results hold for sets up any arbitrarily large finite size M, while fixing 6 to be only (K + 1)-dimensional.
4 CONVOLUTIONAL CONDITIONAL NEURAL PROCESSES
In this section discuss the architectures and implementation details for CONvCNPs. Similar to NPs, CONvCNPs model the conditional distribution as
P x, Z) = I_ py D (z.)) = II (y Mn. Σn) with (H4, E..) = (b((2)<x)))
(2)
where Z is the observed data and a CEENDEEPSET. The key considerations are the design of p. o. and và for t. We provide separate models for data that lie on-the-grid and data that lie off-the-grid.
4

---

Published as a conference paper at ICLR 2020
Form of Φ. The applications considered in this work have single (potentially multi- dimensional) output per input location, so the multiplicity of Z is one (i.e., K = 1). It then suffices to let ø be power series of order one, which is equivalent to appending constant to y in all data sets, i.e. ((y) = [1 y] The first output Φ1 thus provides the model with information regarding where data has been observed, which is necessary to distinguish between observed datapoint at a: and a datapoint at æ with = O. Denoting the functional representation as h, we can think of the first channel (0) as a "density channel". We found it helpful to divide the remaining channels (11)) by h, FFguuee 1b. line 5), as this improved performance when there large variation in the density of input locations. In the image processing literature, this is known as normalized convolution (Knutsson & Westtn 1993). The normalization operation can be reversed by and is therefore not restrictive.
CoNvCNPs for off-the-grid data. Having specified o, it remains to specify the form of to and p. Our proof of Theorem (suggests that &b should be a stationary, non-negative, positive-defimie kernel. (EQ) kernel with a learnable length scale parameter is natural choice. This kernel is multiplied by Φ to form the functional representation E(Z) (Figure 1b line 4; and
The sxponentiatedqquadraiee
Figure Ta. arrow 1).
Next. Theorem T suggests that / should be a continuous, translation-equivariaa map between function spaces. Kondor & Trivedi (2018) show that, in deep learning, any censlltion-cquivarte model has representation as a CNN. However, CNNs operate on discrete (on-the-grid) input spaces and produce discrete outputs. order to approximate p with a CNN, we discretize the input p. apply the CNN, and finally transform the CNN output back to a continuous function X' + y. To do this, for each context and test set, we space points (t:);= S X on uniform grid (at a pre-specified density) over a hyper-cube that covers both the context and target inputs. We then evaluate ( (F( (E)))())) ) (Figure 1b lines 2-3: Figure la. arrow 2). This discretized representation of E()) is then passed through a CNN (Figure 1b] line 6: Figure 1a artow 3).
To map the output of the CNN back to a continuous function X' + V/ we use the CNN outputs as weights for evenly-spaced basis functions (again employing the EQ kernel), which we denote by v (Figure 16 lines Figure la| arrow 3). The resulting approximation to is not perfectly translation equivariant, but will be approximately The resulting continuous functions are then used to generate the (Gaussian) predictive mean and variance at any input. This, in turn, can be used evaluate the log-likelihood.
for length scales larger than the spacing of (E(Z) (t))
CONVCNP for on-the-grid data. While CONVCNP is readily applicable to many settings where data live on grid, in this work we focus the image setting. As such, the following description uses the image completion task as an example, which is often used to benchmark NPs (Garnelo et al. 2018a Kim al. 2019). Compared the off-the-grid case, the implementation becomes simpler as we can choose the discretization (t.) to be the pixel locations.
Let E XC be an image - H, W, C denote the height, width, and number of channels, respectively and let Mc be the context mask, which is such that [M.]i. if pixel location (i, j) is in the context set, and 0 otherwise. To implement é, we select all context points, Z.- := Me ① I, and prepend the context mask: á [Me, Ze] Fiiuuee 1c line 4).
Next, we apply convolution the context mask form the density channel: (0) (O.) (Figure lc line 4). To all other channels, we apply normalized convolution: h (1:)))
CoNVo(y)/ht (0) Figure 1c line 5), where the division is element-wise. The filter of the con- volution is analogous to 1h. which means that is the functional representation, with the convolution performing the role of E (the summation in [Fiuure 1b line 4). Although the theory suggests using non-negative, positive-definite kernel, we did not find significant empirical differences between an EQ kernel and using a fully trainable kernel restricted to positive values to enforce non-negativity (see Appendices D.4 and D.5 for details).
Lastly, we describe the on-the-grid version of p(.), which consists of two stages. First, we apply CNN to E(Z) (Figure Ic| line 6). Second, we apply a shared, pointwise MLP that maps the output of the CNN at each pixel location in the target set to 2C where we absorb MLP into the CNN (MLP can be viewed as an 1x1 convolution). The first C' outputs are the means of a Gaussian predictive distribution and the second C the standard deviations, which then pass through a positivity-enforrnn
5

---

Published as a conference paper at ICLR 2020
function FFigure lc. line 7-8). To summarise, the on-the-gee the-grid algorithm is given by
E(context set)
(a)) CNNN (NN((coN(MM.) CONV(M, & / CONV(M
(3)
P
density channel
multiplies by sums
where (He, or) are the image mean and standard deviation, p is implemented with CNN, and E is implemented with the mask M. and convolution CONV.
Training. Denoting the data set D [Zn S 2 and the parameters by 0, maximum-likelihood training involves (Garnelo et al. 2018a
0 arg max Σn E(n.y))...., logply Lo (o(Zn.))(=)))
where we have split 7, into context (Z,n,c and target (Z.n.) sets. This standard practice in the NP (Garnelo et al. 2018a b) and meta-lcarning settings (Finn et al. 20177] Gordon et al. 2019)) and relates to neural auto-regressive models (Requeima et al. 2019). Note that the context set and target set disjoint (Z,,c Zn. 0), which differs from the protocol for the NP (Garnelo et al. 2018a).
Practically, stochastic gradient descent methods (Bottou 2010) can be used for optimization.
5 EXPERIMENTS AND RESULTS
We evaluate the performance of CONVCNPs in both on-the-grid and off-the-grid settings focusing on two central questions: (i) Do translation-equiivrianl models improve performance in appropriate domains? (ii) Can translation equivariance enable CONvCNPs to generalize settings outside of those encountered during training? We use several off-the- the-grid data-sets which are irregularly sampled time series (X' R), comparing to Gaussian processes (GPs; Williams & Rasmussen (2006)) and ATTNCNP(whict is identical to the ANP (Kim et al.) 2019). but without the latent path in the encoder), the best performing member of the CNP family. We then evaluate on several on-the-grid image data sets (X' = 2²²). In all settings we demonstrate substantial improvements over existing neural process models. For the CNN component of our model, we propose small and large architecture for each experiment (in the experimental sections named CONvCNP and CONv respectively). We note that these architectures are different for off-the--riid and on-the-grid experiments, with full details regarding the architectures given in the appendices.
5.1 SYNTHETIC ID EXPERIMENTS
First we consider synthetic regression problems. At each iteration, function sampled, followed by context and target sets. Beyond EQ-kerel GPs (as proposed in Garnelo et al. (2018a); Kim et al. (2019).. we consider more complex data arising from Matern 5 and geeakl--periodie kernels, as well as challenging, non-Gaussian sawtooth process with random shift and frequency (see Figure 2 for example). CONVCNP is compared to CNP (Garnelo al.) (2018a) and ATTNCNP. Training and testing procedures are fixed across all models. Full details on models, data generation, and training procedures are provided in Appendix
Table 1: Log-likelihood from synthetic 1-dimensional experiments.
Model
Params
EQ
Weak Periodic
Matern
Sawtooth
CNP
66818
-0.86 + 3e-3
-1.23 + 2e-3
-0.95 + 1e-3
-0.16 + le-5
ATTNCNP
149250
0.72 + 4e-3
-1.20 = 2e-3
0.10 + 2e-3
-0.16 + 2c-3
CONVCNP
6537
0.70
5e-3
-0.92 = 2c-3
0.32 4e-3
1.43 i 4e-3
CONVCNPXL 50617
1.06 + 4e-3
-0.65 + 2e-3
0.53 + 4e-3
1.94 + 1e-3
Table reports the log- likellhooood means and standard errors of the models over 1000 tasks. The context and target points for both training and testing lie within the interval [-2, 2] where training data was observed (marked "training data range" in Figure 2) Table deemonstattes that, even when extrapolation is not required, CoNvCNP significantly outperforms other models in all cases, despite having fewer parameters.
Figure 2 demonstrates that CONVCNP generates excellent fits, even for challenging functions such from the Matern-5 kernel and sawtooth. Moreover, Figure compares the performance

---

Published as a conference paper at ICLR 2020
Nodel
Conrett
AFINCNP
CONVCNP
Trabiing
sage
Figure 2: Example functions learned by the ATTNCNP (top row), and CONVCNP (bottom row), when trained on a Matern- kernel with length scale 0.25 (first and second column) and sawtooth function (third and fourth column). Columns one and three show the predictive posterior of the models when data is presented same range as training, with predictive posteriors continuing beyond that range on either side. Columns two and four show model predictive posteriors when presented with data outside the training data range. Plots show means and two standard deviations.
of CONVCNP and ATTNCNP when data is observed outside the range where the models were trained: translation equivariance enables CONVCNP to elegantly generalize to this setting, whereas ATTNCNP is unable to generate reasonable predictions.
5.2 PLASTICC EXPERIMENTS
The PLASTiCC data set (Allam Jr et al. 2018 is a simulation of transients observed by the LSST telescope under realistic observational conditions. The data set contains 3,500,734 "light curves", where each measurement is of an object's brightness as a function of time, taken by measuring the photon flux six different astronomical filters. The data can be treated as a six-dimensional time series. The data set was introduced in Kaggle competition." where the task was to use these light curves to classify the variable sources. The winning entry (Avocado, [Boone| (019) modeled the light curves with GPs and used these models to generate features for a gradient boosted decision tree classifier. We compare a mulli-input-multiouutpu CONVCNP with the GP models used in Avocado CONvCNP accepts six channels as inputs, one for each astronomical filter, and returns 12 outputs- the means and standard deviations of six Gaussians. Full experimental details are given in Appendix C.3 The mean squared error of both approaches is similar, but the held-out log-likelihood from the CONVCNP is far higher (see Table 2}.
Table 2:
Mean and standard errors of log-likelihood and root mean squared error over 1000 test objects from the PLastiCC dataset.
Model
Log-likelihood
MSE
Kaggle GP (Boone 2019)
-0.335 + 0.09
0.037 + 4e-3
ConvCP (ours)
1.31 + 0.30
0.040 + 5e-3
5.3 PREDATOR-PREY MODELS: SIM2REAL
The CONVCNP model is well suited for applications where simulation data is plentiful, but real world training data is scarce (Sam2Real). The CONvCNP be trained simulation data and then be deployed with real-world training data as the context set. We consider the Lotka- model (Wilkinson] 2011). I), which is used to describe the evolution of predator-prey populations. This model has been used in the Approximate Bayesian Computation literature where the task is to infer the parameters from samples drawn from the VolterVol process (Papamakarios & Murray 2016). These methods do not simply extend to prediction problems such as interpolation or forecasting. In contrast, we train CONvCNP on synthetic data sampled from the Volterra
large amount of
"https//www.kagg: www.kaggll.coom wttggll.coo//PPAAAsTTT ASTCCC20018 Full code for Avocado, including GP models, is available https//gitthh..cm///ooooawwaaa
7

---

Published as a conference paper at ICLR 2020
model and can then condition on real-world data from the Hudson's Bay fynx-hare data set (Liggh 1968) to perform interpolation (see Figure 3/ full experimental details are given in Appendix C.4) The CONVCNP performs accurate interpolation as shown in Figure 3 We were unable to successfully train the ATTNCNP for this task. We suspect this is because the simulation data are variable length- time series, which requires models to leverage translation equivariance training time, As shown in Section 5.1 the ATTNCNP struggles to do this (see Appendix f.r complete details).
Codirel
Tagn Poida
beac medeor
rook Life
Tagacaa LVD
1,60 1.00
1803
Left and centre: two samples from the Lotka- Volterra process (sim). Right: CONv trained on simulations and applied to the Hudson's Bay lynx-hare dataset (real). Plots show means and two standard deviations.
Figure 3:
5.4 2D IMAGE COMPLETION EXPERIMENTS
To test CONVCNP beyond me-dimensioonl features, we evaluate our model on on-the-grid image completion tasks and compare to ATTNCNP. Image completion can cast as prediction of pixel intensities y (E R$ for RGB, E IR for greyscale) given a target 2D pixel location æt conditioned on an observed (context) set of pixel values Z = (In, Vr Y In the following experiments, the context set can vary but the target set contains all pixels from the image. Further experimental details are in Appendix D.I
Table 3: Log-likelitee log-likelihood from image experiments (6 runs).
Model
Params
MNIST
SVHN
CelebA32
CelebA64
ZSMM
ATTNCNP
410k
1.08 +.0.4
3.94 +0.02
3.18 +0002
-0.83 +0008
CONVCNP
113k
1.21
#0.00
3.89
#0.01
3,22
#0.02
3.66 #0.01
1.18 #0.04
CONVCNPXL
400k
1.27
10.01
3.97
10.02
3.39 =0.02
3.73 00.11
0.86 10.12
Standard benchmarks. We first evaluate the model on four common benchmarks: MNIST (LeCun al. 1998). SVHN (Netzer et al., 2011), and 32 32 and 64 x 64 CelebA (Liu et al.) 2018). Importantly, these data sets are biased towards images containing a single, well-centered object. As result, perfect translation-equiiariance might hinder the performance of the model when the test data are similarly structured. We therefore also evaluated larger CONvCNP that can learn such non-stationarity, while still sharing parameters across the input space (CONyCNPXL).
Table 3 shows that CONVCNP significantly outperforms ATTNCNP when it has large receptive field size, while being at least as good with small receptive field size. Qualitative samples for various context sets can be seen in Figure 5) Further qualitative comparisons and ablation studies can be found in Appendix D.3 and Appendix D.4 respecteivv
Generalization to multiple, non-centered objects. The data sets from the previous paragraphs were centered and contained single objects. Here we test whether CoNvCNPs trained on such data can generalize to images containing multiple, non-centered objects.
The last column of Table Jevaluates the models a zero shot images contain multiple digits at test time (Appendix D.2). CONVCNP significantly outperforms ATTNCNP on such tasks, Figure 4a shows a histogram of the image log- liglliilisooods for CON and ATTNCNP, well as qualitative results at different percentiles of the CONVCNP distribution. test set, while ATTNCNP appears to
(ZSMM) setting, where
CONvCNP is able to extrapolate to this model the bias of the training data and predict centered "mean" digit independently of the context. Interestingly, CoNvCNPXL. does not perform as well on this task. In particular, we find that, as the receptive field becomes very large, performance this task decreases. We hypothesize that this has
8

---

Published as a conference paper at ICLR 2020
to do with behavior of the model at the edges of the image. CNNs with larger receptive fields the region of input pixels that affect a particular output pixel are able to model non non-stationary behavior by looking at the distance from any pixel to the image boundary. We expand this discussion and provide further experimental evidence regarding the effects of receptive field on the 7SMM task in Appendix 1.6)
Aithough ZSMM is a contrived task, note that our field of view usually contains multiple independent objects, thereby requiring translation equivariance. As more realistic example, we took a CONVCNP model trained on CelebA and tested it on natural image of different shape which contains multiple people (Figure 4b). Even with 95% of the pixels removed, the CONVCNP was able to produce a qualitatively reasonable reconstruction. A comparison with ATTNCNP is given in Appendix D.3
Computational efficiency. Beyond the performance and generalization improvements, a key ad- vantage of the CONVCNP is its computational efficiency. The memory and time complexity of single self-attention layer grows quadratically with the number of inputs M (the number of pixels for images) but only linearly for a convolutional layer.
Empirically, with a batch size of 16 on 32 x 32 MNIST, CONVCNPXI, requires 945MB of VRAM, while ATTNCNP requires 5839 MB. For the 56 x 56 7SMM CONVCNPXI. increases its requirements to 1443 MB, while ATTNCNP could not fit onto a 32GB GPU. Ultimately, ATTNCNP had to be trained with a batch size of 6 (using 19139 MB) and we were not able to fit it for CelebA64, Recently, restricted attention has been proposed to overcome this computational issue (Parmar et al. 2018), but we leave an investigation of this and its relationship to CoNvCNPs to future work.
RELATED WORK AND DISCUSSION
We have introduced CONVCNP, a new member of the CNP family that leverages embedding sets into function space to achieve translation equivariance. The relationship to (i) the NP family, and (ii) representing functions on sets, each imply extensions and avenues for future work.
Deep sets. Two key issues in the existing theory on learning with sets (Zaheer et al. 2017 Qi et al., 2017a Wagstaff et al.) 2019) are (i) the restriction to fixed-size sets, and (ii) that the dimensionality of the embedding space must be no less than the cardinality of the embedded sets. Our work implies that by considering appropriate embeddings into a function space, both issues are alleviated. In future work, aim to further this analysis and formalize it in more general context.
Point-cloud models. Another line of related research focuses on 3D point-cloud modelling (Qi et al. 2017a b). While original work focused on permutation invariance (Qi al. 2017a; Zaheer et al. 2017). more recent work has considered translation equivariance as well (Wu et al. 2019), leading to model closely resembling CONYDEEPSETS. The key differences with our work are
the following: (i) Wu et al. (2019) implement & as an MLP with learned weights, resulting in
more flexible parameterization of the convolutional weights. (ii) /Wu al. (2019) interpret the computations as Monte Carlo approximations to an underlying continuous convolution, whereas we consider the problem of function approximation directly on sets. (iii) Wu et al. (2019) only consider the point-cloud application, whereas our derivation and modelling work considers general sets.
Correlated samples and consistency under marginalization. the predictive distribution of CON- VCNP (Equation (2)). predicted y/s are conditionally independent given the context set. Consequently, samples from the predictive distribution lack correlations and appear noisy. One solution is to instead define the predictive distribution in an autoregressive way, like e.g. PixelCNN+ (Salimans et al.) 2017). Although samples are now correlated, the quality of the samples depends on the order in which the points sampled. Moreover, the predicted ys are then not consistent under marginalization (Garnelo et al.) 20188; Kim et al., 2019). Consistency under marginalization is more generally an issue for neural autoregressive models (Salimans et al. 2017 Parmar et al., 2018), although consistent variants have been devised (Louizos el al.) 2019). To overcome the consistency issue for CONvCNP, exchangeable neural process models (e.g. Korshunova et al. 2018 Louizos al. 2019) may provide an interesting avenue. Another way to introduce dependencies between ys is to employ latent variables as is done in neural processes (Garnelo et al.) 2018b). However, such an approach only achieves conditional consistency: given context set, the predicted ys will be dependent and consistent under marginalization, but this does not lead to a consistent joint model that also includes the context set itself.

---

Published as a conference paper at ICLR 2020
ConveNP AttnCNP
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
1.2
Test togs Rellood
Context
2
(
ConVCNN
2
1
2
ACtICNP
8
C
1%
10%
30% 50%
100%
(b) Qualitative evaluation of a CON- VCNPXL trained on the unscaled CelebA (218 x 178) and tested on Ellen's Oscar unscaled (337 599) selfie (DeGeneres) (0144) with 5% of the pixels as context (top).
(a) Log- likelihood and qualitative results on ZSMM. The top row shows the log-likelihood distribution for both models. The images below correspond to the context points (top), CONyCNP larget predictions (middle), and ATTNCNP target predictions (bottom). Each column corresponds to given percentile of the CONvCNP distribution.
Figure 4: Zero shot generalization tasks that require translation equivariance.
MNIST
5
C
S
S
5
5
5
S
3
O
0
3
0
116
1
10
116
116
116
116
11
115
CelenA 37x32
CelebA 64x64
Figure 5: Qualitative evaluation of the CONVCNP(XL). For each dataset, an image is randomly sampled, the first row shows the given context points while the second is the mean of the estimated conditional distribution. From left to right the first seven columns correspond to a context set with 1 1%, 5%, 10%, 20%, 30%, 50%, 100% randomly sampled context points. In the last two columns, the context sets respectively contain all the pixels the left and top half of the image. CONvCNPXI- shown for all datasets besides ZSMM, for which we show the fully translation equivariant CONVCNP.
10

---

Published as a conference paper at ICLR 2020
ACKNOWLEDGEMENTS
We would like thank Mark Rowland for help with checking the proofs, and David R. Burt, Will Tebbutt, Robert Pinsler, and Cozmin Ududec for helpful comments on the manuscript. Andrew K. Foong is supported by Trinity Hall Research Studentship and the George and Lilian Schiff Foundation. Richard E. Turner is supported by Google, Amazon, ARM, Improbable and EPSRC grants EP/M0229571 and E0//7//077//
REFERENCES
Tarek Allam Jr, Anita Bahmanyar, Rahul Biswas, Mi Dai, Lluís Galbany, Renée Hložek, Emille EO Ishida, Saurabh W Jha, David O Jones, Richard Kessler, et al. The photometric Isst astronomical time-series classification challenge (plastice): Data set. arXiv preprint 11f0:v1811000000 2018.
Nachman Aronszajn. Theory of reproducing kernels. Transactions of the American mathematical society, 68(3):337-404, 1950.
Benjamin Bloem-
Bloem-Reddy Reddy and Yee Whye Teh. Probabilistic symmetry and invariant neural networks. arXiv preprint
2019.
Kyle Boone. Avocado: Photometric classification of astronomical transients with gaussian process augmentation. arXiv preprint 1907:0400.04000 2019.
Léon Bottou. Large-scale machine learning with stochastic gradient descent. In Proceedings of COMPSTAT pp. 177-186. Springer, 2010.
François Chollet. Xception: Deep learning with depthwise separable convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1251-1258, 2017.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In Maria Florina Balcan and Kilian Q Weinberger (eds.), Proceedings The 33rd International Conference Machine Learning, volume 48 of Proceedings of Machine Learning Research, pp. 2990-2999, New York, New York, USA, 20-22 Jun 2016. PMLR.
Ellen DeGeneres.
If only Bradley's arm was longer. pictwwiterr.com///unnogg.pp Mar 2014.
Best
photo
ever.
Oscars
James Dugundji et al. An extension of tietze's thcorem. Pacific Journal of Mathematics, 1(3): 353-367, 1951.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In Doina Precup and Yee Whye Tch (eds.). Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pp. 1126 1135, International Convention Centre, Sydney, Australia, 06-11 Aug 2017. PMLR. URL .ww.ang...........m...........ee wtw..pppes//.////......
html
Marta Gareel,, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Mur- ray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional neu- ral processes. In Jennifer Dy and Andreas Krause (eds.). Proceedings of the 35th Interna- tional Conference on Machine Learning, volume 80 of Proceedings of Machine Learning Re- search, pp. 1704-1713, Stockholmsmässaan, Stockholm Sweden, 10-15 Jul 2018a. PMLR. URL http://proceediin
mlr. pwwss/v80 wttpp//ev///aaeeeellll... gwrneellalagaee.. hat.eee.nllaaammmmmm.......m..m.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J Rezende, SM Eslami, and Yee Whye Teh. Neural processes. arXiv preprint I 2018b.
Daniel T Gillespie. Exact stochastic simulation of coupled chemical reactions. The journal of physical chemistry, 81 (1(25):230--236, 1977.
Jonathan Gordon, John Bronskill, Matthias Bauer, Sebastian Nowozin, and Richard Turnr. Meta- learning probabilistic inference for prediction. In International Conference on Learning Represen- fations, 2019. URL http://///ppeeeeeeeeeeeee/ornrna

---

Published as a conference paper at ICLR 2020
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Gamelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference Learning Representations, 2019. URL.https::/ppeneeeee
JC
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In In International Conference on Learning Representations (ICLR), 2015.
Hans Knutsson and C-F Westin. Normalized and differential convolution. In Proceedings of IEEE Conference on Computer Vision and Puttern Recognition, pp. 515-523. IEFE, 1993.
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 2747-2755, Stockholmsmässan, Stockholm Sweden, 10-15 Jul 2018. PMLR.
Iryna Korshunova, Yarin Gal, Joni Dambre, and Arthur Gretton. Conditional bruno: A deep recurrent process for exchangeable labelled data. In Bayesian Deep Learning NeurIPS Workshop. 2018.
Tuan Anh Le, Hyunjik Kim, Marta Garnelo, Dan Rosenbaum, Jonathan Schwarz, and Yee Whye Teh. Empirical evaluation neural process objectives. In NeurlPS workshop on Bayesian Deep Learning, 2018.
Yann LeCun, Léon Bottou, Yoshua Bengio, Patrick Haffner, al. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2333 1998.
Juho Lee, Yoonho Lee, Jungtaek Kim, Adam Kosiorek, Seungjin Choi, and Yee Whye Teh. Set transformer: A framework for attention-based permutation-ines invariint neural networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 3744-3753, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Egbert R Leigh. The ecological role of volterra's equations. Some mathematical problems in biology, 1968.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15:2018, 2018.
Christos Louizos, Xiahan Shi, Klamer Schutte, and Max Welling. The functional neural process. 2019. hRRl:tt:://aae wtwps/.rgggbbbb/bbbbbbe C8324
J.R. Munkres. Topology; First Course. Prentice-Hall Hall, 1974.
Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Bo Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. 2011.
George Papamakarios and lain Murray, Fast e-free inference of simulation models with bayesian conditional density estimation. In D. D. Lee, M. Sugiyama, U. V. Luxburg, Guyon, and R. Garnett (eds.), Advances in Neural Information Processing Systems 29, pp. 1028-1036. 2016.
Niki Parmar, Ashish Vaswani, Jakob Uszkoreit, Lukasz Kaiser, Noam Shazeer, Alexander Ku, and Dustin Tran. Image transformer. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp.
Stockholmsmässaan Stockholm Sweden, 10-15 Jul 2018. PMLR.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. Proc. Computer Vision and Pattern Recognition (CVPR), IEFF, 122),4 2017a.
12

---

Published as a conference paper at ICLR 2020
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. +ointnet++: Deep hierarchical feature learning on point sets in a metric space. I. Guyon, U. Luxburg. S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 5099-5108, 2017b.
James Requeima, Jonathan Gordon, John Bronskill, Sebastian Nowozin, and Richard E Turner. Fast and flexible mult classification using conditional neural adaptive processes. arXiv preprint 0rXXv::906.07697, 2019.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. International Conference on Medical image computing and computer- assisted intervention, pp. 234 234.241. Springer, 2015.
Tim Salimans, Andrej Karpathy, Xi Chen, and Diederik P Kingma. Pixelenn++: Improving the pixelcnn with discretized logistic mixture likelihood and other modifications. In In International Conference Learning Representations (ICLR), 2017.
Jürgen Schmidhuber. Evolutionary principles self-referential learning. PhD thesis, Technische Universität München, 1987.
Sebastian Thrun and Lorien Pratt. Learning to learn. Springer Science & Business Media, 2012.
Edward Wagstaff, Fabian Fuchs, Martin Engelcke, Ingmar Posner, and Michael A. Osborne. On the limitations of representing functions on sets. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6487-6494, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
Darren Wilkinson. Stochastic modelling for systems biology. CRC press, 2011.
Christopher Williams and Carl Edward Rasmussen. Gaussian processes for machine learning, volume 2. MIT press Cambridge, MA, 2006.
Wenxuan Wu, Zhongang Qi, and Li Fuxin. PointConv: Deep convolutional networks on 3d point clouds. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2019.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Ruslan R Salakhutdinov, and Alexander Smola. Deep sets. In Guyon, U. Luxburg, S. Bengio, H. Wallach, Fergus, S. Vishwanathan, and R. Garnett (eds.), Advances in Neural Information Processing Systems 30, pp. 3391-3401. 2017.
13

---

Published as a conference paper at ICLR 2020
A THEORETICAL RESULTS AND PROOFS
In this section, provide the proof of Theorem I Our proof strategy is follows. We first define an appropriate topology for fixed-sized sets (Appendix A.T). With this topology in place, we demonstrate that our proposed embedding into function space is homeomorphic (Lemmas 1 and 2). We then show that the embeddings of fixed-sized sets can be extended to varying-sized sets by "pasting"
the embeddings together while maintaining their homeomorphic properties (Lemma 3) Following
this, we demonstrate that the resulting embedding may be composed with a continuous mapping to
our desired target space, resulting in a continuous mapping between two metric spaces (I Lemma 4). Finally, in Appendix A.3 we combine the abovenmentioned results to prove Theorem
We begin with definitions that we will use throughout the section and then present our results. Let X' = _2d and let J C I be compact. Let & be symmetric, positive-definite kernel on X. By the Moore-Aromee MooreeAAonnzaan Theorem, there is a unique Hilbert space (H. (.x) of real-valued functions on A' for which d is reproducing kernel. This means that (i) (((,zx) E H for all æ E X and (ii) (..( 1b(, ,)) = (xx for all f E and x (reproducing property). For w: X x X - R.
X (01).... Fn) E X and X" = (aff....,, .rnn) E X", we denote
()),
(01 x[,
(XXX))=
bb a²))
( 2(,)
Definition 3 (Interpolating RKHS). Call H interpolating if it interpolates any finite number of points: for every ((x4, ())) 90 S X x y with (z,) all distinct, there is an f e H such that
f(æ1) = ;.,..., f f(x)) = yn-
For example, the RKHS induced by any strictly positive-definite kernel, e.g. the exponentiated quadratic (EQ) kernel (at x') = o² exp((= 22 | - x²11²), is interpolating: Let c = x) and consider f Ei= cov(- 2x) E H. Then f(X) = (x,XX)cc y.
x)
A.1 THE QUOTIENT SPACE A S...
Let A be Banach space. For (I1 2nn E A" and y (31..... .n) E A", let x y if a permutation of y; that is, x ~ y if and only if x = ay for some π E Sn where
yy (Mm(1): . Vn(n))
Let A be the collection of equivalence classes of ~. Denote the equivalence class of æ by [x]; for A C A", denote [A] {[a] E A}. Call the map + [x] A - An S, the canonical map. The natural topology on A / S,, is the quotient topology, in which a subset of A / S, is open if and only if its preimage under the canonical map is open in A". In what follows, we show that the quotient topology is metrizable.
On A", since all norms ditte-dimensionll vector spaces equivalent, without loss of generality consider
| - Σi= ||=11
Note that || Al is permutation invariant: || A = || I for all π E S,, On A" Sm, define d: A" / Sn SS, -> [0, oo). d(<z)) (y]) = minzes, || - myllA..
Call a set [A] C A S, bounded [d([z). (0) [x] E [A] is bounded.
Proposition 1. The function d is metric.
Proof. We first show that d is well defined on / / Sn. Assume x ~ x' and ~ yy. Then, x' = Tw and y' = nyy. Using the group properties of Sn and the permutation invariance of I|A>:
d( [<x'), [v]]) min || T
minxes,
TT.0 Tyll
min | _. m A
min || - @e-Tylla Tyllan
d([x:] (D])
14

---

Published as a conference paper at ICLR 2020
It is clear that d((a=]. (y]) d([y). [a)) and that d((x)), [b]) = O if and only if [x] [z]] To show the triangle inequality, note that
|| - my T2yllaa> - x - n. ln + || - T2 AA- An = x| - T + || T2yllaa>
using permutation invariance of || Il.... Hence, taking the minimum over T1.
d([z)) [y]) S d([x). [=]) + || - T27 lAa
so taking the minimum over T2 gives the triangle inequality for d.
Proposition 2. The canonical map A - / is continuous under the metric topology induced by d.
Proof. Follows directly from d([æc], (l) S | yll
Let A S A" be topologically closed and closed under permutations. Then [A] is topologically closed A Sn under the metric topology.
Proposition 3.
Proof. Recall that a subset [A] of a metric space is closed iff every limit point of [A] is also in [A]. Consider a sequence ([am)))== S [A] converging some [ar] / Then there permutations (Tm)m== C Sn such that Tx an + æ. Here Tn E A, because A is closed under permutations. Thus A, A is also topologically closed. We conclude that [x] E [A].
Proposition 4. Let A C A" be open. Then [A] open in A"/S, under the metric topology. In other words, the canonical map is open under the metric topology.
Proof. Let [xz] [A]. Because is open, there some ball (y) with 0 and y E A such
that x E By) C A. Then [x]] E B. (ly)), since d((x)). [y]) I || y yll < E, and we claim that
Be ((x)) C [A]. Hence [x] E ((y) C [A]. so [A] is open.
To show the claim, let [z] ( Then d(<zz v) < for some E Sn, Hence TH E Bely) C A. so mH E Therefore, [z] [T=] E [A].
Proposition 5. The quotient topology on / Sn induced by the canonical map is metrizable with the metric d.
Proof. Since the canonical map is surjective, there exists exactly one topology on A" /S, relative to which the canonical map is quotient map: the quotient topology (Munkres 1974).
Let p: A + / denote the canonical map. It remains to show that p is quotient map under the metric topology induced by d: that is, show that U c A"/ S. is open in A" /S, under the metric topology if and only if p (U) is open in A
Let t ((U) be open in An. We have that U p(p (U)), so U is open in A" S, under the metric topology by PPropssiiion 4. Conversely, U is open in A S, under the metric topology, then p (U) open A by continuity of the canonical map under the metric topology.
A.2 EMBEDDINGS OF SETS INTO AN RKHS
Whereas A previously denoted arbitrary Banach space. in this section we specialize to A X x V We denote an element in A by (x, 8) and an element in ZM = M by ((x1, //1 yi),..., ( yM }) Alternatively, denote ((a1,) V1 (1)1,,))..... (#M,, yMs)) by (X, y) where X (#I,..., xAs) X
and y = (U1,..., -.M) E Y.M We clarify that an element in Z = M is permuted as follows: for SM.
t (XX, y) ((.1,11)),-- (MM))M)) (I)) (a (x))))))((()) (x my).
15

---

Published as a conference paper at ICLR 2020
Note that permutation- -nvvriiat functions on 2M are in correspondence to functions the quotient space induced by the equivalence class of permutations, 2M/ Sm The latter is a more natural representation.
Lemma 3 states that it possible to homcomorphically embed sets into an RKHS. This result is key proving main result. Before proving Lemma 3] we provide several useful results. We begin by demonstrating that an embedding of sets of a fixed size into RKHS continuous and injective.
Lemma 1. Consider collection 2' S ZM that has multiplicity K. Set
y R K+1 bly) -
(y)) .y x²*)
and let vb be an interpolating, continuous positive-definite kernel. Define
M
blu))b( ,, ,.)) (1: S I'M ) The CHK+1
HM
(5)
where HK+1 = H x ... x H is the (K + 1) wwwprrsiionn-veeoo-v-viiii vector-valued ffnnctinnn Hilbert space constructed from the RKHS H. for which sb is a reproducing kernel and endowed with the inner product (f.9)=411 = -EE (fis g.y.. Then the embedding
EM: [2 HMs EM(((21,)1) (E (MM.)M)) Σ
is injective, hence invertible, and continuous.
Proof. First, we show that EM is injective. Suppose that
Σ Moy)) = Σ (l
Denote (Z1,..., Fas) and y (91 (.1..... yM). and denote X' and y' similarly. Taking the inner product with any f H. on both sides and using the reproducing property of sh this implies that
Σ -
ー (()))((z)) ==1
for all f E H. In partieular, since by construction
21 1.
M
M
Σ f(x)) Σ f(x))
for all f E H. Using that H interpolating, choose particular & E X U X', and let f E H be such that f f() and f(-) at all other x; and z/. Then
1= 1,
the number of such in X and the number of such क in X' the same. Since this holds for every  X is permutation of X': x= xXX) for some permutation π E SM. Plugging in the permutation, can write
M Σ (1)))(()) Σ (w))(()) xx=x-1(x)) Σ
M
M
(am)))(a (it (1+= (0)) Σ ((
Then, by a similar argument, for any particular &,
Σ p(vi) Σ H(
16

---

Published as a conference paper at ICLR 2020
Let the number of terms in each sum equal S. Since A has multiplicity K, 5 I K. By Lemma 4 from Zaheer et al. (2017), the "sum-of-power mapping' from {ye æ; = $} to the first S + 1 elements of Liæ. (V))), i.e. (Ei v...... Ev v.). is injective. Therefore,
(v.)aa=== is permutation of (Un(s) )
Note that a; ē for all above yı- Furthermore, note that also x m1() wx a for all above Va(yy We may therefore adjust the permutation л such that = Vx(()) for all i such that æ; = a whilst retaining that æ (æ'). Performing this adjustment for all it we find that y (ty') and x r(x'..
Second, we show that Eas continuous. Compute
M
M Σ
(
p()))((
3=1
K+1
Σ
(of (y) X, X (oi(y) 207 (v) X ('))(()))))))))))))))))))))))) + t (x))(X, (X))((XX'''))
which goes to zero X' y + | y) by continuity of g/.
Having established the injection, we now show that this mapping is a homeomorphism, i.e. that the inverse is continuous. This is formalized in the following lemma.
Lemma 2. Consider Lemma 1. Suppose that Zao is also topologically closed in AM and closed under permutations, and that vo also satisfies (i) v((x 2 0, (ii) (( x) = o² 0, and (iii) (b(x,, x²) - 0 as || + oo. Then Has is closed HK+1 and E continuous.
Remark 1. To define 2/2 with multiplicity one, one might be tempted to define
Z = {(x11 31), (x2, 82)) Z2 x1 + x2).
which indeed has multiplicity one. Unfortunately, Z is not closed: if [0, [ S X and [0, 2] S y, then (0,1), (1/n, 2))²² S 2/2, but ((0, 1), (1/n,2) 2)) - ((0. 1), (0, 2)) & 2g because 0 then has two observations and 2. To get around this issue, one can require an arbitrarily small, but non-zero spacing 0 between input locations:
2,2- = (((11,))) (æ2, 92)) E 22: || - - c
This construction can be generalized to higher numbers of observations and multiplicities as follows:
ZM, = (( 3 E Zar || x, 2 e for i,j e [K], π E SM
Remark 2. Before moving on to the proof of Lemma 2 we remark that Lemma 2 would directly follow Th, were bounded: then 2 compact, so Ex a continuous, invertible map between a compact space and a Hausdorff space, which means that EN must be continuous. The intuition that the result must hold for unbounded 1 is follows. Since (1(-) summation of M "bumps" (imagine the EQ kernel) of the form w æz;) placed throughout X'. If one of these bumps goes off to infinity, then the function cannot uniformly converge pointwise, which means that the function cannot converge in H (if e is sufficiently nice). Therefore, if the function does converge in H. ( must be bounded, which brings us the compact case. What makes this work is the density channel (-) 1. which forces (x, M argument is formalized in the proof of Lemma
for every f E HM. fi is
M to be well behaved. The above
Proof. Define
2, = =[-, )) n 2/1
which is compact in M as closed subset of the compact set ([=J x J) We aim show that Har is closed in K and E is continuous. To this end, consider convergent sequence
p(m) (( (1)) )x)) (n))
H++1
17

---

Published as a conference paper at ICLR 2020
VMM Claim: (x(1)) is bounded sequence. so (x (x )== S [-J, J das for J large enough. which means that (x (i) y )= S Z,
Denote (m) [=] (n)
..... 2 2M ) and (n))
(vi (n)
.....
where ž, is compact. Note that [2,] is compact in AA Sar by continuity of the canonical map.
First, we demonstrate that, assuming the claim, HM is closed. Note that by boundedness of (x(*), (())) ( is in the image of Farl(2)) [2,] + HM- By continuity of Ear [1,] and compactness of [Zy], the image of EM|[2,) is compact and therefore closed, since every compact subset of metric space is closed. Therefore, the image of EM [2,) contains the limit f. Since the image of EM [2,]] is included in H., we have that f H.M which shows that HM is closed.
Next, we prove that, assuming the claim, EM is continuous. Consider Eai[ [2]] + E ([2,))
restricted to its image. Then (FMl (|,,)) ) is continuous, because continuous bijection from compact space to a metric space is a homeomorphism. Therefore
E - (x (m), y (*)) - (EM|1z,))
- (EM (1) (X, y).
By continuity and invertibility of EM, then f(x) + EM y), so EAs (X, x,y) f by uniqueness of limits. We conclude that End (f(())- + EM (f), which means that EM is continuous.
It remains to show the claim. Let fi denote the first element of f. i.. the density channel. Using the reproducing property of ub
| (x: - -i (ax) = ((,( ), - fi}l s I1,e "l - fillH = - -ill
so f(a) + fi in H. means that it does uniformly pointwise (over æ).. Hence, can let N E N be such that n 2 N implies that (S()) (z) - fi (x) < f0² for all æ. Let R be such that (e, x (N)) < 3 / for ||a||| 2 It and all i E [M] Then, for ||>||| 2 R
(x)) Iv 2; ) o
[[1())) S (x)) + (x) f1())) < 30²
At the same time, by pointwise non non-nigativity of th, we have that
( (a = Σド ((z) (x) (m)) 2 (xx) (n) x (n) = =².
Towards contradiction, suppose that (*)) ) is unbounded. Then (z) (o(())o ym is unbounded for some E [M]. Therefore,
||ar(")|| R for some N,
(a! ( Z ( (a; (m)) - 1 (m) (a(")) f((z) )() 2 02 f02 = O0²2
which is a contradiction.
The following lemma states that we may construct an encoding for sets containing no more than M elements into function space, where the encoding is injective and every restriction to fixed set size is a homeomorphism.
Lemma 3. For every E [M), consider collection Z' C 2m that (i) has multiplicity K, (ii) is topologically closed, and (íil) closed under permulaations. Set
Φ y + + k+ o()) =
K
and let vo be an interpolating, continuous positive-definiz kernel that satisfies (i) sb( x') 2 0. (ii) (bx x)) = x² > 0, and (ii) (b x') + 0 as ||>||| - oc. Define
m
Hm
(()))((( a.)) U:) C 2 C HK+1 Σ
where HK+1 = H x ... x H is the (K
Hilbert space
constructed from the RKHS for which ab is a reproducing kernel and endowed with the inner
product (f.g)xx11 = Σk1 (fi, giyH. Denote
M [2 = U [zm and HeM U m=l
M
Hm-
m=l
18

---

Published as a conference paper at ICLR 2020
Then (Hm)m= are pairwise disjoint. follows that the embedding E
E: (2'MM] - Hgg, B([Z]) = Fm ((21) if [2] E 2m
injective, hence invertible. Denote this inverse by E- where F F (f) Hm.
Proof. Recall that Em is injective for every m E [M]. Hence, demonstrate that injective remains to show that (Hm) m are pairwise disjoint. To this end, suppose that
m'
Σ
wi))
Σ M(vl)((
for m # Then, by arguments like in the proof of Lemma
m
m'
Σ ((u)) = o
Since Φ1
1, this gives m m', which is contradiction. Finally, by repeated application of
Lemma 2 Em 1-11 is continuous for every m [M].
Lemma 4. Let 4: [2's]] - Cx(X,)) be a map from [Z'g]] Cr(X,)), the space of continuous bounded functions from X V, such that every restriction $ ([[2,]] is continuous, and let be from
Lemma 3 Then
o E-L: :HEM + Cx(x3)
continuous.
Proof. Recall that, due to Lemma
for every m E [M]. Em is continuous and has image [Z ]. By the continuity of & (1[2,,1- then + | o Em is continuous for every [M]. Since & o E' E' Hm [1.2,11 E for all m [M), we have that + E-1 H... continuous for all m [M]. Therefore, as Hm is closed Hgg as for every m E [M]. the pasting lemma (Munkres) 1974) yields that $ o E-1 continuous.
From here we let & be stationary kernel, which means that it only depends on the difference of its arguments and can be seen as a function X' + R.
A.3 PROOF OF THEOREM
With the above results in place, we are finally ready to prove our central result, Theorem 1
Theorem 1. For every m [M], consider a collection 2 S Zm that (i) has multiplicity K, (ii) is topologically closed, (iii) is closed under permutations, and (iv) is closed under translations. Set
K1 σ(v) (1".v' .y*)
and let vi be an interpolating, continuous dositive-definite kernel that satisfies (i) (xx,, x') 2 (ii) vb (x, v) = o² > 0, and (ii) (<x, x') + 0 as ||>||| - oc. Define
(v.))b( (a)) (o) S 2/m C CHK+1
Hm
where HK+1 H ... H is the (K + j.-ddmmenionnallecoo-vaaalie vuntti-tntunnntio Hilbert space constructed from the RKHS H for which và is reproducing kernel and endowed with the inner product (J.g)
*++1 (Jis g.H Denote
M
M 2 and HaM = U Hm.
E'z = U
m=l
m=l
19

---

Published as a conference paper at ICLR 2020
Then function @: Z'SM Cx(x))) satisfies (i) continuity of the restriction + [M)] (ii) permutation invariance (Property I,. and (iii) translation equivariance Property 2) if and only it has representation the form
2. for every
$ (2) - p (E(Z)) (((21,))),.. (m Umm) = Σ ((s vo( xi))
where p: HgM ()) is continuous and translation equivariant.
Proof of sufficiency. To begin with, note that permutation invariance (Property and translation equivariance (Property 2) for are well defined, because 2'<M is closed under permutations and translations by assumption. First, & is permutation invariant, because addition is commutative and associative. Second, that is translation equivariant (Property follows from a direct verification and that is also translation equivariant:
Φ 2) p
blu)) (0:; + 7))
Σ (()))((() - - xz;)
p
M
Σ
o(v)) (():)((-a) (00(--21)) xzi) (. (--T)
((2)(.- T)
(Z)
Proof of necessity. Our proof follows the strategy used by Zaheer et al. (2017): Wagstaff et al. (2019) To begin with, since is permutation invariant (Property 1), we may define
M U - Co(,D)) ((Z) = (((1)) mm
+
for which we verify that every restriction
+ $1122,1) is continuous. By invertibility of E from Lemma 3
have [Z] E-1 (E((Z))) Therefore,
M
= = +5 = OF( o F-1)
(l ) ax:)
Define HgM - C& by p= $ E-1, First, is continuous by Lemma 4 translation equivariant, because 1/r is stationary. Also, by assumption + translation equivariant (Property 2). Thus, their composition p is also translation equivariant.
Second, E-1
Remark 3. The function p: Hea (x,))) may be continuously extended the entirety of KK+1 using generalisation of the Tietze Extension Theorem by Dugundji et al. (191). There variants of Dugundji's Theorem that also preserve translation equivariance.
20

---

Published as a conference paper at ICLR 2020
B BASELINE NEURAL PROCESS MODELS
In both our 1d and image experiments, main comparison is to conditional neural process models. In particular, we compare to a vanilla CNP (Id only; Garnelo et al. (2018a)) and an ATTNCNP (Kim et al. 2019). Our architectures largely follow the details given in the relevant publications.
CNP baseline. Our baseline CNP follows the implementation provided by the authors x is a 3-layer MLP with 128 hidden units in each layer, and RELU -nnliinarssiies. The encoder embeds every context point into a representation, and the representations are then averaged across each context set. Target inputs then concatenated with the latent representations, and passed to the decoder. The decoder follows the same architecture, outputting mean and standard deviation channels for each input.
The encoder
Attentive CNP baseline. The ATTNCNP we use corresponds to the deterministic path of the model described by Kim et al. (2019) for image experiments. Namely, encoder first embeds each context point to a latent representation (x(e), y(() + ray E IR 128 For the image experiments, this is achieved using 2-hidden layer MLP of hidden dimensions 128. For the 1d experiments, we use the same encoder as the CNP above. Every context point then goes through two stacked self-attention layers. Each self-attention layer is implemented with 8-headed attention, a skip connection, and two layer normalizations (as described in Parmar et al. (2018), modulo the dropout layer). To predict values at each target point embed (()) rf and x(e) H rz using the same single hidden layer MLP of dimensions 128. A target representation rhy is then estimated by applying cross-attention (using 8-headed attention described above) with keys K (r/)) ) values V = |C 1- and query q = r((') Given the estimated target representation f the conditional predictive posterior is given by Gaussian pdf with diagonal covariance parametrised by
(c)
(1))), (0) = decoder(rzy) ( where (6) (1) E IR3 and decoder is 4 hidden layer MLP with 64 hidden units per layer for the images, and the same decoder as the CNP for the 1d experiments.
Following Le al. (2018), we enforce we set minimum standard deviation a (t) avoid infinite log-likelihoon licelhoooss by using the following post-processed standard deviation:
min [0.1; 0.1: 0.1] to
(t) post
0.1o!!! (t) min + (1 -0.1) log(1 + cxp(o
(8)
21

---

Published as a conference paper at ICLR 2020
C I-DIMENSIONAL EXPERIMENTS
In this section, give details regarding experiments for the 1d data. We begin by detailing model architectures, and then provide details for the data generating processes and training procedures. The density at which evaluate the grid differs from experiment to experiment, and the values given in the relevant subsections. all experiments, the weights are optimized using Adam (Kingma & Ba 2015) and weight decay of 10 10 is applied to all model parameters. The learning rates specified in the following subsections.
C.1 CNN ARCHITECTURES
Throughout the experiments (Sections 5.1 to 5.3 utilizes a smaller architecture), and CONvCNPXL (with a larger architecture). For all architectures, the input kernel s was EQ (exponentiated quadratic) kernel with learnable length scale parameter, as detailed in [Section 4, as was the kernel for the final output layer dhp. When dividing by the density channel, we add & 10-a to avoid numerical issues. The length scales for the EQ kernels are initialized to twice the spacing 1/71/d between the discretization points (t:) where y is the density of these points and d is the dimensionality of the input space X.
we consider two models: CONvCNP (which
Moreover, we emphasize that the size of the receptive field product of the width of the CNN filters and the spacing between the discretization points. Consequently, for fixed width kernel of the CNN, as the number of discretization points increases, the receptive field size decreases. One potential improvement that was not employed in our experiments, is the use of depthwise-separabee convolutions (Chollet 2017). These dramatically reduce the number of parameters in a convolutional layer, and can be used increase the CNN filter widths, thus allowing one to increase the number of discretization points without reducing the receptive field.
The architectures for CONvCNP and CONVCNPXL described below.
CONvCNP. For the 1d experiments, we use a simple, 4-layer convolutional architecture, with RELU nonlinearities. The kernel size of the convolutional layers was chosen to be 5, and all employed a stride of length and zero padding of 2 units. The number of channels per layer was set to [16,32, 16, 2). where the final channels where then processed by the final, EQ-based layer of p as mean and standard deviation channels. We employ SOFTPIJS nonlinearity on the standard deviation channel enforce positivity. This model has 6,537 parameters.
CONVCNPXL Our large architecture takes inspiration from UNet (Ronneberger et al. 2015). We employ 12-layer architecture with skip connections. The number of channels doubled every layer for the first 6 layers, and halved every layer for the final 6 layers. We use concatenation for the skip connections. The following describes which layers are concatenated, where L; + [L,, L&] means that the input to layer is the concatenation of the activations of layers and k:
La + [Ts, Lz],
Lg + [1,, Ls).
L10 + [L,, Lyj:
Lu [L2 L10):
L2 [L, Ln1]
Like for the smaller architecture, we use RELU nonlinearities, kernels of size 5, stride 1, and zero padding for two units on all layers.
C.2 SYNTHETIC ID EXPERIMENTAL DETAILS AND ADDITIONAL RESULTS
The kernels used for the Gaussian Processes which generate the data in this experiment are defined as follows:
EQ:
k(xx e 1
22

---

Published as a conference paper at ICLR 2020
CONVCNP
True function Ground Truth
Model Trainkig
ALINCNY
True function
Ground
function
Trannlo FYOOR
Figure 6: Example functions learned by the (top) CONVCNP, (center) ATTNCNP, and (bottom) CNP when trained on an EQ kernel (with length scale parameter 1). "True function" refers to the sample from the GP prior from which the context and target sets were sub-sampled. "Ground Truth GP" refers to the GP posterior distribution when using the exact kernel and performing posterior inference based on the context set. The left column shows the predictive posterior of the models when data is presented in same range training. The centre column shows the model predicting outside the training data range when no data is observed there. The right-most column shows the model predictive posteriors when presented with data outside the training data range.
weakly periodic:
-
with fi = cos((8r) and f2(x) sin(87r), and
and
Matern-g:
k(x,'') = (1 + 4v5d + e)
with d = 4/x-21 z'1.
During the training procedure, the number of context points and target points for training batch each selected randomly from uniform distribution over the integers between 3 and 50. This number of context and target points are randomly sampled from a function sampled from the process (a Gaussian process with one of the above kernels or the sawtooth process), where input locations are uniformly sampled from the interval [-2, 2]. All models this experiment were trained for 200 epochs using 256 batches per epoch of batch size 16. We discretize E(Z) by evaluating 64 points per unit this setting. We use learning rate 3e-4 4 for all models, except for CONv CONvCNPXL on the sawtooth data, where models).
learning rate of 1e-3 (this learning rate too large for the other
The random sawtooth samples generated from the following function:
A
A
80
winssnnnnn (2nkft)))
Usawtooth (t)
2
π
where A is the amplitude, f is the frequency, and is "time". Throughout training, we fix the amplitude to one. We truncate the series at an integer K. At every iteration, we sample frequency uniformly in 53, K in [10, 20), and random shift in [-5, 5]. As the task is much harder, sample context and target set sizes over [3, 100). Here the CNP and ATTNCNP employ learning rates of 10-3 All other hyperparameters remain unchanged.
23

---

Published as a conference paper at ICLR 2020
Thue Bunction
CONVCNP
Thiiiing
AUTNCN
Ground Truth
Ground Truth
Thunlie
Figure Example functions learned by the (top) CONYCNP, (center) ATTNCNP, and (bottom) CNP when trained on a Matérn-5/2 kernel (with length scale parameter 0.25). "True function" refers to the sample from the GP prior from which the context and target sets were sub-sampled. "Ground Truth GP" refers to the GP posterior distribution when using the exact kernel and performing posterior inference based on the context set. The left column shows the predictive posterior of the models when data is presented in same range training. The centre column shows the model predicting outside the training data range when no data is observed there. The fight-most column shows the model predictive posteriors when presented with data outside the training data range.
Variable
m
time
5.94 x x0d
8.74 x 10
1sstu
1.26
1.63 x102
1sstg
-0.13
3,84
x102
1sstr
3.73
3,41 x10
1ssti
5.53
2.85
x102
1sstz
6.43
2.69
x102
issty
6.27
2.93
x102
Table 4:
Values used normalise the data in the PLASTICC experiments.
We include additional figures showing the performance of CONVCNPs, ATTNCNPs and CNPs on GP and sawtooth function regression tasks in Figures 6
C.3 PLASTICC EXPERIMENTAL DETAILS
The CONVCNP was trained for 200 epochs using 1024 batches of batch size 4 per epoch. For training and testing, the number of context points for batch are each selected randomly from uniform distribution over the integers between I and the number of points available in the series (usually between 10-30 per bandwidth). The remaining points in the series are used as the target set. For testing, a batch size of was used and statistics were computed over 1000 evaluations. We compare CONVCNE to the GP models used in (Boone) 2019) using the implementation in inhttps:: : github. ccm www.:boww..ecoecccooooooooooooca wwwp://oooooeooeoooooeooo avocado The data used for training and testing is normalized according to f(v) = (u m)/s with the values in Table 4. These values are estimated from a batch sampled from the training data. To remove outliers in the GP results, log ligeliieehooo values less than - 10 removed from the evaluation. These same datapoints were removed from the CONvCNP results as well.
24

---

Published as a conference paper at ICLR 2020
CONVCNP
Nuncion
Model
ALTOCN
furction
Model
"rariing
UNP
"raining
Figure 8: Example functions learned by the (top) CONVCNP, (center) ATTNCNP, and (bottom) CNP when trained on random sawtooth sample. The left column shows the predictive posterior of the models when data is presented in the same range as training. The centre column shows the model predicting outside the training data range when no data is observed there. The right-most column shows the model predictive posteriors when presented with data outside the training data range.
For this dataset, we only used the CONvCNPXL, we found the CONVCNP to underfit. The learning rate was set to 10-3 and we discretize E(Z) by evaluating 256 points per unit.
C.4 PREDATOR-PRET EXPERIMENTAL DETAILS
We describe the way simulated training data for the experiment in Section 5.3 was generated from the model. The description is borrowed from (Wilkinson) [011].
Let X the number of predators and Y the number of prey at any point in our simulation. According the model, of the following four events
A: A single predator is born according rate 0, XY, increasing X by one.
B: A single predator dies according to rate 02 decreasing X by one.
C: A single prey is born according to rate 02Y increasing Y by one.
D: A single prey dies (is eaten) according to rate 0,XY, decreasing Y by one.
The parameter values 0,, 02, 03. and 04, as well as the initial values of X and Y govern the behavior of the simulation. We choose 0, 0.01, 02 0.5, 0g 1, and 04 0.01, which are also used in (Papamakarios & Murray, 2016) and generate reasonable time series. Note that these are likely not the parameter values that would be estimated from the Hudson's Bay lynx-hare data set (Leigh) 1968), but they are used because they yield reasonably oscillating time series. Obtaining oscillating time series from the simulation is sensitive the choice of parameters and many parametrizations result in populations that simply die out.
Time series are simulated using Gillespie's algorithm (Gillespie 1977;:
1. Draw the time the next event from exponential distribution with rate equal to the total rate 0, XY + 02X + 03Y + XY..
2. Select one of the above events A, B, C, or at random with probability proportional to its rate.
3. Adjust the appropriate population according to the selected event, and go 1.
25

---

Published as a conference paper at ICLR 2020
The simulations using these parameter settings can yield maximum population of approximately 300 while the context set in the lynx lynx-hare data set has an approximate maximum population of about 80 so we scaled our simulation population by a factor of 2/7. We also remove time series which are longer than 100 units of time, which have more than 10000 events, or where one of the populations is entirely zero. The number of context points for training batch are each selected randomly from uniform distribution between 3 and 80, and the number of target points is 150 n. These target and context points are then sampled from the simulated series. The Hudson's Bay lynx-hare data has time values that range from 1845 to 1935. However, the values supplied to the model range from 0 to 90 to remain consistent with the simulated data.
For evaluation, an interval of 18 points is removed from the the Hudson's 's Bay lynx hare data set to act as a target set, while the remaining points act as the context set. This construction highlights the model's interpolation as well as its uncertainty in the presence of missing data.
Models in this setting were trained for 200 epochs with 256 batches per epoch, each batch containing 50 tasks. For this data set, we only used the CONVCNP, as we found the CONvCNPXL to overfit. The learning rate was set to 10-$, -3 and we discretize (Z)) by evaluating 100 points per unit.
We attempted to train an ATTNCNP for comparison, but due to the nature of the synthetic data generation, many of the training series end before 90 time units, the length of the Hudson's Bay lynx-hare series. Effectively, this means that the ATTNCNP was asked to predict outside of its training interval, a task that struggles with, as shown in Section 5.1] The plots in Figure show that the ATTNCNP is able to learn the first part of the time series but is unable model data outside of the first 20 or so time units. Perhaps with more capacity and training epochs the ATTNCNP training would be more successful. Note from Figure that our model does better on the synthetic data than on the real data. This could be due to the parameters of the Lotka-Voltera model used being a poor estimate for the real data.
1.0
Predetor
Target:
120
Madel: Prodator
Contoxt: Predator
Target: Predator
100
|oddll Pray
Context: Pray
Tarpet:
1.0
Muddet: Pray
Contect: Pray
Tanget: Prey
60
40
20
-20
26
60
B0
300
20
40
60
100
Figure 9: ATTNCNP performance on two samples from the Lotka-Voltent process (sim).
26

---

Published as a conference paper at ICLR 2020
D IMAGE EXPERIMENTAL DETAILS AND ADDITIONAL RESULTS
D.1 EXPERIMENTAL DETAILS
Training details. all experiments, we sample the number of context points uniformly from U( 100 *yoo) ), and the number of target points is set to notal-- The context and target points are sampled randomly from each of the 16 images per batch. The weights are optimised using Adam (Kingma & Ba 2015) with learning rate 5 x 10 We use maximum of 100 epochs, with early stopping of 13 epochs patience. All pixel values are divided by 255 to rescale them to the [o, 1] range. In the following discussion, we assume that images are RGB, but very similar models be used for greyscale images or other gridded inputs (e.g. 1d time series sampled at uniform intervals).
Proposed convolutional CNP. Unlike ATTNCNP and off-the-grid CONVCNP, on-the-grid CON- VCNP takes advantage of the gridded structure. Namely, the target and context points can be specified terms of the image, a context mask Mc. and target mask M, instead of sets of input-value pairs. Although this is an equivalent formulation, it makes it more natural and simpler to implement in standard deep learning libraries. the following, we dissect the architecture and algorithmic steps succinctly summarized in [Section 4) Note that all the convolutional layers are actually depthwise separable (Chollet 2017); this enables large kernel size (i.e. receptive fields) while being parameter and computationally efficient.
1. Let I denote the image. Select all context points signal == Mc @ I and append a density channel density == Mg. which intuitively says that "there is point at this position": [signal, density] T_ Each pixel value will now have 4 channels: 3 RGB channels and density channel M Note that the mask will set the pixel value to 0 location where the density channel is O, indicating there are no points at this position (a missing value).
2. Apply convolution the density channel density' CONV (clensity) and normalized convolution to the signal signal' == CONV o /signal)/an /lensityy The normalized convolution makes sure that the output mostly depends on the scale of the signal rather than the number of observed points. The output channel size is 128 dimensional. The kernel size of CONV, V depends on the image shape and model used (Table 5). We also enforce element wise positivity of the trainable filter by taking the absolute value of the kernel weights O before applying the convolution. As discussed in AAppendix D.4, the normalization and positivity constraints do not empirically lead to improvements for on-the-grid data. Note that in this setting, ((Z) is [signal', density']
3. Now we describe the on-the-grid version of p(-). which decompose into two stages. In the first stage, we apply a CNN to (signal', density" This CNN is composed of residual blocks (He et al. 2016), each consisting of 1 or 2 Table 5) convolutional layers with ReLU activations and no batch normalization. The number of output channels in each layer 128. The kernel size the same across the whole network, but depends on the image shape and model used (Table 5).
4. In the second stage of p(-), we apply shared pointwise MLP: 122
R2C (we the same architecture as used for the ATTNCNP decoder) to the output of the first stage at each pixel location in the target set. Here C' denotes the number of channels the image. The first C' outputs of the MLP are treated as the means of Gaussian predictive distribution, and the last C' outputs are treated the standard deviations. These then pass through the positivity-enforrii function shown in Equation (8)
Table 5: CNN architecture for the image experiments.
Model
Input Shape
CONV8 Kernel Size
CNN Kernel Size
CNN Num. Res. Blocks
Conv. Layers per Block
CONVCNP
50 pixels
9
5
4
50 pixels
7
3
4
CONVCNP XL.
any
9
11
6
27

---

Published as a conference paper at ICLR 2020
D.2 ZERO SHOT MULTI MNIST (ZSMM) DATA
3
6
7
3
4
3
/
8
(a) Train
(b) Test
Figure 10: Samples from our generated Zero Shot Multi MNIST (ZSMM) data set.
In the real world, it is very common have multiple objects in our field of view which do not interact with each other. Yet, many image data sets in machine learning contain only a single, well-centered object. To evaluate the translation equivariance and generalization capabilities of our model, introduce the zero-shot multi-MNIST setting.
The training set contains all 60000 28 28 MNIST training digits centered black 56 x 56 background. (Figure 10a). For the test set, we randomly sample with replacement 10000 pairs of digits from the MNIST test set, place them black 56 x 56 background, and translate the digits in such a way that the digits can be arbitrarily close but cannot overlap (Figure 10b). Importantly, the scale of the digits and the image size the same during training and testing.
D.3 ATTNCNP AND CONVCNP QUALITATIVE COMPARISON
Figure 11 shows the test log- ligeliiel distributions of ATTNCNP and CONvCNP model well as some qualitative comparisons between the two.
Although most mean predictions of both models look relatively similar for SVHN and CelebA32, the real advantage of CONvCNP becomes apparent when testing the generalization capacity of both models. FFigure 12 shows CONvCNP and ATTNCNP trained CelebA32 and tested downscaled version of Ellen's 's famous Oscar selfie. We see that CONVCNP generalizes better in this setting.
D.4 ABLATION STUDY: FIRST LAYER
Table Log-likel liceliidl from image ablation experiments (6 runs).
Model
MNIST
SVHN
CelebA32
CelebA64
7SMM
CONVCNP
1.19 10.01
3.89 #0.01
10.02
3.64 #0.01
10.00
no density
1.15 +0001
3.88 +0.01
+0.02
3.62 +0.01
1.13 +0.08
no norm.
1.19 +0001
3.16
3.62 +0.01
1.20
abs.
1.15 +0.02
3.83 40.02
10.03
3.56
10.01
no abs. norm.
1.19 10.01
3.86
10.03
3.62 10.01
1.20 10.01
EQ
1.18 +0.00
3.89 =0.01
3.18 #0.02
3.63 #0.01
1.21 +0.00
'The reconstruction looks worse Figure 4b despite the larger context set, because the test image has been downscaled and the models are trained on a low resolution CelebA32. These constraints come from ATTNCNP's large memory footprint.
28

---

Published as a conference paper at ICLR 2020
AttrnNN
Deno
0.6
1.0
2.25
2.50
2.75
3.00
3.25
3.50
Loge
Log LiKal
Contect
Contrr
3
9
/
COMCCPPX
3
67
8
67
5
9
g
%
50%
(a) MNIST
(b) SVHN
DONWCMPXL
AIDICNP
Danity
CONMONPPL
2.0
3,5
Test Log Likel hood
2.00
Contort
Test Log hoto
Contoct
ConCANPL
CorVEIPILL
20%
90%
100%
19k
10%
20%
30%
50%
100%
(d) CelebA 64 x 64
(c) CelebA 32 x 32
Figure 11: Log-likelihood and qualitative comparisons between ATTNCNP and CONVCNP on four standard benchmarks. The top row shows the log-likelihood distribution for both models. The images below correspond to the context points (top), CONVCNP target predictions (middle), and ATTNCNP target predictions (bottom). Each column corresponds to given percentile of the CONVCNP distribution. ATTNCNP could not be trained on CelebA64 due its memory inefficiency.
Figure 12: Qualitative evaluation of CONvCNP (center) and ATTNCNP (right) trained on CelebA32 and tested downscaled version (146 259) of Ellen's 's Oscar selfie (DeGeneres) 2014) with 20% of the pixels as context (left).
To understand the importance of the different components of the first layer, we performed an ablation study by removing the density normalization (CONYCNP no norm.), removing the density channel (CoNvCNP dens.), removing the positivity constraints (CoNvCN)
abs.), removing the positivity constraints and the normalization (CONyCNP no abs. norm.), and replacing the fully trainable first layer by EQ kernel similar the continuous case (CoNvCNP EQ).
Table shows the following: (i) Appending a density channel helps. (ii) Enforcing the positivity constraint is only important when using normalized convolution. (iii) Using less expressive EQ filter does not significantly decrease performance, suggesting that the model might be learning similar filters Appeeddix 1).5).
29

---

Published as a conference paper at ICLR 2020
D.5 QUALITATIVE ANALYSIS OF THE FIRST FILTER
MNIST
SVHN
CelebA 32x32
CelebA 64x64
Zero Shot Multi-MNIST
ConvCNPXL
ConvCNP
ConvCNP EQ
0.1
Figure 13: First filter learned by CONvCNPXL, CONvCNP, and CONvCNP EQ for all our datasets. In the case of RGB images, the plotted filters are for the first channel (red). Note that not filters are of the same size.
As discussed in Appendix D.4 using less expressive EQ filter does significantly decrease per- formance. Figure 13 shows that this happens because the fully trainable kernel learns to approximate the EQ filter.
D.6 EFFECT OF RECEPTIVE FIELD ON TRANSLATION EQUIVARIANCE
As seen in Table 3) CONvCNPXL with large receptive field performs significantly worse on the 7SMM task than CONVCNP, which has smaller receptive field. Figure 14|shows more detailed comparison of the models, and suggests that CONVCNPXI learns to model non-stationary behaviour, namely that digits in the training set are centred. We hypothesize that this issue stems from the the treatment of the image boundaries. Indeed, if the receptive field is large enough and the padding values are significantly different than the inputs to each convolutional layer, the model can learn position-dependen behaviour by "looking" the distance from the padded boundaries.
For ZSMM, Figure 15 suggests that "circular" padding, where the padding is implied by tiling the the size of the receptive
image. helps prevent the model from learning statiinntnassssss field becomes larger. We hypothesize that this is due to the fact that "circularly" padded values are harder distinguish from actual values than zeros. We have not tested the effect of padding other datasets, and note that "circular" padding could result in other issues.
30

---

Published as a conference paper at ICLR 2020
Density
5
ConVCNP
ConvCNPXL
-0.50
-0.25
0.00
0.25
0.50
0.75
1.00
1.25
Test Log Likellhood
Context
2
ConvCNP
2 7
2
ConvCNPXL
1%
10%
30%
50%
100%
Figure 14: Log-likelihood and qualitative results on ZSMM. The top row shows the log- log-likelihood distribution for both models. The images below correspond the context points (top), CONVCNP target predictions (middle), and CONvCNPXL target predictions (bottom). Each column corresponds to given percentile of the CONVCNP distribution.
1.2
0.5
Tox Unciirdd
0.53
-0.23
Redding
-0.50
20
30
40
M0
10
Size of Receptive Field
Figure 15: Effect of the receptive field size on ZSMM's log-likelihood. The line plot shows the mean and standard deviation over runs. The blue curve corresponds model with zero padding, while the orange one corresponds to "circular" padding.
31

---

Published as a conference paper at ICLR 2020
E AUTHOR CONTRIBUTIONS
Richard, Jonathan, and Wessel formulated the project. Richard and Jonathan helped coordinate the team and were closely involved with all aspects the project. Andrew realised that a density channel was necessary in the model. Yann proposed using normalized convolutions and showed that they led to improvements. All authors contributed to the writing and editing of the paper. Wessel and Jonathan developed the theory. Andrew verified the proof and suggested improvements. James also suggested improvements. Wessel wrote the majority of Appendix A with assistance from Andrew and Jonathan. Andrew and Jonathan performed the initial experiments and James refined these experiments and produced the final versions for the paper. James wrote this section of the paper. James led the experiments on complex time-scries and wrote this section of the paper. Andrew worked on the first on-the- grid experiments. Yann redesigned and reimplemented the on-the-gric computational framework and performed all the image experiments shown in the paper. Yann wrote this section of the paper.
simple time-series. Wessel, Jonathan,
32


Reviews:
[{"review_id": "r1erPYdXFH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper introduces ConvCNP, a new member of the neural process(NP) family that models translational equivariance in the data, which uses convolutions and stationary kernels to aggregate the context data into a functional representation.\n\nThis problem is well-motivated as there are various domains where such an inductive bias is desirable, such as spatio-temporal data and images, and will help especially with predictions for out-of-distribution tasks. This inductive bias was never built into NPs, and it remained unanswered whether the NP can learn such a behaviour. This paper shows that the answer is negative and that one needs to make modifications to create such inductive bias.\n\nThe architecture of the ConvCNP is motivated by theory that completely characterises the set of translation equivariant functions Phi that maps sets of (x,y) pairs to bounded continuous functions that map x to y (disclaimer: I haven’t read through the proof in the appendix, so will not make any claims on its correctness). Theorem 1 defines the set of such functions using rho, phi and psi, and the choices for each on on-the-grid data and off-the-grid data are listed in Section 4. There are ablation studies in Appendix D.4 that justify the choices.\n\nOverall the paper is very well-written and clear for the most part, with helpful pseudo-code and well-laid out quantitative + qualitative results, and a very detailed appendix that allows replicating the setup. The evaluation is extensive, and the results are significant.\n    - The results on 1D synthetic data show a noticeable improvement of the ConvCNP compared to the AttnCNP, with improved interpolation as well as accurate extrapolation for the weakly periodic function. I do think however that a more competitive baseline for AttnCNP would have been to parameterise the logits of the attention weights as a periodic function with learnable length scale (e.g. stationary periodic kernel), since this is another way of building in periodicity into the model. Arguably this is more explicit and restrictive than the translational equivariance built into ConvCNP, but would have made for a more interesting comparison.\n    - Having said that, I like how the evaluation was performed on a variety of stochastic processes - previous literature only used GP + EQ kernel, but here more challenging non-smooth functions such as GP + Matern kernels and sawtooth functions are explored - and it’s very convincing to see the outstanding performance of ConvCNPs here.\n    - It’s also nice to see results on regression tasks on real data (sections 5.2, 5.3), which was never explored in the NP literature as far as I know. 5.2 shows that ConvCNPs can be competitive against other methods that model stochastic processes, and 5.3 shows an instance of where ConvCNPs do a reasonable job whereas (Attn)CNP fails.\n    - The results on images is also extensive, covering 6 different datasets (including the 2 zero shot tasks), and show convincing qualitative and quantitative results. The zero shot tasks are nice examples that explicitly show the consequences of not being able to model translation equivariance in more realistic images composed of multiple objects/faces.\n\nI have several comments/questions regarding the disccusion & related work section:\n    - One link that might be worth pointing out regarding functional representation of context is that ANP (or AttenCNP) can also be seen as giving a functional representations of the context; the ANP computes a target-specific representation of the context, which can be seen as a function of the target inputs.\n    - I think it’s incorrect to say that latent-variable extensions enforce consistency. Even with the latent variable, if the encoder is seen as part of the model, then the NP isn’t consistent (pointed out in the last paragraph of section 2.1 in the ANP paper). So there still are issues regarding AR sampling. There does however seem to exist variants of NPs that satisfy consistency e.g. https://arxiv.org/abs/1906.08324\n    - What is preventing the incorporation of a latent variable in the ConvCNP? Is this just something that can be easily done but you haven’t tried, or do you see any non-trivial issues that arise when doing so e.g. maintaining translation equivariance?\n\nOther minor comments:\n    - Are there any guidelines on choice of filter size of CNN in the image case? E.g. have you chosen the filter size of ConvCNP such that the receptive field is smaller than the image, whereas it’s bigger for ConvCNPXL? It’s not clear why having a bigger receptive field allows to capture non-stationarity, and it would be helpful to expand on that, perhaps in the appendix.\n    - Also it’d help for the sake of clarity to explain why AttnCNP uses significantly more memory than ConvCNP, i.e. because memory for self-attention is O(N^2) where N=HW is the number of inputs, whereas for convolutions it’s O(HW).\n    - I think it’d also help to state explicitly in the body that AttnCNP is ANP without the latent path when it is introduced.\n    - typos: first paragraph of Section 2: Z_M <- Z_m (twice), finitely <- infinitely, Appendix D.1: separabe <- separable\n\nOverall, I think this is a very strong submission and I vote for its acceptance.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "SkeY6z_6tH", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "-- Summary \n\nThis paper considers the problem of developing neural processes which\nare translation-equivariant. The authors derive a necessary and sufficient\nfunctional form that the neural process \\Phi function must exhibit\nin order to be permutation invariant, continuous and translation\nequivariant.\n\nUsing the derived functional form, the authors construct a\ntranslation-equivariant neural process, the convolutional conditional\nneural process.\n\nResults in several experimental settings are given: 1d synthetic\nexperiments, an astronomical time-series modelling experiment, a\nsim2real experiment, and several image completion experiments.  All\nthe experiments show performance improvements over the AttnCNP, the\nmain baseline tested against. In the astronomy setting the authors\ntest against the winning kaggle entry, against which they get better\nlog likelihood. The authors give several qualitative experiments,\nincluding image completion tasks from a small number of pixels.\n\nProofs of all the theorems and full details of all the experiments\nare given in the appendix, along with ablations of the model.\n\n\n-- Review\n\nOverall I found this paper very impressive. It is clear how the theoretical\nresults motivate the choice of architecture. The fact that Theorem 1\ncompletely characterises the design of all translation-equivariant\nneural processes is a remarkable result which precisely specifies the\ndegrees of freedom available when constructing a convolutional NP.\n\nThe implementation gives state of the art results against\nthe AttnCNP while using fewer parameters on a variety of tasks. The image\ncompletion tasks are impressive.\n\nIt seems that the authors close an open question posed in (Zaheer 2017)\nregarding how to form embeddings of sets of varying size by embedding\nthe sets into an RKHS instead of a finite-dimensional space. This in itself\nis an interesting idea, and I am interested to see how this embedding method\nbe applied outside of the CNP framework.\n\nThe experimental results are comprehensive and diverse, showing good\nperformance on both toy examples and more real-world problems. The ablations\nand qualitative comparisons in the appendix are helpful in showing where\nthe ConvCNP outperforms the AttnCNP.\n\nMy main criticism of the work is that it's very dense, requiring a few\npasses to really grasp the theoretical contribution and the concrete\narchitecture used in the ConvCNP. I would recommend enlarging figure 1\n(b), which is illuminating but quite cluttered due to the small\nsize. Perhaps the section on multiplicity could be moved to the\nappendix to make space as it seems for all real-world datasets the\nmultiplicity would be equal to 1. \n\n\nMisc Comments\n\n- It would be good to have a brief discussion of why the ConvCNPPXL performs\nvery badly on the ZSMM task, while being the best performing method in all\nof the other tasks. I couldn't find such a discussion.\n- Did the authors try emitting a 36-dimensional joint covariance matrix over the\nsix-dimensional output in the plasticc experiment?\n- In the synthetic experiments, for the EQ and weak periodic kernels it would\nbe nice to see the `ground truth' log-likelihood given by the actual GP,\njust to have some idea of what the upper bound of LL could be.\n- In appendix C.2 Figure 6, what is the difference between the `true function' and the\n`Ground Truth GP'? I thought the true function was a gp...", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "S1eZ24ug9r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper describes a method for model neural for neural processes considering  translation-equivariant embeddings.\n\nThe paper seems to be quite specific topic. Maybe, the author could add more empirical results to it to show the impact on translation-equivariant examples. The theoretical claims seem to be valid. So the question is a bit open what are the applications. The empirical results are also narrow as there is not much other competitive work. The results seem to be increment extension to previous work.  \n\nThe work looks solid to me, currently I am probably not able to appreciate and judge  relevance to its full extend. I would judge, it is more of interest to view specific people working on this - maybe, the authors could for the final version make this more clear.   \n\nThe questions that should be more addressed maybe is also the  applications - why is this relevant and how does it improve your specific cases. Why do we want to develop this. State of the art is quite relative if authors come from a quit narrow area which not much papers on the topic and data sets. \n\nOne of the main points of the paper did not get clear how does translation-equivariant helps to solve or improve the empirical results. Could you add some examples where this improves results. \n\nI remain ambivalent. It seems to be solid work with not much convincing applications and somewhat incremental. Maybe the authors might address this in their introduction more. The motivation remains unclear to me and hence difficult to judge its potential and impact.   \n\n", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents Convolutional Conditional Neural Process (ConvCNP), a new member of the neural process family that models translation equivariance. Current models must learn translation equivariance from the data, and the authors show that ConvCNP can learn this as part of the model, which is much more generalisable and efficient. They evaluate the ConvCNP on several benchmarks, including an astronomical time-series modelling experiment, a sim2real experiment, and several image completion experiments and show excellent results. The authors wrote extensive responses the the reviewers, uploading a revised version of the paper, and there was some further discussion. This is a strong paper worthy of inclusion in ICLR and could have a large impact on many fields in ML/AI. 
Decision: Accept (Talk)