Frequency Decomposition in Neural Processes
Authors: Jens Petersen
Paul F Jaeger
Gregor Koehler
David Zimmerer
Fabian Isensee
Klaus Maier-Hein
Venue:  (2021)

Abstract: Neural Processes are a powerful tool for learning representations of function spaces purely from examples, in a way that allows them to perform predictions at test time conditioned on so-called context observations. The learned representations are finite-dimensional, while function spaces are infinite-dimensional, and so far it has been unclear how these representations are learned and what kinds of functions can be represented. We show that deterministic Neural Processes implicitly perform a decomposition of the training signals into different frequency components, similar to a Fourier transform. In this context, we derive a theoretical upper bound on the maximum frequency Neural Processes can reproduce, depending on their representation size. This bound is confirmed empirically. Finally, we show that Neural Processes can be trained to only represent a subset of possible frequencies and suppress others, which makes them programmable band-pass or band-stop filters.

Under review as a conference paper at ICLR 2021
FREQUENCY DECOMPOSITION IN NEURAL PROCESSES
Anonymous authors Paper under double-bling review
ABSTRACT
Neural Processes are a powerful tool for learning representations of function spaces purely from examples, in way that allows them to perform predictions at test time conditioned on so-called context observations. The learned represen- tations are finite-diment miiee-dimensional, while function spaces are infiiiteddimessional, and so far it has been unclear how these representations are learned and what kinds functions can be represented. We show that deterministic Neural Processes implicitly perform a decomposition of the training signals into different frequency components, similar to a Fourier transform. In this context, we derive a theoretical upper bound the maximum frequency Neural Processes reproduce, depend- ing on their representation size. This bound is confirmed empirically. Finally, we show that Neural Processes can be trained only represent subset of possible frequencies and suppress others, which makes them programmable band-pass or band-stop filters.
INTRODUCTION
Neural Processes (Garnelo et al. 2018a b) are a class of models that can learn a distribution over functions, more generally function space. In contrast to many other approaches that do the same, for example Bayesian Neural Networks, Neural Processes learn an explicit representation of such a function space, which allows them to condition their predictions on an arbitrary number of observations that are only available at test time. This representation is finite-dimensional, while function spaces are infiiie-diimessoonal and so far it has not been understood how they are able to bridge this gap and under what conditions they can successfully do so.
Our work reveals how Neural Processes learn to represent fefintteddimesiinaa function spaces in finiteddimensiona space, and in the process describes constraints and conditions that decide what kinds of function spaces can represented. We begin with an observation that prior art in the con- text of learning sets can be reinterpreted from signal-processing perspective, which allows to derive a theoretical upper bound on the frequencies. i.c. Fourier components, of functions that can be represented. We subsequently confirm this bound empirically, which suggests that the learned representations should contain a notion of frequency. To further investigate this hypothesis, we con- tinue with visualization of the learned representations, which reveals that Neural Processes can decompose a function space into different frequency components, essentially finding representa- tion in Fourier space without any explicit supervision the representations to elicit such behaviour. As further evidence of this we train Neural Processes to represent only certain frequencies, which results in them suppressing those frequencies that were not observed in the training data. Our con- tributions can be summarized as follow!!]
We derive a theoretical upper bound on the signal frequency Neural Processes of given representation size can reconstruct. As we show, the bound is observed either in the ex- pected way-by suppressing high frequencies-ord by implicitly limiting the signal interval.
We investigate learned representations qualitatively, presenting evidence that Neural Pro- cesses perform a frequency decomposition of the function space, akin to a Fourier trans- form. This behaviour is not incentivized externalyy but rather emerges naturally.
'The complete source code reproduce our experiments available

---

Under review as a conference paper at ICLR 2021
We show that by choosing the training distribution appropriately, Neural Processes can made to represent certain frequencies and suppress others, which turns them into pro- grammable band-pas. or band-stop filters.
2 BACKGROUND
Neural Processes (Garnelo et al. 20T b) are maps P C, Y. where C is set of tuples {(z, f(=))]== = (Xe: f(xe) P with arbitrary but positive cardinality N, and f FEF: + Y. often called the context, because Neural Processes perform predictions for values
X (t for target), conditioned on these points. F is the function space we would like to find a representation of. Note that some sources define function spaces any set of functions with shared domain and co-domain, while others require them to be vector spaces as well. We don't t concern ourselves with this distinction and further restrict our work to X Y R. because it allows us visualize learned representations. We only look at the original Neural Processes, namely the deterministic Conditional Neural Processes (CNP) (Garnelo al. 2018a) and the variational Neural Processes (NP) Garnelo et al. (018b), because newer contributions in the field work in ways that preclude them from being analyzed the same way. We discuss this further Section 5 CNPs and NPs, the map P is separated into two parts, a so called encoding E C + Z and a decoding or generating part
X + Y. is referred to as the representation or latent space. To allow Neural Processes to approximate arbitrary function spaces F. E and G are typically chosen to be powerful approximators, specifically neural networks, as the name suggests.
The defining characteristic of CNPs and NPs is that E encodes individual pairs (xc (xx) from the context separately, and the resulting representations are averaged to form a global representation, meaning that is independent of the target points x; which we then evaluate the Neural Process. This often not the case in later work, for example in Attentive Neural Processes (Kim et al.) 2019), where the individual representations are instead aggregated using an attention mechanism that depends on Xt. In CNPs the representations are deterministic, while in NPs they parametrize mean and (log-)variance of Gaussian distribution, so the latter are trained using variational inference. For details on implementation and training we refer to Appendix I Our work will investigate how these global representations, which are finite tiitteddimesionnl. represent ceffnite-dimensinne function spaces.
As stated above, E' and by extension the Neural Process / acts on set-valued inputs. This is contrary to the vast majority of machine learning work where inputs are vectors of fixed dimension and ordering. Recall that sets are permutation invariant, so we must ensure that the same is true for the output of E. It is easy to see that this is given when we average individual encodings, but Zaheer al. (2017) show that is in fact the only way to ensure if it has so-called sum nenopecomposition, i.c. it can be represented in the form
permutation-invarian and only
N
E(x) = P
Σ (as))
where p. é are appropriately chosen functions. Wagstaff al. (2019) further show that to be able to represent all continuous permutation-invarann functions on sets with cardinality of at most N, the dimension of the image Z must at least be N. This will become relevant the following section.
AN UPPER BOUND ON SIGNAL FREQUENCIES
We mentioned the previous section that the encoder E in Neural Process should have a sum- decomposition. so that the global representations are 2017). Expanding on this, Wagstaff al. (2019) show that we require representation size of at least N to be able to represent arbitrary continuous functions on sets of cardinality smaller or equal to N. What these works do not consider are the implications for situations where the elements of
as shown in Zaheer et al.
p We use boldface as a shorthand for sets, not vectors.
This will depend on the implementation of E. and G, and for neural networks F practically restricted to continuous and differentiable functions.
2

---

Under review as a conference paper at ICLR 2021
the sets are input-output tuples of some function f. as it is typically the case in Neural Processes. We will use these previous findings to derive an upper bound on the frequencies v any f E F may contain so that they can be represented in Neural Process. In order to do this, we must first define what means to successfully learn a representation of a function space.
Definition 3.1 (Representation of Function Spaces in Neural Processes). We say that a Neural Pro- cesses P has learned representation of a function space F, defined on an interval [a, bj C R, if, for some error tolerance e, it holds for all æ E [a, bj and for all f E F, represented as a suitable set of discrete measurements (Xe, (x+). that [P f(x)),,
x) f(x) <.
That means the learned representation must be such that we can encode a particular element of the function space f into it and are able to reconstruct up to a predefined error tolerance. The choice of this tolerance is essentially arbitrary, but should reflect that for & F the reconstructions should generally not be accurate within c. We also write that f is represented as suitable set of discrete measurements, by which we mean that it must be possible to reconstruct f from those measurements.
Switching to processin terminology, we know that to represent a continuous signal as a set of discrete measurements, we need to sample it at points with distance of at most 12/(2 max ) where Vmax is the maximum frequency component of the signal. This is most commonly known as the Nyquist-Shannon sampling theorem (Whittaker 1915 Kotelnikov 1933 Shannon 1949). For any finite real interval [a, b], this translates to a number of sampling points N > 2 a. Mmaa- The latter allows to make connection to the findings by Wagstaff et al. (2019). so that can deduce an upper bound on the maximum signal frequency Neural Processes with a given representation size can reconstruct.
Theorem 3.1 (Maximum Frequency in Neural Process Representations). A Neural Process P with latent dimension D, can only learn a representation of some function space F defined on a finite interval (a, b] C R if for all f E F with maximum frequency content Vmmx. it holds that:
D,
Immax
(2)
2 al
Note that this means we should in theory be able to represent any function space that obeys Eq. 2 to within arbitrarily small c. In practice, we will typically have less control over F, and we only find approximate representations. Part of our experiments will test how Neural Processes behave if the signals contain frequencies larger than those allowed by Eq. (2). It should also be noted that the Nyquist-Shannon theorem used for the above derivation assumes equidistant sampling points. During training, we work with randomly sampled imputs, but at test time equidistant points are used, as we outline in Appendix A.2
4 EXPERIMENTS & RESULTS
4.1 VALIDATION OF THE FREQUENCY BOUND
Our experiments are grouped into three parts. The first experiment seeks test the validity of the bound we just derived in Eq. (2). In particular, we train Neural Processes with varying representation sizes on two exemplary function spaces, so that for some models the representation size is insuffi- cient to represent all frequencies. The function spaces we base our experiments on are those defined by Gaussian Process priors (for an introduction see for example Rasmussen & Williams) (2006) with an apponentated-quudaeee by random real-valuee Fourier series for details we refer to Appendix AA2 While the Gaussian Process samples have an average Fourier magnitude that smoothly decays to zero, the distribution of Fourier magnitudes uniform for the Fourier series, as shown in Fig. A. The Fourier series
(EQ) kernel with lengthscale parameter l, as well as those deñned
space also grants precise control over the frequency domain, which will be useful in subsequent experiments.
Figure [ shows example reconstructions in deterministic Neural Process (CNP) for samples from a Gaussian Process prior with EQ kernel (1 = 0.05) and from a random Fourier series. For the GP example, the CNP essentially acts like low-pass filter when the representation size is insufficient, which qualitatively confirms the bound we derived in Eq. (2). Interestingly, the bound can also be observed in different way: for the Fourier series example, the CNP hardly suppresses high
3

---

Under review as a conference paper at ICLR 2021
Regeowrntiaaia Size 61
Reprosent nd las Sion 52
Repmosnttaionn Sizu 16
Prie,
M MMW
MMM
Figure 1: Example reconstructions from a deterministic Neural Process (CNP) with varying repre- sentation sizes. For the Gaussian Process example, reducing the representation size removes higher frequency components. For the Fourier series example, the CNP instead limits the effective interval. Both behaviours respect the bound derived in Eq.
frequencies, but instead limits the effective interval of the signal. simply igmoring the outer regions of it. Both behaviours are in agreement with the bound in Eq. (2). The Fourier example also serves as good sanity check: with K 19 (the maximum angular Irequency) the data has maximum frequency of V/nax K/(2x) 3.02. For D, 32 this would limit the size of the interval to }b a 5.29, for D, 16 al < 2.65. The reconstructed signal regions in Fig. are a bit narrower, and thus in good agreement with the bound we derived. For a variational Neural Process, we observe the same behaviour, but with stronger dampening high frequencies both cases, seen in Fig. A.2, In Fig. A.3 we show the average reconstruction error for CNPs and NPs of different representation sizes, applied to GP examples with varying lengthscale, which results in smooth decrease in error for larger representations and larger lengthscale parameters, would expect.
4.2 HOW DO NEURAL PROCESSES REPRESENT FUNCTION SPACES?
Having found that Neural Processes do indeed observe the bound we derived Eq. 2 we seek to understand how this happens. To this end, we visualize the learned representations in Neural Processes, which is possible because we restrict ourselves to X Y R. Again looking at the two function spaces from the previous experiment, we sample pairs (Ir, y) with æ E [-3.3] which is our training input range, and also y E [-3,3] 3,,3) as it suitably covers the value range outputs. We then encode each pair individually to a representation, thus constructing a map r=(2,8) y) for each representation channel. The latter allows us to uncover potential patterns and to gain a better understanding of how Neural Processes learn representations of function spaces.
regular grid (50 x 50)
Figure 2 presents example representation channels for CNPs and NPs, trained on samples from Gaussian Process with an EQ-kernel (1 0.2) and on random Fourier series. The individual channels were selected to illustrate the general patterns of behaviour we observed. First, we find that representations are almost always anti-symmetric across = This is not surprising, as the function spaces we look at are average symmetric-in the sense that f and - will occur with the same probability-so the Neural Process learns the same representation, just with different sign. More importantly, we find that both NPs and CNPs implicitly form representation of the input space (i.e. the relevant interval of the function space domain). in the sense that different regions of the input space map different representation channels. In CNPs this results in an oscillating pattern, with different channels exhibiting different frequencies. other words, the CNP performs frequency decomposition of the function space, unlike Fourier transform. At the
4

---

Under review as a conference paper at ICLR 2021
GP Prior, t=
-3.0
Coatedt y [a...)
-1.5
CNP
0.0
1.5
3.0
F
Context [s.n.
-1.5
NP
0.0
1.5
3.0
Fourier Scries, K
19
-3.0
Cont y (....)
CNP
0.0
1.5
3.0
==7
2-
Coatedt [aa.)
-155
NP
0.0
1.5
3.0
-3.0
-1.5
0.0
1.5
-300
-15
0.0
1.
3.0
-3.0
-1.5
0.0
1.5
-300
-1.5
0.0
1.5
3.0
-3.0
-1.5
0.0
15
3.0
Context [a.u.]
Figure 2: Influence of the context on the learned representations deterministic Neural Process (CNP) and variational Neural Process (NP) with D, 128 and data coming from Gaussian Process prior with an EQ kernel with / = 0.2 (top half, kernel description in Eq. (00) or a Fourier series with K 19 described by Eq. I (bottom half). refers to the input space (i.e. the domain), y to the output space (i.e. the co-domain). Representation channels are ordered by their average magnitude. We find that the learned representations are almost always anti-symmetric across O. The CNP exhibits an oscillating behaviour along the x-axis, with varying frequency across representation channels. In contrast, the NP seems to mostly partition the input space into separate regions, only exhibiting oscillating behaviour in channels with lower magnitude. The representation channels in this example were selected for illustrative purposes. Note that each panel is normalized separately, so color values are not comparable.
same time, there is nothing that would enforce orthogonality between the different representation dimensions, and the Fourier series example highlights that we can generally expect mixture of multiple frequencies for given dimension. It should be noted that this frequency decomposition emerges naturally and is not incentivized externally (e.g. by special loss).
Even though NPs behaved very similarly to CNPs in the previous section, their learned representa- tions look vastly different from those CNP. Instead of a frequency decomposition, they seem to partition the input space, so that given representation dimension is written to by specific, narrow region of the input space. Only for channels with a low average magnitude (i.e. a large index in Fig. 2 do we find behaviour similar to CNPs. We conclude that NPs can in principle learn frequency decomposition, but their variational formulation the only difference to CNPs disincentivizes it. We show more representations for CNPs and NPs trained GP data in Fig. A.4 and Fig, [.5) and for CNPs and NPs trained on Fourier series data in Fig. A.6 and Fig. A.7 sorting channels by their average magnitude.

---

Under review as a conference paper at ICLR 2021
Recienmur
Bamblaa
Bandlua
Sipua Darahh
Eonter Decats (iturnua))
01-
50-
2.5
2.1
Unllitered
Hundeop
Baxalpane
30
00
11
Prayurncy
Figure 3: Deterministic Neural Process (CNP) trained to be band-pass and a band-stop filter. We train three CNP models Fourier series, where only show certain frequencies during training for the models that should serve as frequency filters. The bottom row shows the distribution of Fourier components in the training data (left) and after applying each model to the reference data (right). The models only learn to represent frequencies seen during training, and as result act like band-pass and band-stop filters. The top rows show example of this. See Fig. A.8 for the same experiment using a variational Neural Pracess.
4.3 NEURAL PROCESSES AS BAND FILTERS
Our final experiment designed to show that we can exert more control over the learned represen- tations, and it will serve additional evidence that deterministic Neural Processes (CNP) perform a frequency decomposition possible practical application of Neural Processes. We saw in Section 4.1 that CNPs sometimes like low-pass filters, which could be a useful application, but the emergence of that behaviour not reliable. We now train CNPs with a sufficiently large representation size (D,- 128) be band- pass and band-stop filters. To this end, we train the models on the Fourier series defined by Eq. (II), but for the band-stop set all components ag zero for which < k s 14, and likewise all aj to zero outside of that range for the band-pass We then look at the reconstructions of examples from the original series with all components present. For more details on the training procedure and how we sample points for function evaluation, please see Appendix A. and Appendix A.2
the function space they represent. At the same time, it suggests a
The average Fourier magnitude of the training functions for the different models is given by the bottom left panel in Fig. 3 In the first model (Reference), all components are allowed; in the second (Band-stop), components in the middle of that range are suppressed; in the third (Band-pass) only components in the middle of the range allowed. We then apply these models to examples from the reference data distribution, the result which can be seen in the bottom-right panel of Fig. 3 The models that are only shown certain frequencies during training will suppress those frequencies that were not found in the training data, meaning they effectively become programmable band-stop band-pass filters. This is confirmed by the example in the top rows of the figure, where show

---

Under review as a conference paper at ICLR 2021
both the signal and its Fourier transform magnitude. Note that one needs to adjust the value range of the reference data before passing them through the band filters to prevent gain in the non-suppressed frequency regions. We give more details in Appendix A.2
Unfortunately, we were only partly able to clicit the same behaviour in variational NPs. While the trained band-stop filter worked exactly like the CNP band-stop, were not able train band- pass filter. The models collapsed during training, meaning the loss plateaued and no meaningful representations were learned. There is no obvious reason why a band-pass shouldn't 't work when band-stop does, so we suspect our hyperparameter configuration was not ideal and that with more tuning would be possible to train a pass as well. The NP results are shown in Fig. A.8
5 RELATED WORK
Neural Processes broadly relate to the topic of learning distributions of functions, even though speak of the less restrictive term function space in our work. In this context, Bayesian Neural Networks (see for example Neal (1996); Graves (201T); Hernández-Looato & Adams (2015)
a popular choice, which place distributions on the weights of a network. However, in doing so they only implicitly represent distributions over functions, while Neural Processes learn explicit miite-dimensional representation that can be leveraged for predictions, so as to condition on context observations given test time. Perhaps the most well known class of methods that do the same are Gaussian Processes (for an introduction see Rasmussen & Williams )2066). These are stochastic processes represented by a joint Gaussian distribution over context and target points, defined via the covariance matrix by kernel. All flexibility of Gaussian Processes to represent different distribu- tions of functions is decided by this kernel, so many works try to learn it (Yang et al. 2015] Wilson et al. 2016b at Tossou et al. 2019] Calandra al. 2016). Even though Neural Processes were orig- inally motivated by Gaussian Processes, they can be understood as orthogonal methods: Gaussian Processes represent function space using (potentially learned) kernel, while Neural Processes represent them in a learned finite-dimensional space.
Neural Processes can also be interpreted from the perspective of deep learning on sets, the earliest work in the field being Zaheer et al. (2017). More theoretical contributions were made by Wagstaff let al. (2019), whose work we use to underpin our finding that the representation size in Neural Processes limits the maximum frequency of signals that can be represented. More applied work in the set-lcarning context has mostly been performed on point-cloud data (Qi et al.) 2017b a Wu let al. 2019), which can be interpreted as a higher- gigher-dimensione instance of learning function spaces. Validating our findings in higher-dimensionss
spaces is an important direction for future work.
Neural Processes have inspired number of follow-up works. Perhaps the most well known addition are Attentive Neural Processes (Kim et al., 2019), which replace the averaging of individual repre- sentations with learned attention mechanism (Vaswani et al. 2017). The aggregate representations are thus no longer independent of the target inputs, and no global representation learned. This holds true for most follow-up work. Convolutional Conditional Neural Processes (Gordon et al.) 2020) propose to no longer learn a finite-dimensional representation at all and instead work in func- tion space by applying CNN suitable and variable discretizations of a kernel density estimate. Similar to ANP, Louizos et al, 2019) propose to not merge observations into a global latent space, but instead learn conditional re relationships between them. [Singh al. 2019) and Willi et al. (2019) address the problem of overlapping and changing dynamics in time series data. Relating this to our work, it would be possible to test how the original Neural Processes would represent functions where the average frequency content is not constant over the domain. We leave this investigation for future work. Neural Processes have also been extended to scenarios where the function space maps to entire images, in the form of Generative Query Networks (GQN) (Eslami et al. 2018 Kumar et al. 2018). Employing vastly more powerful decoders, they can (re- Jconstruct unseen views in 31> scenes, which relates Neural Processes to the field of 3D scene understanding, an area that has re- ceived a lot of attention more recently (Sitzmann et al. 2019 Engelcke et al. 2020 Mildenhall et al.) 2020). Sitzmann et al. (2020) show that periodic activation functions make it easier for networks to learn so-called implicit pppresenationssmappiee mappings from coordinates to a density, RGB values, etc.. We did in fact try periodic activation functions in our experiments, but found no difference to using aotnh-acivations. In the same context, Tancik et al. (2020) show that coordinates in Fourier space are often superior to coordinates in signal space to produce fine detail. We interpret this an indication
7

---

Under review as a conference paper at ICLR 2021
that representation in frequency space is more efficient for many signals, which could explain why Neural Processes implicitly perform a frequency decomposition. Note that the above introduces Fourier features explicitly form of inductive bias, while Neural Processes automatically learn this form of representation.
is well known that neural networks, specifically a MLP with at least one hidden layer, can learn the Fourier transform of an input signal (Gallant & White 1988). In fact, there have been a multitude of works that exploit this ability in one way or the other, leading to the term Fourier Neural Networks. We refer to the recent review by Zhhmekenov et al. (2019) for a comprehensive overview. The difference Neural Processes is that these works typically apply networks directly to a sequence of points, while NPs learn a mapping that is only applied to individual (x.y) pairs, the representations of which are averaged. We emphasize again that the frequency decomposition occurs naturally in NPs, while these works usually employ direct supervision.
6 DISCUSSION
The goal of this work was to gain better understanding of the mechanisms that allow Neural Processes to form finite-dimensional representations of infiniteddimensional function spaces. To the best of our knowledge, ours is the first work investigate this question, and our findings are both surprising and meaningful in this context. We first derived a theoretical upper bound on the frequency of signals that can be represented in Neural Processes with given representation size. We empirically confirmed that the representation size does indeed pose such a limit and that this can result in Neural Processes acting like low-pass filters. Alternatively, models ignore parts of the signal to keep higher frequencies. Both behaviours are in agreement with the derived bound. We then visualized learned representations to understand how the models incorporate the concept of frequency into them. In all cases the models formed an implicit representation of the input space, in the sense that different x-values are mapped to different representation channels. For CNPs, an oscillating pattern emerges, such that different representation channels correspond to different frequencies, from which we concluded that CNPs perform a frequency decomposition of the function space they learn to represent. It should be noted that this behaviour emerges naturally and is not explicitly encouraged (e.g. by a special loss). In contrast to this, NPs tend to partition the space into more or less disjunct regions. They are still able to learn a frequency decomposition like CNPs, but we assume that the variational training objective makes it harder to do so, as sampling from the representation during training can also be understood as a random perturbation. For VAEs, which are conceptually similar to NPs, it was also suggested that models partition their latent space in way that maximally spreads representations of individual data points under the prior distribution (Rezende & Viola 2018). Finally, to further test the models' ability to distinguish frequencies and also as an example of possible practical benefits of our findings, we trained CNPs to be band-pass and band- stop filters. This worked extremely well, the Fourier component magnitudes of the training data are essentially "baked" into the models, and any frequency not found therein is subsequently suppressed in reconstructions from the models. An obvious use case would be programmable frequency filters, when perhaps a more complex frequency response is desired.
Overall, our work offers exciting new insights into the inner workings of Neural Processes and into the learning of representations of function spaces. Many applications of deep learning are concerned with representation learning in some way, and we hope that our findings inspire further research and forge a better understanding of the methods used in the field. Our work also opens up number of exciting questions for future work. We only look at function spaces with scalar domain, and while we expect that our findings translate to higher dimensions, the same should be validated empirically. Seeing that variational Neural Processes can in principle learn frequency decompositions, would be interesting to investigate how we can further incentivize this behaviour in them. Likewise, it should be possible to encourage orthogonality between the individual representation dimensions, so that frequencies are more cleanly separated. Further theoretical exploration of the conditions, besides frequency content, that allow function spaces to be represented could also be worthwhile. Finally, it is not immediately obvious how our findings translate to scenarios that disallow a classical definition of frequency, for example when the observations are entire images as in Eslami et al. (2018).
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Christopher M. Bishop. Pattern recognition and machine learning. Information science and statis- tics. Springer, 2006.
Roberto Calandra, Jan Peters, Carl Edward Rasmussen, and Mare Peter Deisenroth. Manifold gaus- sian processes for regression. In International Joint Conference on Neural Networks, 2016.
Martin Engelcke, Adam R. Kosiorek, Oiwi Parker Jones, and Ingmar Posner. GENESIS: Gener- ative scene inference and sampling with object-centric latent representations. In International Conference on Learning Representations, 2020.
S. M. Ali Eslami, Danilo Jimenez Rezende, Frederic Besse, Fabio Viola, Ari S. Morcos, Marta Garnelo, Avraham Ruderman, Andrei A. Rusu, Ivo Danihelka, Karol Gregor, David P. Reichert, Lars Buesing, Theophane Weber, Oriol Vinyals, Dan Rosenbaum, Neil Rabinowitz, Helen King, Chloe Hillier. Matt Botvinick, Daan Wierstra, Koray Kavukcuoglu, and Demis Hassabis. Neural scene representation and rendering. Science, 360(6394)) 224-11210 2018
Gallant and White. There exists a neural network that does not make avoidable mistakes. In IEEE International Conference Neural Networks, volume 1, pp. 657-664, 1988.
Marta Gareeo,, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Teh, Danilo Rezende, and S. M. Ali Eslami. Conditional neural processes. In International Conference on Machine Learning, pp. 1704-1713, 2018a.
Marta Garnelo, Jonathan Schwarz, Dan Rosenbaum, Fabio Viola, Danilo J. Rezende, S. M. Ali Eslami, and Yee Whye Teh. Neural processes. In International Conference on Machine Learning Workshop on Theoretical Foundations and Applications of Deep Generative Models, 2018b.
Jonathan Gordon, Wessel P. Bruinsma, Andrew Y. K. Foong, James Requeima, Yann Dubois, and Richard E. Turner. Convolutional conditional neural processes. In International Conference Learning Representations, 2020.
Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, pp. 2348-2356, 2011.
José Miguel Hernández- Lobatto and Ryan P. Adams. Probabilistic backpropagation for scalable learning of bayesian neural networks. In International Conference Machine Learning, vol- ume 37, pp. 1861-1869, 2015.
D. Jimenez Rezende, S. Mohamed, and D. Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International Conference on Machine Learning, 2014.
Hyunjik Kim, Andriy Mnih, Jonathan Schwarz, Marta Garnelo, Ali Eslami, Dan Rosenbaum, Oriol Vinyals, and Yee Whye Teh. Attentive neural processes. In International Conference on Learning Representations, 2019.
D. Kingma and J. Ba. Adam: method for stochastic optimization. In International Conference on Learning Representations, 2015.
D. P Kingma and M. Welling. Auto-encoding variational bayes. In International Conference Learning Representations, 2014.
V. Kotelnikov. On the transmission capacity of the "ether" and wire in electrocommunicaiion.. In Proceedings of the first All-Union Conference on the technological reconstruction of the commu- nications sector and the development of low-current engineering, 1933.
Ananya Kumar, S. M. Ali Eslami, Danilo J. Rezende, Marta Gamelo, Fabio Viola, Edward Lockhart, and Murray Shanahan. Consistenttjump predictions for videos and scenes. Advances Neural Information Processing Systems Bayesian Deep Learning Workshop. 2018.
Tuan Anh Le, Hyunjik Kim, Marta Garnelo, Dan Rosenbaum, Jonathan Schwarz, and Yee Whye Teh. Empirical evaluation of neural process objectives. In Advances Neural Information Pro- cessing Systems Bayesian Deep Learning Workshop, 2018.

---

Under review as a conference paper at ICLR 2021
Christos Louizos, Xiahan Shi, Klamer Schutte, and Max Welling. The functional neural process. In Advances in Neural Information Processing Systems, pp.
2019.
Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoor- thi, and Ren Ng. NeRF: Representing scenes as neural radiance fields for view synthesis. arXiv:2003.08954 [cs], 2020.
Radford M. Neal. Bayesian Learning for Neural Networks. Lecture Notes in Statistics. Springer, 1996.
Charles R. Qi, Hao Su, Mo Kaichun, and Leonidas J. Guibas. PointNet: Deep learning on point sets for 3d classification and segmentation. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 77-85, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas Guibas. PointNet++: Deep hierarchical feature learning on point sets in a metric space. In Advances in Neural Information Processing Systems, pp. 5099-5108, 2017b.
Carl Edward Rasmussen and C. K. I. Williams. Gaussian Processes for Machine Learning. MIT Press, 2006.
Danilo Jimenez Rezende and Fabio Viola. Taming VAEs. arXiv:1810.00597 [cs, stat), 2018.
C.E. Shannon. Communication in the presence of noise. Proceedings of the IRE, 37(1):10-21, 1949.
Gautam Singh, Jaesik Yoon, Youngsung Son, and Sungjin Ahn. Sequential neural processes. In Advances in Neural Information Processing Systems, pp. 10254-10264, 2019.
Vincent Sitzmann, Michael Zollhoefer, and Gordon Wetzstein. Scene representation networks: Con- tinuous 3d-structure-ave awrrr neural scene representations. In Advances in Neural Information Pro- cessing Systems, pp. 11 1119-1130, 2019.
Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, and Gordon Wet- zstein. Implicit neural representations with periodic activation functions. wrXiv:2006.0966 [cs, eess], 2020.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. In Advances in Neural Information Processing Systems, 2020.
Prudencio Tossou, Basile Dura, Francois Laviolette, Mario Marchand, and Alexandre Lacoste. Adaptive deep kernel learning. 1rXiv:1905.12131 [cs, stat), 2019.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances Neural Infor- mation Processing Systems, pp. 5998-6008. 2017.
Edward Wagstaff, Fabian B. Fuchs, Martin Engelcke, Ingmar Posner, and Michael Osborne. On the limitations of representing functions on sets. In International Conference Machine Learning, 2019.
E. Т. Whittaker. Xviii. the functions which represented by the expansions of the interpolation-theory. Proceedings of the Royal Society of Edinburgh, 35:181-194, 1915.
Timon Willi, Jonathan Masci, Jürgen Schmidhuber, and Christian Osendorfer. Recurrent neural
processes. arXiv:1906.059/ [cs, stat], 2019.
Andrew G Wilson, Zhiting Hu, Russ R Salakhutdinov, and Eric P Xing. Stochastic variational deep kernel learning. In Advances in Neural Information Processing Systems, pp. 2586-2594, 2016a.
Andrew Gordon Wilson, Zhiting Hu, Ruslan Salakhutdinov, and Eric Xing. Deep kernel learning. In International Conference on Artificial Intelligence and Statistics, 2016b.
10

---

Under review as a conference paper at ICLR 2021
Wenxuan Wu, Zhongang Qi, and Li Fuxin. PointConv: Deep convolutional networks on 3d point clouds. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 9621 9630, 2019.
Zichao Yang, Andrew Wilson, Alex Smola, and Le Song. A la carte learning fast kernels. In International Conference Artificial Intelligence and Statistics, pp. 1098-1106, 2015.
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep sets. In Advances Neural Information Processing Systems, pp- 3391-3401, 2017.
Abylay Zhumekenov, Malika Uteuliyeva, Olzhas Kabdolov, Rustem Takhanov, Zhenisbek Assyl- bekov, and Alejandro J. Castro. Fourier neural networks: A comparative study. 1rXiv:1902.03011 [cs]. 2019.
A APPENDIX
A.1 OPTIMIZATION & IMPLEMENTATION
To train Neural Processes, represent individual examples f E F sets of randomly sampled evaluations (x, fxx = y). which we partition into context set (Xe.Ye) and target set (x). We further have encoder E and decoder G of Neural Process implemented for which we summarize the parameters in 0. In our implementation, both are multilayer percep- trons (MLP), meaning simple fully connected networks. Our goal is then find the optimal set of parameters O* that maximizes the likelihood of yt, given Xc. Ye and xt, over all f:
neural networks,
0" arg max
log Nc)
FEF
where po placeholder for some parametrized likelihood function. We introduce the logarithm because we assume the likelihood factorizes across individual f. turning the expression into sum. So what would this optimization look like practice? For example, we could minimize the mean squared error between yt and the predictions Yt from network. This would implicitly assume Gaussian likelihood with a fixed variance. However, we would like our model to predict a variance, so that it indicate how uncertain it is about prediction, and because Le et al. (2018) found that this results in overall better performance. We achieve this by implementing G as a network that predicts both the mean and the variance of diagonal Gaussian distribution, and Eq. becomes:
0" arg max
Σ Σ log (1,,,, x:)) FEF
In deterministic Neural Processes (CNP), we can directly optimize this with maximum likelihood training. In variational Neural Processes (NP), Z is also parametrized by Gaussian, meaning just like G. E predicts mean and variance of Gaussian with D, dimensions. In this case, we need to rewrite the summands of Eq. (3)
log (ye Xo, Xo, Yo) = log
pe (e (x,
(5)
zmupl (2 Xx)
Here, p(Z|xe: Yc) is not the distribution predicted by our encoder, but some true distribution don't have access to. The idea of variational inference (see for example Bishopl (200)) for an introduction) is to approximate this p by some other distribution qe and then to optimize pa and qo simultancously. 9o is what our encoder E predicts, just like pp is what our decoder G predicts. Continuing from Eq. 5
11

---

Under review as a conference paper at ICLR 2021
p xc: Nc.
LHS = log
E
po(yc|xe,.2
(6)
go (2 N+)
Xt, Yt)
Nc) 40 (2) |x:
p( |ex Nc)
E
log
polye
20 (
90 Nc.Ye
E
log po poo 2)
z xx)
go (xt,
E
log (y(c
20 2 Nc.ve) - 3e))o
(9)
where LHS refers to the left hand side of Fq. (). In the first line, we have switched the underlying distribution from the true meaning conditioned on the context-to an approximate poste- rior-meaning conditioned on both context and target, but for notational simplicity we only write out the target set. The second line follows from Jensen's incquality while in the third line we have replaced the true prior with the approximate prior. Finally, we have rewritten the right hand side using the Kullback-Leibler (KL) divergence, a measure of distance between two distributions. Be- cause we predict Gaussian distributions, the KL divergence has a closed-fore expression. Otherwise would be impractical to use it in an optimization context. The last line is often called the evidence lower bound (ELBO) in variational inference.
Let us put the above into more practical terms. When presented with an example consisting of context and target sets, we first use the encoder network E to encode each context tuple separately. The encoder a MLP with two input channels (for X and Y), 6 hidden layers with 128 channels, and final layer mapping to D, channels, i.e. to the representation. While all hidden layers have a fixed dimension of 128, we vary the representation dimension D, for our experiments (but never make it larger than 128). For the variational case, the final layer maps 2D, channels, half for the mean and half for the variance of the predicted Gaussian (in practice, we predict the log-variance allow negative values). The individual representations are then averaged, and the variational case we call this the prior (qa (z) x/xee Ye) in Eq. (9). For the posterior, we also encode the target pairs and then average over all individual representations, including the context. During training forward passes, we sample once from the posterior and use this sample as the representation for the decoder. Ideally, we should sample many times to integrate the expectation mini-batch training it was found empirically that single sample suffices (Jimenez Rezende et al.) 2014 Kingma & Welling, 2014). The decoder predicts a Gaussian from the representation and an input Z2, It is implemented symmetrically to the encoder, meaning it's MLP with D, + 1 input channels, 6 hidden layers with 128 channels, and two output channels for mean and (log-)variane We use tanh-activations as well. As loss directly evaluate the likelihood of reference point under a Gaussian parametrized by the predicted mean and variance. Finally, we average over all predicted points, which are the target points as well as the context points. We use the Adam optimizer Kingma & Bal with an initial learning rate of 0.001, repeatedly decaying with factor of 0.995 after 1000 batches. We train with a batch size of 256 for a total of 600 000 batches.
Eq. (9), but for stochastic
the negative lielliie,, meaning
A.2 EXPERIMENT DETAILS
We conduct our experiments on two kinds of function spaces. The first is defined by a Gaussian Processes prior using an EQ kernel given by:
|| xzll
k(11, = exp
(10)
21
where is lengthscale parameter. This example was also used in the original works (Garnelo et al. 2018 b). The second are random Fourier series, defined by:
12

---

Under review as a conference paper at ICLR 2021
f(x) 00 ag cos (ker Φx) K 19
(11)
where we sample 6b and ag (including ao) randomly from the interval [-,, Note that ls is an angular frequency, while results are presented for regular frequencies.
To construct training examples, we sample N context inputs and M target input values uniformly from the range [-3, 3]. N is random integer from the range [3. 100), while M is random integer from [N, 100). This sampling strategy was adopted from the original works and Le et al. (2018). y- values generated by evaluating the above functions (or drawing from the distribution in the case of the GP) on the random input values. The models are trained by letting them predict/reconstruet both context and target points, conditioned only the context. At test time, reconstructions, meaning target points and context points are identical, and we work with 200 equally spaced input values across the full range.
only interested
In the band filter experiment, we train models on Fourier series with some frequencies intentionally left out of the training data. When train model data where some frequency components blocked, the distribution of y-values model sees during training becomes narrower. As result, passing functions from the reference distribution (where no components are blocked) through band-filter CNP will suppress the desired frequencies, but will also amplify non- con-blocked frequencies. To counteract this, we have to multiply the y-values of the reference data, which are approximately normally distributed, by Thana/ ress i.e. the ratio of standard deviations of the relative y-distributions.
A.3 ADDITIONAL VISUALIZATIONS
GP Priur,
GP Pricr,
Rasrier Serieee
Eaxapie (bx4
Auoea Rrate Mattinhh 16-00
70-
10-
2.5
Ensepancc
Precsonce [uX]]
Pioquency [c
Figure A.1: Examples from the function spaces employed in this work (top row) and the correspond- ing average Fourier magnitudes (bottom row), averaged over 1000 samples. The Gaussian Process kernel and the expression for the Fourier series are given in Eq. (TO) and Eq. (II).
13

---

Under review as a conference paper at ICLR 2021
Regersentatio Siec
Regresentatton Size 32
Reposentation Size
Unditood Fileodd
AMA
NM4
MMM
Figure A.2: Example reconstructions from variational Neural Process (NP) with varying represen- tation sizes. For the Gaussian Process example, the NP essentially behaves like the CNP as seen in Fig. I but with stronger dampening the higher frequencies. Like the CNP, also limits the effec- tive interval for the Fourier series example, but again with stronger dampening of high frequencies.
14

---

Under review as a conference paper at ICLR 2021
CNP Reconstruction RMSE
128
0.23
0.18
0.13
0.09
0.07
0.05
0.04
0.5
112
0.23
0.18
0.13
0.10
0.07
0.05
0.05
Representation Size
96
0.18
0.13
0.07
0.04
0.4
80
0.24
0.20
0.13
0.10
0.08
0.06
0.04
0.3
0.26
0.22
0.16
0.11
0.08
0.05
0.05
48
0.29
0.25
0.18
0.12
0.09
0.06
0.04
0.2
32
0.35
0.30
0.25
0.20
0.14
0.06
0.04
0.1
16
0.52
0.45
0.35
0.28
0.24
0.22
0.12
0.01
0.02
0.05
0.1
0.2
0.5
1.0
GP Kernel Lengthscale
NP Reconstruction RMSE
128
0.36
0.27
0.20
0.16
0.12
0.09
0.08
0.5
112
0.36
0.28
0.20
0.16
0.13
0.09
0.07
Representation Size
0.27
0.21
0.16
0.13
0.07
0.4
80
0.36
0.30
0.21
0.16
0.13
0.10
0.08
0.3
0.38
0.32
0.21
0.17
0.13
0.09
0.08
48
0.39
0.32
0.22
0.17
0.14
0.10
0.08
0.2
0.43
0.37
0.28
0.21
0.16
0.11
0.09
0.1
0.58
0.52
0.43
0.35
0.28
0.19
0.17
0.01
0.02
0.05
0.1
0.2
0.5
1.0
GP Kernel Lengthscal
Figure A.3: Reconstruction performance (RMSE) of CNPs (top) and NPs (bottom) on data generated from a GP prior with an EQ kernel. We vary the kernel lengthscale smaller lengthscale means higher frequency content-ann and the representation size in the Neural Processes. Evidently, larger representation allows the models to better reconstruct data with smaller lengthscale. Overall, the RMSE in CNPs is little lower than in NPs.
15

---

Under review as a conference paper at ICLR 2021
-30
Cantaat (axu]
-1.3
1.5
3.0
3.0
Caitaa [uu]
0.0
13-
Contest y [ua]
-1.5
0.-
1.5
3.00
Contost [ua]
-1.5-
0.-
3.0
-000
Chilext Jau]
-1.5
3.0
3300
Cuuled Joa]
-1.5
0.0
38
Qualext y [ua]
-1.5
0.0
1.5
30
Caulaxt y [ua]
-1.5
0.-
-3.0
.15
1.5 3.0-3.0
-1.5 no 1.3.
-1.5
-1.5 0.0
-1.5 ola 30-3.0 -1.5 0.0
Context [an]]
Contest (an.
Cuunect (a.u.]
Coutect [piu]
Content Jai..
Clonleat jun.]
Figure A.4: Influence of the context on the learned representations in a deterministic Neural Process (CNP) with D, 128 and data coming from Gaussian Process prior with EQ kernel with == 0.2. The description of the kernel is given in Eq. (10). refers to the input space (i.e. the domain), y to the output space (i.e. the co-domain). These are the first 48 representations ordered by their average magnitude (lett-to-rihht, top-to-bottom). Note that each panel is normalized separately, so color values are not comparable. While the representations are more or less anti-symmetric across y O, they exhibit an oscillating pattern along the x-axis. The frequency of those oscillations varies for the individual representation channels, leading us to conclude that CNPs perform a frequency decomposition of the input space.
16

---

Under review as a conference paper at ICLR 2021
1w0
Cantod y (a.)
-1.-
0.0-
3.0
1-6
Contodd ((o)]
-1.5
3.0-
300-
Cuntod (ax)
1.5
1.5
30
300
Chubted ((a)]
0.0
1.3
2-
2=22
2H
Chutod (ax)
1.5
0.0
-3.0
Chaboot [aab)
1.55
-3.0
Cuuttal (ax)
1,5
1.5
3.0
-3.0
1-42
2-13
x--05
2-
Clureat [[a]
1.5
3.0
3.0
-1.5
3.0 3.0
11.5
3.0
115
-1.5
0.0
115
0.0
3.0 -1.5
0.0
[[s]]
[cn.]
Contest (aa]]
Clonkeat (un]]
Figure A.5: Influence of the context on the learned representations in variational Neural Process (NP) with D, = 128 and data coming from a Gaussian Process prior with an EQ kernel with = 0.2. The description of the kernel given in Eq. (10). a refers the input space (i.e. the domain), to the output space (i.c. the co-domain). These are the first 48 representations ordered by their average magnitude tleft-to-right, top-to-bottomm.. Note that each panel normalized separately, so color values are not comparable. Unlike the representations learned in CNP seen in Fig. A.4 the NP seems to partition the input space into almost discrete regions.
17

---

Under review as a conference paper at ICLR 2021
1-0
Contod y (a.)
-1.5
3.-
Contodt ((o)]
-1.5
3.0
Cunted (ax)
1.5
1.5
30-
300
Cuubed ((а)]
0.0
1.5
Chutod y (ao)
1.5
0.-
33.0
Chubott (au)]
1.5-
-3.0
Cuuttat [ax)
4,5
1.5
3.0
-3.0
Clureat [[a]
-1.5
0.0
1.5
3.0
3.0
-1.5
3.0 3.0
1.5
3.0 3.0
-1.5 0.0
1155
0.0
3.0
1155
0.0
[[
[8.0.]
Contest (aau
Clonkeat jauu]]
Figure A.6: Influence of the context the learned representations in deterministic Neural Process (CNP) with D, 128 and data coming from a random Fourier series with K = 19 as described by Eq. (T).. x refers to the input space (i.e. the domain), y to the output space (i.e. the co- domain). These are the first 48 representations ordered by their average magnitude (left-to-right, cop-to-bottom). Note that each panel normalized separately, so color values are not comparable. Similar to the GP example in Fig. A.4] the CNP exhibits an oscillating pattern along the x-axis. The individual frequencies are not cleanly separated by the representation channels.
18

---

Under review as a conference paper at ICLR 2021
-30
Caniaal (ax)]
-1.3
1.5
30.0
3.0
[-
30
Cantaa [uu]
0.0
13-
1-12
Contest: y [[a]
-1.5
0.0
3.00
Chntost [ua]]
-1.5-
3.0
-300
Chilest [au]
-1.5
300
-3.0
Chuled Joa]
-1.5
0.00
3.0
20000
Chutext [ua]
0.0
1.5
-3.0
Caulext y [ua]
-1.5
0.0
-3.0
-1.5 1.5
3.0-3.0
-1.5
1.3.
-1.5 0.0
-1.5
30-3.
-1.5
Context [an]]
Contest (an.
Cunnest [u..]
Coutect
Contedt Jan]]
Conteat jun.]
Figure A.7: Influence of the context on the learned representations in variational Neural Process (NP) with D, 128 and data coming from random Fourier series with K 19 described by Eq. (T).. x refers to the input space (i.e. the domain), to the output space (i.e. the co- domain). These are the first 48 representations ordered by their average magnitude (left-to-right, top-to-bottom). Note that each panel is normalized separately, so color values are not comparable. Similar the GP example in Fig. A.3 the NP partitions the input space into virtually discrete regions, exhibiting even narrow and more sharply separated regions. Representations with higher index also show oscillating behaviour similar to a CNP, indicating that it's in principle possible for NPs to learn frequency decomposition.
19

---

Under review as a conference paper at ICLR 2021
Redeencce
Banktop
Dundpass
Spral Danaki
Founse (Mtastrek))
100
100-
46
00-
20
20
Broganacy J.AU)
Preperaryy jos.]
Presucoee [oo]
Unfiltrrd
Filternd
Reforace
Budlop
54
50
Bardpan
21-
35
Figure A.8: Variational Neural Process (NP) trained to be a band-pass and a band-stop filter. We train three NP models Fourier series, where we only show certain frequencies during training for the models that should serve as frequency filters. The bottom row shows the distribution of Fourier components in the training data (left) and after applying each model the reference data (right). While the band-stop NP appears to work quite similar to the CNP seen in Fig. 3 we were not able successfully train band-pass NP.
20


Reviews:
[{"review_id": "xix2Dkwp85s", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents an analysis on the neural processes in the signal processing point of view and gives a bound on the highest frequency of the function that a neural process can represent.\n\nI recommend to reject this manuscript. My comments are below.\n\nThe key point of this work is Theorem 3.1. However the theorem itself is just a direct outcome of the Nyquist–Shannon sampling theorem, and it is generally true to not only neural processes but also to all the other approaches. Meanwhile, the authors did not talk about the relationship quantitatively between the representability and the error tolerance in Definition 3.1. In addition, the analysis is limited to only scalar-valued function on a 1D interval. The writing could also be improved.\n\nConcerns:\n- The definition of neural processes in the background section is confusing. Despite the way of defining a map, P is a mathematical object defined by a set of tuples and a map,  meaning that the neural processes are also defined by data. In the original paper, the neural processes were however defined as random functions.\n\n- In the background section, the words say 'some sources define ...'. Could the authors give the sources?\n\n- In Def 3.1, what do the authors mean by 'discrete measurements'?\n\n- In the experiment section, do the authors mean sampling from a Gaussian process by saying GP prior? I don't see a GP plays the role of prior in terms of Bayesian inference.\n\n- The examples given in the experiment section lack quantitative results. It is better for evaluating the reconstruction by showing the posterior or predictive distribution instead of single reconstructions.\n\n- In Sec. 4.2. how did the authors sample regular grid on the 2D plane as y is determined by x. \n\n- Eq.11 is defined in the appendix. Better to use separate numbering.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZvEa9hhrWuy", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses an interesting and timely problem, which is to understand how Neural Processes work to learn a representation of a function space. Offering a closer investigation into a recently introduced framework, this work will likely be of interest to the ICLR community. The work focuses on the 1-dimensional case and tries to analyze the simplest case in a rigorous way, which I think is a good approach in general.\n\nHowever, I have some concerns about the main claims of this paper, as listed below:\n\n- One of the main findings of the paper is an observation that Neural Processes perform a \"frequency decomposition\". However, I think this is an insufficiently supported, and even misleading, over-statement. Indeed, Figure 2 shows that there are different modes dominated by varying characteristic frequencies, where a higher-rank mode shows a more slowly varying feature; but there is no further evidence that the decomposition is actually based on the frequency of the signal. One would get a similar result by simply doing a Principal Component Analysis too. When you say \"frequency decomposition\" it carries a clear mathematical meaning, and it is a much stronger statement than what the paper reports empirically.\n    - That said, I agree that the empirical observations are interesting. Perhaps the observations in the paper's experiments may be better described in a frame of global mode decomposition (CNP) vs. local feature detection (NP)?\n\n- I also think that the claim about the theoretical upper bound on the frequency is overstated, the way it is stated currently. The validity of the statement (Theorem 3.1) really depends on the assumption of uniform sampling, which is mentioned as a note after Theorem 3.1. Of course, I fully agree that it is an important starting step to get rigorous results in simplified conditions. But those conditions should be mentioned as part of the statement, especially when it is highly likely that the conditions are not met in the use case (there is no reason to expect that the x values in the context set is close to uniform). For example, it is possible to encode functions with a localized feature whose (local) frequency is higher than your derived bound, by using more samples around that high-frequency feature.\n\nThis paper will get views, partly because it is actually asking an interesting question, and partly because of the boldness and attractiveness of the claims made. How exciting is it to discover a naturally emerging Fourier transform? Except... that's not exactly what one can say just yet (I think). I believe the authors should either support the paper's claims by further work, or tone down their overall framing — major changes either way. While I think this work is headed to a promising direction, given the concerns described above, I recommend a rejection at this time.\n\n**UPDATE:** I appreciate the authors' responses and the engaged discussion. However, I still think that the claims of the paper are not sufficiently supported by the presented results, and maintain my original rating.", "extra_scores": {}, "extra_text": {}}, {"review_id": "V7dLlnEeZjT", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper tries to analyze the behavior of Neural Processes in the frequency domain and concludes that such Processes can only represent oscillations up to a certain frequency.\n\nWhile drawing a parallel between Neural Processes and signal processes, I think that there is some weakness in the experiments of the paper. In particular, the authors only seem to consider the exponential quadratic kernel to generate examples which would mostly show examples of smooth functions as would sampling Fourier linear combinations.\n\nI am also unsure how this paper could be helpful to our community in its present form as it sheds some light on the inner workings of Neural Processes but only in a very limited practical setting.", "extra_scores": {}, "extra_text": {}}, {"review_id": "KikP9tDQaX8", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The work examines properties of Neural Processes (NP). More precisely, of deterministic NPs and how they for finite-dimensional representations of infinite-dimensional function spaces. NP learn functions f that best represent/fit discrete sets of points in space. Based on signal theoretic aspects of discretisation, authors infer a maximum theoretical upper bond of frequencies of functions f that can be used to represent the points. The bond depends on the latent dimension/representation size and the finite interval spawn by the points. Simulations are computed to test the validity of the upper bond. Authors find that NPs behave like a Fourier Transform and decompose the spectrum of the signal. Since the representation during training learns to represent specific frequencies, NPs can be used as band pass/stop filter.\n\nThe paper is well written, and the basic approach is clearly outlined. The quality of the work and the evaluation are good and support the authors claims. However, it is not fully clear to which extend the claims translate to other data or generalise well. The finding that NPs interpret points in space as signals and implement a frequency decomposition like Fourier/Wavelet transforms seems reasonable. Not sure, however, if an application as filter is ecological in terms of computational complexity. \n\nThe paper provides a strong theoretical foundation of the method and authors support their claims by  empirical stimulation. Also, explainability and more importantly interpretability of how methods generate results is essential. So, the message the paper sends is relevant. However ,the relevance and significance of the findings, and the consequences thereof are not clear.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper analyses the behaviour of Neural Processes in the frequency domain and, in particular, how it suppresses high-frequency components of the input functions. While this is entirely intuitive, the paper adds some theoretical analysis via the Nyquist-Shannon theorem. But the analysis remains too generic and it is not clear it will be of broad interest to the community. 
Decision: Reject