Frequency Decomposition in Neural Processes

Neural Processes are a powerful tool for learning representations of function spaces purely from examples , in a way that allows them to perform predictions at test time conditioned on so-called context observations . The learned representations are finite-dimensional , while function spaces are infinite-dimensional , and so far it has been unclear how these representations are learned and what kinds of functions can be represented . We show that deterministic Neural Processes implicitly perform a decomposition of the training signals into different frequency components , similar to a Fourier transform . In this context , we derive a theoretical upper bound on the maximum frequency Neural Processes can reproduce , depending on their representation size . This bound is confirmed empirically . Finally , we show that Neural Processes can be trained to only represent a subset of possible frequencies and suppress others , which makes them programmable band-pass or band-stop filters . 1 INTRODUCTION . Neural Processes ( Garnelo et al. , 2018a ; b ) are a class of models that can learn a distribution over functions , or more generally a function space . In contrast to many other approaches that do the same , for example Bayesian Neural Networks , Neural Processes learn an explicit representation of such a function space , which allows them to condition their predictions on an arbitrary number of observations that are only available at test time . This representation is finite-dimensional , while function spaces are infinite-dimensional , and so far it has not been understood how they are able to bridge this gap and under what conditions they can successfully do so . Our work reveals how Neural Processes learn to represent infinite-dimensional function spaces in a finite-dimensional space , and in the process describes constraints and conditions that decide what kinds of function spaces can be represented . We begin with an observation that prior art in the context of learning on sets can be reinterpreted from a signal-processing perspective , which allows us to derive a theoretical upper bound on the frequencies , i.e . Fourier components , of functions that can be represented . We subsequently confirm this bound empirically , which suggests that the learned representations should contain a notion of frequency . To further investigate this hypothesis , we continue with a visualization of the learned representations , which reveals that Neural Processes can decompose a function space into different frequency components , essentially finding a representation in Fourier space without any explicit supervision on the representations to elicit such behaviour . As further evidence of this we train Neural Processes to represent only certain frequencies , which results in them suppressing those frequencies that were not observed in the training data . Our contributions can be summarized as follows1 : • We derive a theoretical upper bound on the signal frequency Neural Processes of a given representation size can reconstruct . As we show , the bound is observed either in the expected way—by suppressing high frequencies—or by implicitly limiting the signal interval . • We investigate learned representations qualitatively , presenting evidence that Neural Processes perform a frequency decomposition of the function space , akin to a Fourier transform . This behaviour is not incentivized externally but rather emerges naturally . 1The complete source code to reproduce our experiments is available at https : //github.com/ * * * • We show that by choosing the training distribution appropriately , Neural Processes can be made to represent certain frequencies and suppress others , which turns them into programmable band-pass or band-stop filters . 2 BACKGROUND . Neural Processes ( Garnelo et al. , 2018a ; b ) are maps P : C , X → Y , where C is a set of tuples { ( x , f ( x ) ) } Nc=1 = : ( xc , f ( xc ) ) 2 with arbitrary but positive cardinality N , and f ∈ F : X → Y . C is often called the context , because Neural Processes perform predictions for values xt ∈ X ( t for target ) , conditioned on these points . F is the function space we would like to find a representation of . Note that some sources define function spaces as any set of functions with a shared domain and co-domain , while others require them to be vector spaces as well . We don ’ t concern ourselves with this distinction and further restrict our work to X = Y = R , because it allows us to visualize learned representations . We only look at the original Neural Processes , namely the deterministic Conditional Neural Processes ( CNP ) ( Garnelo et al. , 2018a ) and the variational Neural Processes ( NP ) ( Garnelo et al. , 2018b ) , because newer contributions in the field work in ways that preclude them from being analyzed in the same way . We discuss this further in Section 5 . In CNPs and NPs , the map P is separated into two parts , a so called encoding E : C → Z and a decoding or generating part G : Z , X → Y . Z is referred to as the representation or latent space . To allow Neural Processes to approximate arbitrary3 function spaces F , E and G are typically chosen to be powerful approximators , specifically neural networks , as the name suggests . The defining characteristic of CNPs and NPs is that E encodes individual pairs ( x , f ( x ) ) from the context separately , and the resulting representations are averaged to form a global representation , meaning one that is independent of the target points xt at which we then evaluate the Neural Process . This is often not the case in later work , for example in Attentive Neural Processes ( Kim et al. , 2019 ) , where the individual representations are instead aggregated using an attention mechanism that depends on xt . In CNPs the representations are deterministic , while in NPs they parametrize mean and ( log- ) variance of a Gaussian distribution , so the latter are trained using variational inference . For details on implementation and training we refer to Appendix A.1 . Our work will investigate how these global representations , which are finite-dimensional , represent infinite-dimensional function spaces . As stated above , E and by extension the Neural Process P acts on set-valued inputs . This is contrary to the vast majority of machine learning work where inputs are vectors of fixed dimension and ordering . Recall that sets are permutation invariant , so we must ensure that the same is true for the output of E. It is easy to see that this is given when we average individual encodings , but Zaheer et al . ( 2017 ) show that it is in fact the only way to ensure it : E is permutation-invariant if and only if it has a so-called sum-decomposition , i.e . it can be represented in the form E ( x ) = ρ ( N∑ i=1 φ ( xi ) ) ( 1 ) where ρ , φ are appropriately chosen functions . Wagstaff et al . ( 2019 ) further show that to be able to represent all continuous permutation-invariant functions on sets with a cardinality of at most N , the dimension of the image Z must at least be N . This will become relevant in the following section . 3 AN UPPER BOUND ON SIGNAL FREQUENCIES . We mentioned in the previous section that the encoder E in a Neural Process should have a sumdecomposition , so that the global representations are permutation-invariant , as shown in Zaheer et al . ( 2017 ) . Expanding on this , Wagstaff et al . ( 2019 ) show that we require a representation size of at least N to be able to represent arbitrary continuous functions on sets of cardinality smaller or equal to N . What these works do not consider are the implications for situations where the elements of 2We use boldface as a shorthand for sets , not vectors . 3This will depend on the implementation of E and G , and for neural networks F is practically restricted to continuous and differentiable functions . the sets are input-output tuples of some function f , as it is typically the case in Neural Processes . We will use these previous findings to derive an upper bound on the frequencies ν any f ∈ F may contain so that they can be represented in a Neural Process . In order to do this , we must first define what it means to successfully learn a representation of a function space . Definition 3.1 ( Representation of Function Spaces in Neural Processes ) . We say that a Neural Processes P has learned a representation of a function space F , defined on an interval [ a , b ] ⊂ R , if , for some error tolerance , it holds for all x ∈ [ a , b ] and for all f ∈ F , represented as a suitable set of discrete measurements ( xf , f ( xf ) ) , that |P ( ( xf , f ( xf ) ) , x ) − f ( x ) | < . That means the learned representation must be such that we can encode a particular element of the function space f into it and are able to reconstruct it up to a predefined error tolerance . The choice of this tolerance is essentially arbitrary , but should reflect that for g /∈ F the reconstructions should generally not be accurate within . We also write that f is represented as a suitable set of discrete measurements , by which we mean that it must be possible to reconstruct f from those measurements . Switching to signal-processing terminology , we know that to represent a continuous signal as a set of discrete measurements , we need to sample it at points with a distance of at most τ = 1/ ( 2νmax ) , where νmax is the maximum frequency component of the signal . This is most commonly known as the Nyquist-Shannon sampling theorem ( Whittaker , 1915 ; Kotelnikov , 1933 ; Shannon , 1949 ) . For any finite real interval [ a , b ] , this translates to a number of sampling points N > 2|b − a|νmax . The latter allows us to make a connection to the findings by Wagstaff et al . ( 2019 ) , so that we can deduce an upper bound on the maximum signal frequency Neural Processes with a given representation size can reconstruct . Theorem 3.1 ( Maximum Frequency in Neural Process Representations ) . A Neural Process P with latent dimension Dr can only learn a representation of some function space F defined on a finite interval [ a , b ] ⊂ R if for all f ∈ F with a maximum frequency content νmax , f it holds that : νmax , f < Dr 2|b− a| ( 2 ) Note that this means we should in theory be able to represent any function space that obeys Eq . ( 2 ) to within arbitrarily small . In practice , we will typically have less control over F , and we only find approximate representations . Part of our experiments will test how Neural Processes behave if the signals contain frequencies larger than those allowed by Eq . ( 2 ) . It should also be noted that the Nyquist-Shannon theorem used for the above derivation assumes equidistant sampling points . During training , we work with randomly sampled inputs , but at test time equidistant points are used , as we outline in Appendix A.2 .

Summary: The work examines properties of Neural Processes (NP). More precisely, of deterministic NPs and how they for finite-dimensional representations of infinite-dimensional function spaces. NP learn functions f that best represent/fit discrete sets of points in space. Based on signal theoretic aspects of discretisation, authors infer a maximum theoretical upper bond of frequencies of functions f that can be used to represent the points. The bond depends on the latent dimension/representation size and the finite interval spawn by the points. Simulations are computed to test the validity of the upper bond. Authors find that NPs behave like a Fourier Transform and decompose the spectrum of the signal. Since the representation during training learns to represent specific frequencies, NPs can be used as band pass/stop filter.