Probabilistic Numeric Convolutional Neural Networks

1 INTRODUCTION . Standard convolutional neural networks are defined on a regular input grid . For continuous signals like time series and images , these elements correspond to regular samples of an underlying function f defined on a continuous domain . In this case , the standard convolutional layer of a neural network is a numerical approximation of a continuous convolution operator A. Coherently defined networks on continuous functions should only depend on the input function f , and not on spurious shortcut features ( Geirhos et al. , 2020 ) such as the sampling locations or sampling density , which enable overfitting and reduce robustness to changes in the sampling procedure . Each application of A in a standard neural network incurs some discretization error which is determined by the sampling resolution . In some sense , this error is unavoidable because the features f ( ` ) at the layers ` depend on the values of the input function f at regions that have not been observed . For input signals which are sampled at a low resolution , or even sampled irregularly such as with the sporadic measurements of patient vitals data in ICUs or dispersed sensors for measuring ocean currents , this discretization error can not be neglected . Simply filling in the missing data with zeros or imputing the values is not sufficient since many different imputations are possible , each of which can affect the outcomes of the network . Probabilistic numerics is an emergent field that studies discretization errors in numerical algorithms using probability theory ( Cockayne et al. , 2019 ) . Here we build upon these ideas to quantify the dependence of the network on the regions in the input which are unknown , and integrate this uncertainty into the computation of the network . To do so , we replace the discretely evaluated feature maps { f ( ` ) ( xi ) } Ni=1 with Gaussian processes : distributions over the continuous function f ( ` ) that track the most likely values as well as the uncertainty . On this Gaussian process feature representation , we need not resort to discretizing the convolution operator A as in a standard convnet , but instead we can apply the continuous convolution operator directly . If a given feature is a Gaussian process , then applying linear operators yields a new Gaussian process with transformed mean and covariance functions . The dependence of Af on regions of f which are not known translates into ∗Work done during internship at Qualcomm AI Research †Qualcomm AI Research is an initiative of Qualcomm Technologies , Inc. the uncertainty represented in the transformed covariance function , the analogue of the discretization error in a CNN , which is now tracked explicitly . We call the resulting model Probabilistic Numeric Convolutional Neural Network ( PNCNN ) . 2 RELATED WORK . Over the years there have been many successful convolutional approaches for ungridded data such as GCN ( Kipf and Welling , 2016 ) , PointNet ( Qi et al. , 2017 ) , Transformer ( Vaswani et al. , 2017 ) , Deep Sets ( Zaheer et al. , 2017 ) , SplineCNN ( Fey et al. , 2018 ) , PCNN ( Atzmon et al. , 2018 ) , PointConv ( Wu et al. , 2019 ) , KPConv ( Thomas et al. , 2019 ) and many others ( de Haan et al. , 2020 ; Finzi et al. , 2020 ; Schütt et al. , 2017 ; Wang et al. , 2018 ) . However , the target domains of sets , graphs , and point clouds are intrinsically discrete and for continuous data each of these methods fail to take full advantage of the assumption that the underlying signal is continuous . Furthermore , none of these approaches reason about the underlying signal probabilistically . In a separate line of work there are several approaches tackling irregularly spaced time series with RNNs ( Che et al. , 2018 ) , Neural ODEs ( Rubanova et al. , 2019 ) , imputation to a regular grid ( Li and Marlin , 2016 ; Futoma et al. , 2017 ; Shukla and Marlin , 2019 ; Fortuin et al. , 2020 ) , set functions ( Horn et al. , 2019 ) and attention ( Shukla and Marlin , 2020 ) . Additionally there are several works exploring reconstruction of images from incomplete observations for downstream classification ( Huijben et al. , 2019 ; Li and Marlin , 2020 ) . Most similar to our method are the end-to-end Gaussian process adapter ( Li and Marlin , 2016 ) and the multi-task Gaussian process RNN classifier ( Futoma et al. , 2017 ) . In these two works , a Gaussian process is fit to an irregularly spaced time series and sampled imputations from this process are fed into a separate RNN classifier . Unlike our approach where the classifier operates directly on a continuous and probabilistic signal , in these works the classifier operates on a deterministic signal on a regular grid and can not reason probabilistically about discretization errors . Finally , while superficially similar to Deep GPs ( Damianou and Lawrence , 2013 ) or Deep Differential Gaussian Process Flows ( Hegde et al. , 2018 ) , our PNCNNs tackle fundamentally different kinds of problems like image classification1 , and our GPs represent epistemic uncertainty over the values of the feature maps rather than the parameters of the network . 3 BACKGROUND . Probabilistic Numerics : We draw inspiration for our approach from the community of probabilistic numerics where the error in numerical algorithms are modeled probabilistically , and typically with a Gaussian process . In this framework , only a finite number of input function calls can be made , and therefore the numerical algorithm can be viewed as an autonomous agent which has epistemic uncertainty over the values of the input . A well known example is Bayesian Monte Carlo where a Gaussian process is used to model the error in the numerical estimation of an integral and optimally select a rule for its computation ( Minka , 2000 ; Rasmussen and Ghahramani , 2003 ) . Probabilistic numerics has been applied widely to numerical problems such as the inversion of a matrix ( Hennig , 2015 ) , the solution of an ODE ( Schober et al. , 2019 ) , a meshless solution to boundary value PDEs ( Cockayne et al. , 2016 ) , and other numerical problems ( Cockayne et al. , 2019 ) . To our knowledge , we are the first to construct a probabilistic numeric method for convolutional neural networks . Gaussian Processes : We are interested in operating on the continuous function f ( x ) underlying the input , but in practice we have access only to a collection of the values of that function sampled on a finite number of points { xi } Ni=1 . Classical interpolation theory reconstructs f deterministically by assuming a certain structure of the signal in the frequency domain . Gaussian processes give a way of modeling our beliefs about values that have not been observed ( Rasmussen et al. , 2006 ) , as reviewed in appendix A . These beliefs are encoded into a prior covariance k of the GP f ∼ GP ( 0 , k ) and updated upon seeing data with Bayesian inference . Explicitly , given a set of sampling locations x = { xi } Ni=1 and noisy observations y = { yi } Ni=1 sampled yi ∼ N ( f ( xi ) , σ2i ) , using Bayes rule 1While GPs can be applied directly to image classification , they are not well suited to this task even with convolutional structure baked in , as shown in Kumar et al . ( 2018 ) . Input Data one can compute the posterior distribution f |y , x ∼ GP ( µp , kp ) , which captures our epistemic uncertainty about the values between observations . The posterior mean and covariance are given by µp ( x ) = k ( x ) > [ K + S ] −1y , kp ( x , x ′ ) = k ( x , x′ ) − k ( x ) > [ K + S ] −1k ( x′ ) , ( 1 ) where Kij = k ( xi , xj ) , k ( x ) i = k ( x , xi ) and S = diag ( σ2i ) . Below we shall choose the RBF kernel2 as prior covariance , due to its convenient analytical properties : kRBF ( x , x′ ) = aN ( x ; x′ , l2I ) = a ( 2πl2 ) − d2 exp ( − 12l2 ||x− x′||2 ) . In typical applications of GPs to machine learning tasks such as regression , the function f that we want to predict is already the regression model . In contrast , here we use GPs as a way of representing our beliefs and epistemic uncertainty about the values of both the input function and the intermediate feature maps of a neural network . 4 PROBABILISTIC NUMERIC CONVOLUTIONAL NEURAL NETWORKS . 4.1 OVERVIEW . Given an input signal f : X → Rc , we define a network with layers that act directly on this continuous input signal . We define our neural network recursively from the input f ( 0 ) = f , as a series of L continuous convolutions A ( ` ) with pointwise ReLU nonlinearities and weight matrices which mix only channels ( known as 1× 1 convolutions ) M ∈ Rc×c : f ( ` +1 ) = M ( ` ) ReLU [ A ( ` ) f ( ` ) ] , ( 2 ) and a final global average pooling layer P which acts channel-wise as natural generalization of the discrete case : P ( f ( L ) ) α = ∫ f ( L ) α ( x ) dx for each α = 1 , 2 , . . . , c. Denoting the space of functions on X with c channels byHc , the convolution operatorsA ( ` ) are linear operators fromHc ` toHc ` +1 . Like in ordinary convolutional neural networks , the layers build up increasingly more expressive spatial features and depend on the parameters in A ( ` ) and M ( ` ) . Unlike ordinary convolutional networks , these layers are well defined operations on the underlying continuous signal . While it is clear that such a network can be defined abstractly , the exact values of the function f ( L ) can not be computed as the operators depend on unknown values of the input . However , by adapting a probabilistic description we can formulate our ignorance of f ( 0 ) with a Gaussian process and see how the uncertainties propagate through the layers of the network , yielding a probabilistic output . Before delving into details , we outline the key components of equation 2 that make this possible . Continuous Convolutional Layers : Crucially , we consider continuous convolution operators A that can be applied to input Gaussian process f ∼ GP ( µp , kp ) in closed form . The output is another Gaussian process with a transformed mean and covariance Af ∼ GP ( Aµp , AkpA′ ) where A′ acts 2For convenience , we include the additional scale factor ( 2πl2 ) d/2 relative to the usual definition . to the left on the primed argument of kp ( x , x′ ) .3 In section 4.2 we show how to parametrize these continuous convolutions in terms of the flow of a PDE and show how they can be applied to the RBF kernel exactly in closed form . Probabilistic ReLUs : Applying the ReLU nonlinearity to the GP yields a new non Gaussian stochastic process h ( ` ) = ReLU [ A ( ` ) f ( ` ) ] , and we show in section 4.5 that the mean and covariance of this process has a closed form solution which can be computed . Channel Mixing and Central Limit Theorem : The activations h ( ` ) are not Gaussian ; however , for a large number of weakly dependent channels we argue that f ( ` +1 ) = M ( ` ) h ( ` ) is approximately distributed as a Gaussian Process in section 4.5 . Measurement and Projection to RBF Gaussian Process : While f ( ` +1 ) is approximately a Gaussian process , the mean and covariance functions have a complicated form . Instead of using these functions directly , we take measurements of the mean and variance of this process and feed them in as noisy observations to a fresh RBF kernel GP , allowing us to repeat the process and build up multiple layers without increasing complexity . The Gaussian process feature maps in the final layer f ( L ) are aggregated spatially by the integral pooling P which can also be applied in closed form ( see appendix D ) , to yield a Gaussian output . Assembling these components , we implement the end to end trainable Probabilistic Numeric Convolutional Neural Network which integrates a probabilistic description of missing data and discretization error inherent to continuous signals . The layers of the network are shown in figure 1 .

Summary: This work presents an uncertainty aware continuous convolutional layers for learning from continuous signals like time series/images. This work is most useful in the setting of irregularly sampled data. Gaussian processes (GP) are used to represent the irregularly sampled input. The proposed continuous convolutional layers can be directly applied to input Gaussian Process in a closed form, which subsequently outputs another GP with transformed mean and variance. Finally, the continuous convolutional layers are parameterized in terms of the flow of a PDE. The use of GP for feature representation and the ability of continuous convolutional layers to take GP as input provides the model with the ability to propagate uncertainty.