CKConv: Continuous Kernel Convolution For Sequential Data

1 INTRODUCTION . Recurrent Neural Networks ( RNNs ) have long governed tasks with sequential data ( Rumelhart et al. , 1985 ; Hochreiter & Schmidhuber , 1997 ; Chung et al. , 2014 ) . Their main ingredient are recurrent units : network components with a recurrence formulation which grants RNNs the ability to be unrolled for arbitrarily many steps and handle sequences of arbitrary size . In practice , however , the effective memory horizon of RNNs , i.e. , the number of steps the network retains information from , has proven to be surprisingly small , most notably due to the vanishing gradients problem ( Hochreiter , 1991 ; Bengio et al. , 1994 ) . Interstingly , it is the very recurrent nature of RNNs that allows them to be unrolled for arbitrarily many steps which is responsible for vanishing gradients ( Pascanu et al. , 2013b ) . This , in turn , hinders learning from the far past and induces a small effective memory horizon . Convolutional Neural Networks ( CNNs ) ( LeCun et al. , 1998 ) have proven a strong alternative to recurrent architectures as long as relevant input dependencies fall within their memory horizon , e.g. , Conneau et al . ( 2016 ) ; Oord et al . ( 2016 ) ; Dai et al . ( 2017 ) ; Dauphin et al . ( 2017 ) ; Bai et al . ( 2018a ) . CNNs avoid the training instability and vanishing / exploding gradients characteristic of RNNs by avoiding back-propagation through time ( Werbos , 1990 ) altogether . However , these architectures model convolutional kernels as a sequence of independent weights . As a result , their memory horizon must be defined a-priori , and larger memory horizons induce a proportional growth of the model size . In this work , we provide a solution to these limitations . We propose to view a convolutional kernel as a continuous function parameterized by a small neural network instead of a sequence of independent weights . The resulting Continuous Kernel Convolution ( CKConv ) enjoys the following properties : • CKConvs can define arbitrarily large memory horizons within a single operation . Consequently , Continuous Kernel Convolutional Neural Networks ( CKCNNs ) detach their memory horizon from ( i ) the depth of the network , ( ii ) the dilation factors used , and ( iii ) the size of the network . • CKConvs do not rely on any form of recurrence . As a result , CKCNNs ( i ) can be trained in parallel , and ( ii ) do not suffer from vanishing / exploding gradients or small effective memory horizons . • Continuous convolutional kernels can be evaluated at arbitrary positions . Consequently , CKConvs and CKCNNs can be readily used on irregularly sampled data , and data at different resolutions . We observe that continuous kernel parameterizations previously used to handle irregular data locally , e.g. , Schütt et al . ( 2017 ) ; Wu et al . ( 2019 ) , are not adequate to model long-term dependencies . This is due to the inability of their kernels to model long spatial complex functions ( Sec . 4.2 ) . Contrarily , CKConvs perfectly describe long complex non-linear , non-smooth functions by parameterizing their kernels as SIRENs ( Sitzmann et al. , 2020 ) : implicit neural representations with Sine nonlinearities . Shallow CKCNNs match or outperform state-of-the-art approaches on several tasks comprising stress tests , continuous , discrete and irregular data , as well as resolution changes . To the best of our knowledge , we are first to observe the potential of continuous convolutional kernels to model long-term dependencies , and to provide an useful parameterization to this end . 2 RELATED WORK . Continuous kernel formulation . Continuous formulations for convolutional kernels were introduced to handle irregularly sampled 3D data locally ( Schütt et al. , 2017 ; Simonovsky & Komodakis , 2017 ; Wang et al. , 2018 ; Wu et al. , 2019 ) . As discrete convolutions learn independent weights for specific relative positions , they can not handle irregularly sampled data effectively . Following work focuses on point-cloud applications ( Fuchs et al. , 2020 ; Hu et al. , 2020 ; Shi et al. , 2019 ; Thomas et al. , 2018 ) . Other approaches include Monte Carlo approximations of continuous operations ( Finzi et al. , 2020 ) . Our work proposes a new broad flavor of applications for which continuous kernels are advantageous . Implicit neural representations . Implicit neural representations construct continuous data representations by encoding the input in the weights of a neural network ( Mescheder et al. , 2019 ; Park et al. , 2019 ; Sitzmann et al. , 2020 ) . This leads to numerous advantages over conventional ( discrete ) data representations , e.g. , memory efficiency , analytic differentiability , with interesting properties for several applications , e.g. , generative modelling ( Dupont et al. , 2021 ; Schwarz et al. , 2020 ) . Since we model convolutional kernels as continuous functions and parameterize them via neural networks , our approach can be understood as implicitly representing the convolutional kernels of a conventional CNN . Different is the fact that these convolutional kernels are not known a-priori , but learned as a part of the optimization task of the CNN . Making the connection between implicit neural representations and continuous kernel formulations explicitly brings substantial insights for the construction of these kernels . In particular , it motivates the use of Sine nonlinearities ( Sitzmann et al. , 2020 ) to parameterize them , which leads to significant improvements over the ReLU , LeakyReLU , and Swish nonlinearities used so far for this purpose ( Sec . 4.2 ) . 3 THE CONVOLUTION AND COMMON KERNEL PARAMETERIZATIONS . Notation . [ n ] denotes the set { 0,1 , . . . , n } . Bold capital and lowercase letters depict vectors and matrices , e.g. , x , W , sub-indices index vectors , e.g. , x= { xc } Ninc=1 , parentheses index time , e.g. , x ( τ ) is the value of x at time-step τ , and calligraphic letters depict sequences , e.g. , X= { x ( τ ) } NXτ=0 . Centered and causal convolutions . Let x ∶ R → RNin and ψ ∶ R → RNin be a vector valued signal and kernel on R , such that x= { xc } Ninc=1 and ψ= { ψc } Nin c=1 . The convolution is defined as : ( x ∗ψ ) ( t ) = Nin ∑ c=1 ∫ R xc ( τ ) ψc ( t − τ ) dτ . ( 1 ) In practice , the input signal x is gathered via some sampling procedure . Resultantly , the convolution is effectively performed between the sampled input signal described as a sequence of finite length X= { x ( τ ) } NXτ=0 and a convolutional kernel K= { ψ ( τ ) } NX τ=0 described the same way : ( x ∗ψ ) ( t ) = Nin ∑ c=1 NX/2 ∑ τ=−NX/2 xc ( τ ) ψc ( t − τ ) . ( 2 ) Values x ( τ ) falling outside of X are padded by a constant value often defined as zero ( Fig . 2a ) . The convolutional kernel is commonly centered around the point of calculation t. For sequence modeling this can be undesirable as future input values { x ( t − τ ) } −1τ=−NX/2 are considered during the operation . This is solved by providing a causal formulation to the convolution : a formulation in which the convolution at time-step t only depends on input values at time-steps ( t − τ ) ≤ t ( Fig . 2b ) : ( x ∗ψ ) ( t ) = Nin ∑ c=1 t ∑ τ=0 xc ( τ ) ψc ( t − τ ) . ( 3 ) In practice , causal convolutions are easily implemented via asymmetrical padding . In this work , we consider causal convolutions as default . Nevertheless , our analyses are also valid for centered ones . Discrete convolutional kernels . By a large margin , most convolutional kernels ψ in literature are parameterized as a finite sequence of NK + 1 independent learnable weights K= { ψ ( τ ) } NKτ=0 ( Fig . 2 ) . As these weights are independent of one another , NK must be kept small to keep the parameter count of the model tractable . Hence , the kernel size is often much smaller than the input length : NK ≪ NX . This parameterization presents important limitations : • The memory horizon NK must be defined a priori . • Since NK ≪ NX , this parameterization implicitly assumes that the convolution ( x∗ψ ) at position t only depends on input values at positions up to τ=NK steps in the past . Consequently , no functions depending on inputs x ( t − τ ) for τ > NK can be modeled . • The most general selection of NK is given by a global memory horizon : NK=NX . Unfortunately , as discrete convolutional kernels are modeled as a sequence of independent weights , this incurs an extreme growth of the model size and rapidly becomes statistically unfeasible . Dilated convolutional kernels . To alleviate these limitations , previous works propose to interleave kernel weights with zeros in order to cover larger memory horizons without additional weights ( Fig . 2c ) . This formulation alleviates some of the previous limitations , but introduces additional ones : • Dilated kernels are unable to model dependencies of input values falling in the interleaved regions . • Several authors use dilated convolutions with varying dilation factors as a function of depth , e.g. , ( Bai et al. , 2018a ; Dai et al. , 2017 ; Oord et al. , 2016 ; Romero et al. , 2020 ) . By carefully selecting layer-wise dilation factors , one can assure that some kernel hits each input within the memory horizon of the network . However , due to the extreme sparsity of the formulation , it is difficult to estimate the effective amount of processing applied to the input . In addition , this layout ties together ( i ) the memory horizon , ( ii ) the depth , and ( iii ) the layer-wise dilation factors of the network , which effectively constraints the flexibility of the neural architecture design . In contrast to the ( dilated ) discrete convolutions presented in this section , our proposed formulation allows handling arbitrarily long sequences with arbitrarily large , dense memory horizons in a single layer and under a fixed parameter budget . 4 CONTINUOUS KERNEL CONVOLUTION . In this section , we introduce our approach . First , we define it formally , analyze its properties , illustrate its connection to recurrent units , and elaborate on the functional family they can describe . Next , we discuss concrete parameterizations of continuous convolutional kernels , illustrate their connection to implicit neural representations , and show that our final kernels are able to fit complex functions . 4.1 FORMULATION AND PROPERTIES . Arbitrarily large convolutional kernels . We formulate the convolutional kernel ψ as a continuous vector-valued function parameterized by a small neural network MLPψ ∶ R→ RNout×Nin ( Fig . 1 , left ) . MLPψ receives a relative position ( t−τ ) and outputs the value of the convolutional kernel at that position ψ ( t−τ ) . As a result , an arbitrarily large convolutional kernel K= { ψ ( t−τ ) } NKτ=0 can be constructed by providing an equally large sequence of relative positions { t−τ } NKτ=0 to MLPψ . For NK=NX , the size of the resulting kernel is equal to that of the input sequence X , and thus it is able to model ( global ) long-term dependencies . The Continuous Kernel Convolution ( CKConv ) is given by : ( x ∗ψ ) ( t ) = Nin ∑ c=1 t ∑ τ=0 xc ( τ ) MLPψc ( t − τ ) . ( 4 ) Irregularly sampled data . CKConvs are able to handle irregularly-sampled and partially observed data . To this end , it is sufficient to sample MLPψ at positions for which the input signal is known and perform the convolution operation with the sampled kernel . For very non-uniformly sampled inputs , an inverse density function over the samples can be incorporated in order to provide an unbiased estimation of the convolution response ( see Appx . A.1 , Wu et al . ( 2019 ) for details ) . Data at different resolutions . CKConvs can also process data at different resolutions . Consider the convolution ( x ∗ψ ) sr1 between an input signal x and a continuous convolutional kernel ψ sampled at a sampling rate sr1 . Now , if the convolution receives the same input signal sampled at a different sampling rate sr2 , it is sufficient to sample the convolutional kernel at the sampling rate sr2 in order to perform an “ equivalent ” operation : ( x ∗ψ ) sr2 . As shown in Appx . A.2 , it holds that : ( x ∗ψ ) sr2 ( t ) ≈ sr2 sr1 ( x ∗ψ ) sr1 ( t ) . ( 5 ) That is , convolutions calculated at different resolutions sr1 and sr2 are approximately equal up to a factor given by the resolution change . As a result , CKCNNs ( i ) can be trained in datasets with data at varying resolutions , and ( ii ) can be deployed at resolutions other than those seen during training . We note that the previous features are hardly attainable by regular architectures , with an exception being RNNs with continuous-time interpretations , e.g. , Gu et al . ( 2020a ) ; Kidger et al . ( 2020 ) . ( Linear ) recurrent units are continuous kernel convolutions . Consider a recurrent unit of the form : h ( τ ) = σ ( Wh ( τ − 1 ) +Ux ( τ ) ) ( 6 ) ỹ ( τ ) = softmax ( Vh ( τ ) ) , ( 7 ) where U , W , V depict the input-to-hidden , hidden-to-hidden and hidden-to-output connections of the unit , h ( τ ) , ỹ ( τ ) the hidden representation and the output at time-step τ , and σ a pointwise nonlinearity . As shown in Appx . A.3 , we can express the hidden representation h of a linear recurrent unit , i.e. , with σ=Id , as a convolution between the input x and a convolutional kernel ψ ( τ ) =WτU of size equal to the input . That is , as a continuous kernel convolution with an exponentially increasing or decreasing kernel ( Fig . 3 ) . Different authors show that nonlinear recurrent units are also restricted to the same functional family ( Pascanu et al. , 2013b ; Arjovsky et al. , 2016 ; Zhao et al. , 2020 ) . The functional family of continuous kernel convolutions . From the previous observation , we can conclude that CKConvs are not only more general than discrete convolutions , but that the functional family they describe is also more general than that of ( linear ) recurrent units ( Fig . 3 ) . 4.2 THE CONTINUOUS CONVOLUTIONAL KERNEL MLPψ Convolutional kernels as point-wise MLPs . Let { ∆τi= ( t−τi ) } Ni=0 be a sequence of relative positions . The continuous vector-valued convolutional kernel ψ ∶ R→ RNout×Nin is parameterized as a neural network MLPψwhich maps each relative position ∆τi to the value of the convolutional kernel at that position ( Fig . 1 , left ) . We refer to the nonlinearity used in MLPψas σ . What kind of kernels can MLPψgenerate ? Our method relies on the assumption that the neural network MLPψis able to model complex dependencies densely among all elements within the memory horizon . That is , it assumes that MLPψis able to generate arbitrary convolutional kernels . To test this hypothesis , we fit existing MLP parameterizations , i.e. , with ReLU , LeakyReLU and Swish nonlinearities , to long target functions of varying level of smoothness and non-linearity ( Fig . 5 ) . We observe that existing parameterizations can approximate simple functions , e.g. , Gaussian , step functions , but for increasing levels of non-linearity and non-smoothness , they fail by a large margin . For our analysis , this means that CKConvs with ReLU , LeakyReLU and Swish parameterizations are not able to represent complex input dependencies . In our ablation studies ( Appx . D ) we verify that CKCNNs with these kernels consistently perform worse than our proposed parameterization . Convolutional kernels as implicit neural representations . We notice that parameterizing a convolutional kernel with a neural network is equivalent to constructing an implicit neural representation of the kernel , with the subtle difference that our target objective is not known a-priori , but learned as part of the optimization task of the CNN . Implicit neural representations study generic ways to represent data living in low-dimensional spaces , e.g. , R2 , via neural networks , and thus , despite this difference , constitute an interesting starting point for the parameterization of continuous convolutional kernels . In particular , recent works noticed that neural networks with piece-wise activation functions are unable to model high-frequencies . To alleviate this limitation , they introduce random Fourier features ( Tancik et al. , 2020 ) and Sine nonlinearities ( Sitzmann et al. , 2020 ) . Based on these observations , we repeat the fitting experiment for a SIREN ( Sitzmann et al. , 2020 ) : a MLP with hidden layers of the form y = Sine ( ω0 [ Wx + b ] ) . That is with Sine nonlinearities , and a non-learnable value ω0 that serves as a prior for the oscillations of the output . We observe that a SIREN quickly approximates all target functions to near perfection regardless of their grade of smoothness or nonlinearity . Even a sequence of random noise . This implies that , contrary to other parameterizations , CKConvs with SIREN kernels have the ability to model complex input dependencies across large memory horizons . Our experimental results verify this statement . Our ablation studies in Appx . D show that SIREN kernels consistently outperform all other variants . In addition , our experimental results in Sec . 5 show that shallow CKCNNs with SIREN kernels achieve state-of-the-art across datasets of different nature , i.e. , with continuous and discrete data . The success of Sine nonlinearites : A spline basis interpretation . Sitzmann et al . ( 2020 ) motivates the usage of Sine nonlinearities for implicit neural representations . However , there is no clear understanding as of why Sine nonlinearities are better suited for this task than ( smooth ) piece-wise nonlinearities . For the interested reader , we provide an interpretation to this phenomenon from a spline function approximation perspective in Appx . B . Of most practical relevance from this analysis is the observation that proper initialization of the network parameters , particularly of the bias term { b ( l ) } , is important to create a well-spread set of basis functions suited for function approximation . For SIRENs , this is achieved by initializing the bias term uniformly across the period of each of the Sine components : bi ∼ U ( −π∥Wi , ∶∥−1 , π∥Wi , ∶∥−1 ) . We observe that this initialization leads to better results and faster convergence for all tasks considered .

Summary: The paper introduces convolutions with continuously parametrized kernels for sequential data. Continuous kernel convolutions, CKConv for short, parametrize the kernel associated to a convolutional layer as a continuous mapping, $\psi: \mathbb{R}^+ \rightarrow \mathbb{R}^{N_{out} \times N_{in}}$, from the relative time offset $\Delta\tau \in \mathbb{R}^+$ to a weights matrix $\mathbf{W} \in \mathbb{R}^{N_{out} \times N_{in}}$, i.e. $\psi(\Delta \tau) = \mathbf{W}$. This is in contrast to traditional CNN approaches, which represent the convolution kernel as an explicitly learnable set of weights over an a-priori fixed receptive field.  The corresponding CKConv operation can be applied 1) over arbitrarily long receptive fields, including a global one over the history of the input sequence, without the heavy memory burden incurred by representing the convolution kernel explicitly as a set of weights along the horizon; 2) to sequences that are sampled irregularly or asynchronously, since the admissible set of relative positions are not fixed in advance.  A relation is given, which shows approximately how the CKConv changes under resampling the input sequence (the output gets rescaled by the ratio of the sampling rates), which is suggested by the authors to support well-behavedness under varying input resolutions during training and/or between training and testing. Afterwards, a discussion is given about the parametrization of the kernel. Specifically, various MLP models are considered with different activation functions, such as, ReLU, LeakyReLU and Swish, and it is demonstrated that these MLPs are unable to learn functions with high-frequency oscillations, while a recently introduced Sine activation is able to.   The experimental evaluation includes 1) classic stress tests for RNNs, 2) discrete sequence tasks (sMNIST, pMNIST, sCIFAR10), 3) time series modelling (CharacterTrajectories, Speech Commands) , where additionally robustness to missing/irregularly-sampled data and resolution changes are also investigated. The proposed model overall performs favourably to common alternatives; and it seems robust with respect to missing data, and moderate resolution changes.