CKCONV: CONTINUOUS KERNEL CONVOLUTION FOR SEQUENTIAL DATA
Year: 2021

Abstract:
Conventional neural architectures for sequential data present important limitations. Recurrent neural networks suffer from exploding and vanishing gradients, small effective memory horizons, and must be trained sequentially. Convolutional neural networks cannot handle sequences of unknown size and their memory horizon must be defined a priori. In this work, we show that these problems can be solved by formulating the convolutional kernels of CNNs as continuous functions. The resulting Continuous Kernel Convolution (CKConv) handles arbitrarily long sequences in a parallel manner, within a single operation, and without relying on any form of recurrence. We show that Continuous Kernel Convolutional Networks (CKCNNs) obtain state-of-the-art results in multiple datasets, e.g., permuted MNIST, and, thanks to their continuous nature, are able to handle non-uniformly sampled datasets and irregularly-sampled data natively. CKCNNs match or perform better than neural ODEs designed for these purposes in a faster and simpler manner.

heading: 1 INTRODUCTION
text: Recurrent Neural Networks (RNNs) have long governed tasks with sequential data (Rumelhart et al., 1985; Hochreiter & Schmidhuber, 1997; Chung et al., 2014). Their main ingredient are recurrent units: network components with a recurrence formulation which grants RNNs the ability to be unrolled for arbitrarily many steps and handle sequences of arbitrary size. In practice, however, the effective memory horizon of RNNs, i.e., the number of steps the network retains information from, has proven to be surprisingly small, most notably due to the vanishing gradients problem (Hochreiter, 1991; Bengio et al., 1994). Interstingly, it is the very recurrent nature of RNNs that allows them to be unrolled for arbitrarily many steps which is responsible for vanishing gradients (Pascanu et al., 2013b). This, in turn, hinders learning from the far past and induces a small effective memory horizon.
Convolutional Neural Networks (CNNs) (LeCun et al., 1998) have proven a strong alternative to recurrent architectures as long as relevant input dependencies fall within their memory horizon, e.g., Conneau et al. (2016); Oord et al. (2016); Dai et al. (2017); Dauphin et al. (2017); Bai et al. (2018a). CNNs avoid the training instability and vanishing / exploding gradients characteristic of RNNs by avoiding back-propagation through time (Werbos, 1990) altogether. However, these architectures model convolutional kernels as a sequence of independent weights. As a result, their memory horizon must be defined a-priori, and larger memory horizons induce a proportional growth of the model size.
In this work, we provide a solution to these limitations. We propose to view a convolutional kernel as a continuous function parameterized by a small neural network instead of a sequence of independent weights. The resulting Continuous Kernel Convolution (CKConv) enjoys the following properties:
• CKConvs can define arbitrarily large memory horizons within a single operation. Consequently, Continuous Kernel Convolutional Neural Networks (CKCNNs) detach their memory horizon from (i) the depth of the network, (ii) the dilation factors used, and (iii) the size of the network.
• CKConvs do not rely on any form of recurrence. As a result, CKCNNs (i) can be trained in parallel, and (ii) do not suffer from vanishing / exploding gradients or small effective memory horizons.
• Continuous convolutional kernels can be evaluated at arbitrary positions. Consequently, CKConvs and CKCNNs can be readily used on irregularly sampled data, and data at different resolutions.
We observe that continuous kernel parameterizations previously used to handle irregular data locally, e.g., Schütt et al. (2017); Wu et al. (2019), are not adequate to model long-term dependencies. This is
due to the inability of their kernels to model long spatial complex functions (Sec. 4.2). Contrarily, CKConvs perfectly describe long complex non-linear, non-smooth functions by parameterizing their kernels as SIRENs (Sitzmann et al., 2020): implicit neural representations with Sine nonlinearities. Shallow CKCNNs match or outperform state-of-the-art approaches on several tasks comprising stress tests, continuous, discrete and irregular data, as well as resolution changes. To the best of our knowledge, we are first to observe the potential of continuous convolutional kernels to model long-term dependencies, and to provide an useful parameterization to this end.

heading: 2 RELATED WORK
text: Continuous kernel formulation. Continuous formulations for convolutional kernels were introduced to handle irregularly sampled 3D data locally (Schütt et al., 2017; Simonovsky & Komodakis, 2017; Wang et al., 2018; Wu et al., 2019). As discrete convolutions learn independent weights for specific relative positions, they cannot handle irregularly sampled data effectively. Following work focuses on point-cloud applications (Fuchs et al., 2020; Hu et al., 2020; Shi et al., 2019; Thomas et al., 2018). Other approaches include Monte Carlo approximations of continuous operations (Finzi et al., 2020).
Our work proposes a new broad flavor of applications for which continuous kernels are advantageous.
Implicit neural representations. Implicit neural representations construct continuous data representations by encoding the input in the weights of a neural network (Mescheder et al., 2019; Park et al., 2019; Sitzmann et al., 2020). This leads to numerous advantages over conventional (discrete) data representations, e.g., memory efficiency, analytic differentiability, with interesting properties for several applications, e.g., generative modelling (Dupont et al., 2021; Schwarz et al., 2020).
Since we model convolutional kernels as continuous functions and parameterize them via neural networks, our approach can be understood as implicitly representing the convolutional kernels of a conventional CNN. Different is the fact that these convolutional kernels are not known a-priori, but learned as a part of the optimization task of the CNN. Making the connection between implicit neural representations and continuous kernel formulations explicitly brings substantial insights for the construction of these kernels. In particular, it motivates the use of Sine nonlinearities (Sitzmann et al., 2020) to parameterize them, which leads to significant improvements over the ReLU, LeakyReLU, and Swish nonlinearities used so far for this purpose (Sec. 4.2).

heading: 3 THE CONVOLUTION AND COMMON KERNEL PARAMETERIZATIONS
text: Notation. [n] denotes the set {0,1, . . . , n}. Bold capital and lowercase letters depict vectors and matrices, e.g., x, W, sub-indices index vectors, e.g., x={xc}Ninc=1 , parentheses index time, e.g., x(τ) is the value of x at time-step τ , and calligraphic letters depict sequences, e.g., X={x(τ)}NXτ=0.
Centered and causal convolutions. Let x ∶ R → RNin and ψ ∶ R → RNin be a vector valued signal and kernel on R, such that x={xc}Ninc=1 and ψ={ψc} Nin c=1 . The convolution is defined as:
(x ∗ψ)(t) = Nin
∑ c=1 ∫ R xc(τ)ψc(t − τ)dτ. (1)
In practice, the input signal x is gathered via some sampling procedure. Resultantly, the convolution is effectively performed between the sampled input signal described as a sequence of finite length X={x(τ)}NXτ=0 and a convolutional kernel K={ψ(τ)} NX τ=0 described the same way:
(x ∗ψ)(t) = Nin
∑ c=1
NX/2 ∑
τ=−NX/2 xc(τ)ψc(t − τ). (2)
Values x(τ) falling outside of X are padded by a constant value often defined as zero (Fig. 2a). The convolutional kernel is commonly centered around the point of calculation t. For sequence modeling this can be undesirable as future input values {x(t − τ)}−1τ=−NX/2 are considered during the operation. This is solved by providing a causal formulation to the convolution: a formulation in which the convolution at time-step t only depends on input values at time-steps (t − τ) ≤ t (Fig. 2b):
(x ∗ψ)(t) = Nin
∑ c=1
t
∑ τ=0 xc(τ)ψc(t − τ). (3)
In practice, causal convolutions are easily implemented via asymmetrical padding. In this work, we consider causal convolutions as default. Nevertheless, our analyses are also valid for centered ones.
Discrete convolutional kernels. By a large margin, most convolutional kernels ψ in literature are parameterized as a finite sequence of NK + 1 independent learnable weights K={ψ(τ)}NKτ=0 (Fig. 2). As these weights are independent of one another, NK must be kept small to keep the parameter count of the model tractable. Hence, the kernel size is often much smaller than the input length: NK ≪ NX. This parameterization presents important limitations:
• The memory horizon NK must be defined a priori.
• Since NK ≪ NX, this parameterization implicitly assumes that the convolution (x∗ψ) at position t only depends on input values at positions up to τ=NK steps in the past. Consequently, no functions depending on inputs x(t − τ) for τ > NK can be modeled.
• The most general selection of NK is given by a global memory horizon: NK=NX. Unfortunately, as discrete convolutional kernels are modeled as a sequence of independent weights, this incurs an extreme growth of the model size and rapidly becomes statistically unfeasible.
Dilated convolutional kernels. To alleviate these limitations, previous works propose to interleave kernel weights with zeros in order to cover larger memory horizons without additional weights (Fig. 2c). This formulation alleviates some of the previous limitations, but introduces additional ones:
• Dilated kernels are unable to model dependencies of input values falling in the interleaved regions.
• Several authors use dilated convolutions with varying dilation factors as a function of depth, e.g., (Bai et al., 2018a; Dai et al., 2017; Oord et al., 2016; Romero et al., 2020). By carefully selecting layer-wise dilation factors, one can assure that some kernel hits each input within the memory horizon of the network. However, due to the extreme sparsity of the formulation, it is difficult to estimate the effective amount of processing applied to the input. In addition, this layout ties together (i) the memory horizon, (ii) the depth, and (iii) the layer-wise dilation factors of the network, which effectively constraints the flexibility of the neural architecture design.
In contrast to the (dilated) discrete convolutions presented in this section, our proposed formulation allows handling arbitrarily long sequences with arbitrarily large, dense memory horizons in a single layer and under a fixed parameter budget.

heading: 4 CONTINUOUS KERNEL CONVOLUTION
text: In this section, we introduce our approach. First, we define it formally, analyze its properties, illustrate its connection to recurrent units, and elaborate on the functional family they can describe. Next, we discuss concrete parameterizations of continuous convolutional kernels, illustrate their connection to implicit neural representations, and show that our final kernels are able to fit complex functions.

heading: 4.1 FORMULATION AND PROPERTIES
text: Arbitrarily large convolutional kernels. We formulate the convolutional kernel ψ as a continuous vector-valued function parameterized by a small neural network MLPψ ∶ R→ RNout×Nin (Fig. 1, left). MLPψ receives a relative position (t−τ) and outputs the value of the convolutional kernel at that position ψ(t−τ). As a result, an arbitrarily large convolutional kernel K={ψ(t−τ)}NKτ=0 can be constructed by providing an equally large sequence of relative positions {t−τ}NKτ=0 to MLPψ. For NK=NX, the size of the resulting kernel is equal to that of the input sequence X, and thus it is able to model (global) long-term dependencies. The Continuous Kernel Convolution (CKConv) is given by:
(x ∗ψ)(t) = Nin
∑ c=1
t
∑ τ=0 xc(τ)MLPψc (t − τ). (4)
Irregularly sampled data. CKConvs are able to handle irregularly-sampled and partially observed data. To this end, it is sufficient to sample MLPψ at positions for which the input signal is known and perform the convolution operation with the sampled kernel. For very non-uniformly sampled inputs, an inverse density function over the samples can be incorporated in order to provide an unbiased estimation of the convolution response (see Appx. A.1, Wu et al. (2019) for details).
Data at different resolutions. CKConvs can also process data at different resolutions. Consider the convolution (x ∗ψ)sr1 between an input signal x and a continuous convolutional kernel ψ sampled at a sampling rate sr1. Now, if the convolution receives the same input signal sampled at a different sampling rate sr2, it is sufficient to sample the convolutional kernel at the sampling rate sr2 in order to perform an “equivalent” operation: (x ∗ψ)sr2 . As shown in Appx. A.2, it holds that:
(x ∗ψ)sr2(t) ≈ sr2 sr1 (x ∗ψ)sr1(t). (5)
That is, convolutions calculated at different resolutions sr1 and sr2 are approximately equal up to a factor given by the resolution change. As a result, CKCNNs (i) can be trained in datasets with data at varying resolutions, and (ii) can be deployed at resolutions other than those seen during training.
We note that the previous features are hardly attainable by regular architectures, with an exception being RNNs with continuous-time interpretations, e.g., Gu et al. (2020a); Kidger et al. (2020).
(Linear) recurrent units are continuous kernel convolutions. Consider a recurrent unit of the form: h(τ) = σ(Wh(τ − 1) +Ux(τ)) (6) ỹ(τ) = softmax(Vh(τ)), (7)
where U,W,V depict the input-to-hidden, hidden-to-hidden and hidden-to-output connections of the unit, h(τ), ỹ(τ) the hidden representation and the output at time-step τ , and σ a pointwise nonlinearity. As shown in Appx. A.3, we can express the hidden representation h of a linear recurrent unit, i.e., with σ=Id, as a convolution between the input x and a convolutional kernel ψ(τ)=WτU of
size equal to the input. That is, as a continuous kernel convolution with an exponentially increasing or decreasing kernel (Fig. 3). Different authors show that nonlinear recurrent units are also restricted to the same functional family (Pascanu et al., 2013b; Arjovsky et al., 2016; Zhao et al., 2020).
The functional family of continuous kernel convolutions. From the previous observation, we can conclude that CKConvs are not only more general than discrete convolutions, but that the functional family they describe is also more general than that of (linear) recurrent units (Fig. 3).
4.2 THE CONTINUOUS CONVOLUTIONAL KERNEL MLPψ
Convolutional kernels as point-wise MLPs. Let {∆τi=(t−τi)}Ni=0 be a sequence of relative positions. The continuous vector-valued convolutional kernel ψ ∶ R→ RNout×Nin is parameterized as a neural network MLPψwhich maps each relative position ∆τi to the value of the convolutional kernel at that position (Fig. 1, left). We refer to the nonlinearity used in MLPψas σ.
What kind of kernels can MLPψgenerate? Our method relies on the assumption that the neural network MLPψis able to model complex dependencies densely among all elements within the memory horizon. That is, it assumes that MLPψis able to generate arbitrary convolutional kernels.
To test this hypothesis, we fit existing MLP parameterizations, i.e., with ReLU, LeakyReLU and Swish nonlinearities, to long target functions of varying level of smoothness and non-linearity (Fig. 5). We observe that existing parameterizations can approximate simple functions, e.g., Gaussian, step functions, but for increasing levels of non-linearity and non-smoothness, they fail by a large margin. For our analysis, this means that CKConvs with ReLU, LeakyReLU and Swish parameterizations are not able to represent complex input dependencies. In our ablation studies (Appx. D) we verify that CKCNNs with these kernels consistently perform worse than our proposed parameterization.
Convolutional kernels as implicit neural representations. We notice that parameterizing a convolutional kernel with a neural network is equivalent to constructing an implicit neural representation of the kernel, with the subtle difference that our target objective is not known a-priori, but learned as part of the optimization task of the CNN. Implicit neural representations study generic ways to represent data living in low-dimensional spaces, e.g., R2, via neural networks, and thus, despite this difference, constitute an interesting starting point for the parameterization of continuous convolutional kernels. In particular, recent works noticed that neural networks with piece-wise activation functions are unable to model high-frequencies. To alleviate this limitation, they introduce random Fourier features (Tancik et al., 2020) and Sine nonlinearities (Sitzmann et al., 2020).
Based on these observations, we repeat the fitting experiment for a SIREN (Sitzmann et al., 2020): a MLP with hidden layers of the form y = Sine(ω0[Wx + b]). That is with Sine nonlinearities, and a non-learnable value ω0 that serves as a prior for the oscillations of the output.
We observe that a SIREN quickly approximates all target functions to near perfection regardless of their grade of smoothness or nonlinearity. Even a sequence of random noise. This implies that, contrary to other parameterizations, CKConvs with SIREN kernels have the ability to model complex input dependencies across large memory horizons. Our experimental results verify this statement. Our ablation studies in Appx. D show that SIREN kernels consistently outperform all other variants. In addition, our experimental results in Sec. 5 show that shallow CKCNNs with SIREN kernels achieve state-of-the-art across datasets of different nature, i.e., with continuous and discrete data.
The success of Sine nonlinearites: A spline basis interpretation. Sitzmann et al. (2020) motivates the usage of Sine nonlinearities for implicit neural representations. However, there is no clear understanding as of why Sine nonlinearities are better suited for this task than (smooth) piece-wise nonlinearities. For the interested reader, we provide an interpretation to this phenomenon from a spline function approximation perspective in Appx. B.
Of most practical relevance from this analysis is the observation that proper initialization of the network parameters, particularly of the bias term {b(l)}, is important to create a well-spread set of basis functions suited for function approximation. For SIRENs, this is achieved by initializing the bias term uniformly across the period of each of the Sine components: bi ∼ U(−π∥Wi,∶∥−1, π∥Wi,∶∥−1). We observe that this initialization leads to better results and faster convergence for all tasks considered.

heading: 5 EXPERIMENTS
text: We validate our approach against several existing models and across several tasks selected from the corresponding papers. Specifically, we benchmark its ability to handle long-term dependencies, data at different resolutions and irregularly-sampled data. A complete description of the datasets used as well as additional experiments and ablation studies can be found in the Appendix (Appx. C, D). 1
Network details. We parameterize our convolutional kernels as 3-layer SIRENs. Weight normalization (Salimans & Kingma, 2016) leads to better and faster convergence when applied to the layers in MLP, and we use it across all experiments. All our CKCNNs follow the structure shown in Fig. 8 and vary only in the number of blocks and channels. We use two residual blocks for all experiments reported in this section. Specifications on the architectures and hyperparameters used are given in Appx. E. We speed up the convolution operations in our networks via the convolution theorem: (f ∗ ψ)=F−1{F{f} ⋅F{ψ}}, with F the Fourier transform. Stress experiments. First, we validate that CKCNNs can readily model memory horizons of different lengths. To this end, we evaluate if a shallow CKCNN is able to solve the Copy Memory and Adding Problem tasks (Hochreiter & Schmidhuber, 1997) for sequences of sizes in the range [100,6000]. Success is achieved if 100% accuracy, or a loss ≤ 1e-4 are obtained, for the copy memory and adding problem, respectively. Random predictions for the adding problem lead to a loss of approx. 0.17.
Our results show that a shallow CKCNN is able to solve both problems for all sequence lengths considered without requiring structural modifications (Tab. 1). Recurrent architectures are not able not solve the copy problem at all and could solve the sum problem up to 200 steps. TCNs with k=7, n=7 were able to solve both tasks for up to 1000 steps. However, larger lengths were out of reach as their memory horizon is constrained a priori. To handle larger sequences, TCNs must modify their network structure based on prior knowledge regarding the expected length of the input sequence.
Discrete sequences. The continuous nature of our kernels might give the impression that CKCNNs are only well-suited for continuous data, i.e., time-series. However, we observed that Sine nonlinearities give our convolutional kernels the ability to model complex non-smooth non-linear functions (Fig. 4). Consequently, we validate whether CKCNNs can be applied for discrete sequence modeling on the following tasks: sMNIST, pMNIST (Le et al., 2015) and sCIFAR10 (Trinh et al., 2018).
Shallow CKCNNs outperform recurrent, self-attention and convolutional models on sMNIST and pMNIST (Tab. 2). On sMNIST, a small CKCNN (100K params.) achieves state-of-the-art results
1Our code is publicly available at hidden-repository-due-to-double-blind-review.
Table 1: Evaluation on stress tasks. 3 marks if the problem has been solved.
MODEL SIZE SEQ. LENGTH
100 200 1000 3000 6000
COPY MEMORY GRU 16K - - - - - TCN 16K 3 3 3 - - CKCNN 16K 3 3 3 3 3
ADDING PROBLEM (LOSS) GRU 70K 3 3 - TCN 70K 3 3 3 - - CKCNN 70K 3 3 3 3 3
MODEL DROP PERCENTAGE
(0%) (30%) (50%) (70%)
CHARACTERTRAJECTORIES GRU-ODE 96.2 92.6 86.7 89.9 GRU-∆t 97.8 93.6 91.3 90.4 GRU-D 95.9 94.2 90.2 91.9 ODE-RNN 97.1 95.4 96.0 95.3 NCDE 98.8 98.7 98.8 98.6
CKCNN 99.53 99.30 98.83 98.14
SPEECHCOMMADS_RAW CKCNN 71.66 63.46 60.55 57.50
with a model 80× smaller than the current state-of-the-art. A wider CKCNN (1M params.) slightly increases this result further. On pMNIST, we see an improvement of 0.8% over the best model of size ≤100K, and our wider shallow CKCNN achieves state-of-the-art on this dataset. For sCIFAR10, our small CKCNN obtains similar results to a self-attention model 5× bigger, and our wider variant improves performance by an additional 1%. Our best results are obtained with an even wider model (2.5M params) with which an accuracy of 65.59% is obtained.
Time-series modeling. Next, we evaluate CKCNNs on time-series data. To this end, we consider the CharacterTrajectories (CT) (Bagnall et al., 2018) and the Speech Commands (SC) (Warden, 2018) datasets. We follow Kidger et al. (2020) to obtain a balanced classification dataset with precomputed mel-frequency cepstrum coefficients. In addition, we evaluate the ability of CKCNNs to model long-term dependencies by training on the raw SC dataset (SC_raw), whose records have length 16k.
We compare CKCNNs with representative sequential models with continuous-time interpretations: GRU-ODE (De Brouwer et al., 2019), GRU-∆t (Kidger et al., 2020), ODE-RNN (Rubanova et al., 2019), and NCDE (Kidger et al., 2020). Continuous-time sequential models were selected as they are only sequential methods also able to handle irregularly-sampled data, and data at different resolutions. Our results show that shallow CKCNNs outperform all continuous-time models considered for both the CT and SC datasets (Tab. 3). In addition, CKCNNs obtain promising results on SC_raw, which validates their ability to handle very-long-term dependencies. In fact, CKCNNs trained on SC_raw are able outperform several Neural ODE models trained on the preprocessed data (SC).
In addition, we observed that neural ODE methods considered in Tab. 3 were prohibitively slow for long sequences. For instance, NCDEs were 228× slower than a CKCNN of equivalent size on SC_raw, taking 17 hours per epoch to train. Consequently, training a NCDE on SC_raw for a matching number of epochs would take more than 212 days to conclude. In order to provide results for these models, we train them under the same computational budget than CKCNNs. This is enough to train them for a single epoch. All obtained results are at best only marginally better than random.
Testing at different sampling rates. We now consider the case where a network is trained with data at a sampling rate sr1, and tested with data at a different sampling rate sr2.
Our results show that the performances of CKCNNs remains stable for large sampling rate fluctuations (Tab. 5). This behaviour contrasts with most previous continuous-time models, whose performance rapidly decays upon these changes. CKCNNs outperform HiPPO (Gu et al., 2020a) and set a new

heading: MODEL COMPARISON - CHARACTER TRAJECTORIES

heading: MODEL GRU-D ODE-RNN LMU NCDE HIPPO CKCNN
text: state-of-the-art in this setting. Importantly, depending on the sampling, additional care may be needed to account for spatial displacements and high-frequencies of our kernels (see Appx. E.2 for details).
Irregularly-sampled data. To conclude, we explore the applicability of CKCNNs for irregularlysampled data. To this end, we follow the methodology of Kidger et al. (2020), and drop 30%, 50% and 70% of the data for the CT dataset. An additional channels is added to the input to indicate if values at that position are known. In addition, we provide results under the same methodology for the SC_raw dataset. We omit the SC dataset as the preprocessed bins are calculated with overlapping windows and thus information from dropped points might still be present.
Our results show that CKCNNs exhibit stable performance for varying quantities of missing data, and perform better than several models explicitly developed to this end (Tab. 4). NCDEs perform slightly better than CKCNNs. However, we argue that our method is still advantageous due to its speed gains.

heading: 6 DISCUSSION
text: Parameter-efficient large convolutional kernels. CKConvs construct large complex kernels with a fixed parameter budget. For large input sequences, this results in large savings in the number of parameters required to construct global kernels with conventional CNNs. For sequences from the pMNIST (length = 784) and SC_raw (length = 16000) datasets, a conventional CNN with global kernels would require 2.14M and 46.68M of parameters, respectively, for a model equivalent to our CKCNN (100K). In other words, our kernel parameterization allows us to construct CKCNNs that are 21,84 and 445,71 times smaller than corresponding conventional CNNs for these datasets. Detailed exploration on the effect of our efficient continuous kernel parameterizations in optimization, overfitting and generalization is an interesting direction for future research.
Is depth important? Shallow global memory horizons. Our results are obtained with CKCNNs built with two residual blocks only. Additional experiments (Appx. D.2) indicate that our models do not benefit from larger depth, and suggest that CKCNNs do not rely on very deep features. Though further analysis is required to draw consistent conclusions, it is intriguing to explore if it is sufficient to equip neural networks with global memory horizons even if this happens in a shallow manner.
High-frequency components. Interestingly, our kernels often contain frequency components higher than the resolution of the grid used during training (Fig. 9). As a result, transitions to finer resolutions benefit from smoothing (see Appx. E.3). Nevertheless, we believe that, if tuned properly, these highfrequency components might prove advantageous for tasks such as super-resolution and compression.
Faster continuous-time models. CKCNNs rely on convolutions, and thus can be executed in parallel. As a result, CKCNNs can be trained faster than recurrent architectures. This difference becomes more pronounced with concurrent continuous-time models for sequential data, which are based on neural ODEs and require at least 5× slower than RNNs (Kidger et al., 2020). At the cost of larger memory costs, CKCNNs can be further sped up by using the convolution theorem.
Neural networks parameterizing spatial functions should be able to model high-frequencies. Our findings indicate that, common nonlinearities do not provide MLPs modelling spatial continuous functions the ability to model high-frequencies. Consequently, architectures that model continuous spatial functions via neural networks should transition towards models endowed with this ability, e.g., MLPs with Sine nonlinearities. These models encompass convolutional networks with continuous kernels, e.g., Schütt et al. (2017); Thomas et al. (2018); Wu et al. (2019), positional encodings in transformers, e.g., Romero & Cordonnier (2020); Hutchinson et al. (2020), and graph neural networks, e.g., Defferrard et al. (2020). Sine nonlinearities can be used to reduce the number of parameters needed to model local functions, or to extend the receptive field of the operations efficiently.

heading: Limitations:
text: Memory requirements. Although, CKCNNs can be deployed and trained in parallel, CKCNNs must store the convolution responses at each layer and for all input positions. This induces a linear memory complexity with regard to the sequence length, and largely contrasts with recurrent continuous-time models, whose memory complexity is constant. The memory consumption of the operation is further incremented if the convolution theorem is applied because it requires multiplying the Fourier transform of the convolution and the kernel, and taking them back to the temporal representation. On the other hand, large convolutional kernels seem to allow CNNs to perform well without using many layers, which has a positive effect on memory consumption.
Selection of ω0. We observe that CKCNNs are very susceptible to the selection of ω0. For instance, performance on pMNIST may vary from 98.54 to 65.22 for values of ω0 in [1,100]. Consequently, finding a good value of ω0 induces an important cost in hyperpararameter search (see Appx. E.4). ω0 acts as a prior on the variability of the target function. However, it is not obvious which value of ω0 is optimal for the internal (unknown) features of a network. Learning layer-wise ω0 values yielded sub-optimal results, and better results were obtained by using a predefined ω0 value across all layers.

heading: 7 CONCLUSION AND FUTURE WORK
text: We introduced the Continuous Kernel Convolution (CKConv), a simple, yet powerful approach able to model global long-term dependencies effectively in a parameter-efficient manner. Aside from the ability to get good accuracy, CKConvs are readily able to handle irregularly-sampled data, and data at different resolutions. CKCNNs achieve state-of-the-art results on multiple datasets, and often surpass neural architectures designed for particular settings, e.g., for irregularly-sampled data.

heading: Future work:
text: We are intrigued about the potential of CKCNNs for tasks in which (global) long-term dependencies play a crucial role, e.g., audio, video, reinforcement learning, (autoregressive) generative modeling. The usage of CKConvs to model long-term interactions in images is also very promising.
In addition, CKConvs provide a convenient way to study the effect of the receptive field size of convolutional architectures, as no network modifications are required for different sizes. Our findings may also be useful for specific problems with irregularly-sampled data, e.g., medical, point clouds. We are also excited about structural advances of CKConvs. For instance, attentive versions of CKCNNs, or formulations that further improve computation and parameter efficiency
Alleviating limitations. Reducing the memory consumption of CKConvs is vital for its application on a broad range of scenarios, e.g., embedded devices. Moreover, finding kernel parameterizations more stable to hyperparameter changes is desirable to reduce the need for hyperparameter search.
What is the best implicit kernel parameterization for convolutional kernels? Despite the success of SIRENs, we believe that better kernel parameterizations might still be constructed, e.g., with Random Fourier Features (Tancik et al., 2020). Aside from improvements in implicit neural representations, which are directly transferable to CKConvs, we consider important to analyze the effect that having unknown, changing target objectives has on the approximation. A thorough empirical study of possible kernel parameterizations is an important direction for future research. In addition, a parameterization with which additional desiderata, e.g., smoothness, can be imposed is also desirable.

heading: REPRODUCIBILITY STATEMENT
text: We believe in reproducibility. In order to make our paper reproducible, we have release the source code used in our experiments to the public. In addition to the code, our repository includes the explicit command lines used to execute each of our experiments, as well as the corresponding pretrained models. Appx. E provides the experimental details of our approach. This section includes details regarding the hardware used, the specification of neural architecture as well as the inputs of MLPψ . It also states the method used for hyperparameter tuning and the hyperparameters of our final models. Details regarding the smoothing of high-frequency artifacts are also provided in this section. Details regarding the datasets and any preprocessing steps used are provided in Appx. C. The proofs of our claims can be found in Appx. A.

heading: APPENDIX

heading: A PROPERTIES OF CKCONVS
text: A.1 VERY IRREGULARLY SAMPLED DATA
CKConvs can readily handle irregularly-sampled, and partially observed data. This is a result of the convolutional kernel MLPψ being able to be sampled at arbitrary positions. For very non-uniformed sampled inputs, however, the corresponding sampling of the convolutional kernel can provide a biased estimation of the operation. To overcome this, one can follow the strategy proposed by Wu et al. (2019), which we summarize here for completeness.
For very non-uniformly sampled inputs, the continuous convolution (x∗ψ)(t) = ∫R x(τ)ψ(t−τ)dτ , must account for the distribution of samples in the input. Specifically, it is reformulated as:
(x ∗ ψ)(t) = ∫ R s(τ)x(τ)ψ(t − τ)dτ, (8)
where s(τ) depicts the inverse sample density of the input at point τ . Intuitively, s(τ) controls the contribution of points x(τ) to the output response. If multiple points are close to one another, their contribution should be smaller than the contribution of points in regions where the sample distribution is much sparser. This provides a Monte Carlo estimate of (x ∗ ψ) from biased samples. In particular, one has that:
∫ f(τ)dτ = ∫ f(τ) p(τ) p(τ)dτ ≈∑i f(τi) p(τi) , for τi ∼ p(τ).
With s(τ) = 1 p(τ) , Eq. 8 provides an unbiased estimation of (x ∗ ψ).

heading: A.2 DATA SAMPLED AT DIFFERENT SAMPLING RATES
text: In addition, CKConvs are readily able to handle data at different resolutions. In particular, the continuous kernel convolution between an input signal x and a continuous convolutional kernel ψ calculated at sampling rates sr1: (x ∗ψ)sr1 , and sr2: (x ∗ψ)sr2 , are approximately equal up to a normalization factor given by sr2
sr1 :
(x ∗ψ)sr2(t) ≈ sr2 sr1 (x ∗ψ)sr1(t).
Consequently, CKCNNs (i) can be deployed at sampling rates different than those seen during training, and (ii) can be trained on data with varying spatial resolutions. The later is important for tasks in which data can be given at different resolutions such as super-resolution and segmentation.
Proof. To prove the previous statement, we start with the continuous definition of the convolution:
(x ∗ ψ)(t) = ∫ R x(τ)ψ(t − τ)dτ,
where we assume for simplicity and without loss of generality that the functions x, ψ are scalar-valued.
In practice, an integral on a continuous function f ∶ R → R cannot be computed on finite time. Consequently, it is approximated via a Riemann integral defined on a finite grid {τsr,i}Nsri=1 obtained by sampling τ at a sampling rate sr:
∫ f(τ)dτ ≈ Nsr
∑ i=1 f(τsr,i)∆sr,
where ∆sr = 1sr depicts the distance between sampled points. For two sampling rates sr1, sr2, the convolution can be approximated through the corresponding Riemann integrals:
∫ R x(τ)ψ(t − τ)dτ ≈
Nsr1
∑ i=1 x(τsr1,i)ψ(t − τsr1,i)∆sr1
≈ Nsr2
∑ i=1 x(τsr2,i)ψ(t − τsr2,i)∆sr2
As a result, we have that both approximations are approximately equal to the continuous integral at positions t defined on both discrete grids. By equating both approximations, we obtain that:
Nsr2
∑ i=1 x(τsr2,i)ψ(t − τsr2,i)∆sr2 ≈
Nsr1
∑ i=1 x(τsr1,i)ψ(t − τsr1,i)∆sr1
Nsr2
∑ i=1 x(τsr2,i)ψ(t − τsr2,i)
´¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¸¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¶ (x∗ψ)sr2(t)
1 sr2
≈ Nsr1
∑ i=1 x(τsr1,i)ψ(t − τsr1,i)
´¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¸¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¹¶ (x∗ψ)sr1(t)
1 sr1
(x ∗ ψ)sr2(t) ≈ sr2 sr1 (x ∗ ψ)sr1(t)
which concludes the proof.

heading: A.3 LINEAR RECURRENT UNITS ARE CKCONVS
text: Interesting insights can be obtained by drawing connections between convolutions and recurrent units. In particular, we can show that linear recurrent units are equal to a CKConv with a particular family of convolutional kernels: exponential functions. Besides providing a generalization to recurrent units, this equality provides a fresh and intuitive view to the analysis of vanishing and exploding gradients.
Recurrent unit. Given an input sequence X = {x(τ)}NXτ=0, a recurrent unit is constructed as: h(τ) = σ(Wh(τ − 1) +Ux(τ)) (9) ỹ(τ) = softmax(Vh(τ)), (10)
where U,W,V parameterize the input-to-hidden, hidden-to-hidden and hidden-to-output connections of the unit. h(τ), ỹ(τ) depict the hidden representation and the output at time-step τ , and σ represents a point-wise non-linearity.
The hidden representation h of a linear recurrent unit, i.e., with σ=Id, can be written as a convolution. To see this, consider the hidden representation of the unit unrolled for t steps:
h(t) =Wt+1h(−1) + t
∑ τ=0 WτUx(t − τ). (11)
Here, h(−1) is the initial state of the hidden representation. We see that in fact it corresponds to a convolution between an input signal x and a convolutional kernel ψ given by:2
x = [x(0),x(1), ...,x(t − 1),x(t)] (12) ψ = [U,WU, ...,Wt−1U,WtU] (13)
h(t) = t
∑ τ=0
x(τ)ψ(t − τ) = t
∑ τ=0 x(t − τ)ψ(τ). (14)
Drawing this equality yields some important insights:
The cause of the exploding and vanishing gradients. Eqs. 12-14 intuitively depict the root of the exploding and vanishing gradient problem. It stems from sequence elements x(t − τ) τ steps back in the past being multiplied with an effective convolutional weight ψ(τ)=WτU. For eigenvalues of W, λ, other than one, the resulting convolutional kernel ψ can only represent functions that either grow (λ≥1) or decrease (λ≤1) exponentially as a function of the sequence length (Figs. 3a, 3b). As a result, the contribution of input values in the past either rapidly fades away or governs the updates of the model parameters. As exponentially growing gradients lead to divergence, the eigenvalues of W for converging architectures are often smaller than 1. This explains the effective small effective memory horizon of recurrent networks.
Linear recurrent units are a subclass of CKConvs. Linear recurrent units can be described as a convolution between the input and a very specific class of convolutional kernels: exponential functions (Eq. 13). In general, however, convolutional kernels are not restricted to this functional class. This can be seen in conventional (discrete) convolutions, whose kernels are able to model complex functions within their memory horizon. Unfortunately, discrete convolutions use a predefined, small kernel size,
2We discard h(−1) as it only describes the initialization of h.
and thus possess a restricted memory horizon. This is equivalent to imposing an effective magnitude of zero to all input values outside the memory horizon (Fig. 3c). CKConvs, on the other hand, are able to define arbitrary large memory horizons. For memory horizons of size equal to the input length, CKConvs are able to model complex functions upon the entire input (Fig. 3d).
In conclusion, we illustrate that CKConvs are also a generalization of (linear) recurrent architectures which allows for parallel training and enhanced expressivity.

heading: B AN SPLINE INTERPRETATION OF RELU AND SINE NETWORKS
text: Sitzmann et al. (2020) motivates the usage of Sine nonlinearities for implicit neural representations. However, there is no clear understanding as of why Sine nonlinearities are better suited for this task than (smooth) piece-wise nonlinearities. Here, we provide an interpretation to this phenomenon from a spline function approximation perspective.
B.1 KERNEL PARAMETERIZATION VIA ReLU NETWORKS
The importance of initialization. There is an important distinction between implicit neural representations and conventional neural applications regarding the assumed distribution of the input. Conventional applications assume the distribution of the input features to be centered around the origin. This is orthogonal to implicit neural representations, where the spatial distribution of the output, i.e., the value of the function being implicitly represented, is uniformly distributed.
For ReLU networks, function approximation is equivalent to an approximation via a maxspline basis (Balestriero & Baraniuk, 2018), and its expressiveness is determined by the number of knots the basis provides, i.e., places where a non-linearity bends the space. Naturally, the better the placing of these knots at initialization, the faster the approximation may converge. For applications in which the data is centered around zero, initializing the knots around zero is a good inductive bias.3 However, for spatially uniform distributed inputs, the knots should be uniformly distributed (Fig. 5). As a result, conventional initializations lead to very poor reconstructions (ReLU 0-Init, Fig. 4), and explicitly aggregating positional encodings to the mappings leads to important improvements, e.g, Mildenhall et al. (2020).
For ReLU layers y=max{0,Wx + b} knots appear at the point where 0=Wx + b. To place the knots at x=0, it is sufficient to set the bias to zero: b=0. For uniformly distributed knots in a range [xmin,xmax], however, one must solve the ReLU equation for uniformly distributed points in that range: 0=Wxunif + b. It results that b= −Wxunif, for arbitrary values of W. In multilayered networks, the approximation problem can be understood as reconstructing the target function in terms of a basis h(L−1). Consequently, the expressivity of the network is determined by the number of knots in h(L−1). In theory, each ReLU layer is able to divide the linear regions of the previous layer in exponentially many sub-regions (Montufar et al., 2014; Serra et al., 2018), or equivalently, to induce an exponential layer-wise increase in the number of knots. For the first layer, the positions of the knots are described by the bias term, and for subsequent layers, these positions also depend on W(l). Unfortunately, as depicted by Hanin & Rolnick (2019), slight modifications of {W(l),b(l)} can strongly simplify the landscape of the linear regions, and thus the knots (Fig. 6). More importantly, Hanin & Rolnick (2019) showed that the number of linear regions at initialization is actually equal to a constant times the number of neurons in the network (with a constant very close
3This is why b=0 is common in regular initialization schemes.
to one in their experiments). In addition, they show that this behavior barely changes throughout training.
An improved initialization scheme. Following the previous reasoning, we explore inducing a uniformly distributed initialization of the knots. However, we observe that finding an initialization with an exponential number of knots is a cumbersome and unstable procedure. In fact, it is not always possible, and, whenever possible, it strongly restricts the values the weights W(l) can assume.
Following the findings of Hanin & Rolnick (2019), we instead employ an initialization procedure with which the total number of knots is equal to the number of neurons of the network. This is obtained by replicating the initialization procedure of the first layer throughout the network: For randomly initialized weights W(l), the bias term b(l) is given by the equality b(l)= −W(l)h(l)(xunif), where xunif is a vector of uniformly distributed points in [xmin,xmax]. Interestingly, we observe that this initialization strategy consistently outperforms the standard initialization for a large range of target functions (ReLU Unif-Init, Fig. 7). Unfortunately however, we note that ReLU networks still show large difficulties in representing very nonlinear and non-smooth functions. In Fig. 7, we illustrate that other popular nonlinearities: LeakyReLU, Swish, exhibit the same behavior.
B.2 KERNEL PARAMETERIZATION VIA Sine NETWORKS
Recently, Sitzmann et al. (2020) proposed to replace ReLU nonlinearities by Sine for the task of implicit neural representation learning. Based on their relation with implicit neural representations, we explore using Sine networks to parameterize our continuous kernels. Intriguingly, we observe that this slight modification allows our kernels to approximate any provided function to near perfection, and leads to a consistent improvement for all tasks considered in this paper (Appx. D.1, Fig. 7).
A possible explanation to these astonishing results can be provided via our prior analysis:
Periodic bending of the space. A Sine layer is given by: y = Sin(ω0[Wx+b]), where ω0 works as a prior on the variability of the target function. Orthogonal to ReLU layers, Sine layers periodically bend the space. As a result, the same y value is obtained for all bias values b′i=bi + n2π∥Wi,∶∥−1, ∀n ∈ Z. This is important from a spline approximation perspective. While for ReLU layers a unique value of b exists that bends the space at a desired position, infinitely many values of b do so for Sine ones. Resultantly, Sine layers are much more robust to parameter selection, and can be tuned to benefit pattern approximation at arbitrary –or even multiple– positions in space (Fig. 5, right). We conjecture that this behavior leads to much more reliable approximations and faster convergence.
An exponentially big Fourier basis. It is not surprising for a (large) basis of phase-shifted sinusoidal functions to be able to approximate arbitrary functions with high fidelity. This result was first observed over two centuries ago by Fourier (1807) and lies at the core of the well-known Fourier transform: any integrable function can be described as a linear combination of a (possibly) infinite basis of phase-shifted sinusoidal functions. Sitzmann et al. (2020) proposed an initialization of {W(l)} that allows for the construction of deep Sine networks able to periodically divide the space into
exponentially many regions as a function of depth. Intuitively, approximations via Sine networks can be seen in terms of an exponentially large Fourier-like basis. We conjecture that this exponential growth combined with the periodicity of sine is what allows for astonishingly good approximations: the more terms in a Fourier transform, the better the approximation becomes.
Interestingly, we find that a uniformly distributed initialization of the bias term bi ∼ U(−π∥Wi,∶∥−1, π∥Wi,∶∥−1) also leads to better and faster convergence for Sine networks.

heading: C DATASET DESCRIPTION
text: Copy Memory Problem. The copy memory task consists of sequences of length T+20, for which the first 10 values are chosen randomly among the digits {1, ...,8}, the subsequent T−1 digits are set to zero, and the last 11 entries are filled with the digit 9. The goal is to generate an output of the same size of the input filled with zeros everywhere except for the last 10 values, for which the model is expected to predict the first 10 elements of the input sequence.
The Adding Problem. The adding problem consists of input sequences of length T and depth 2. The first dimension is filled with random values in [0,1], whereas the second dimension is set to zeros except for two elements marked by 1. The objective is to sum the random values for which the second dimension is equal to 1. Simply predicting the sum to be 1 results in a MSE of about 0.1767.
Sequential and Permuted MNIST. The MNIST dataset (LeCun et al., 1998) consists of 70K grayscale 28×28 handwritten digits divided into training and test sets of 60K and 10K samples, respectively. The sequential MNIST dataset (sMNIST) presents MNIST images as a sequence of 784 pixels for digit classification. Consequently, good predictions require the model to preserve long-term dependencies up to 784 steps in the past: much longer dependencies than most language modelling tasks (Bai et al., 2018b).
The permuted MNIST dataset (pMNIST) additionally permutes the order or the sMNIST sequences at random. Consequently, models can no longer rely on on local features to perform classification. As a result, the classification problem becomes more difficult and the importance of long-term dependencies more pronounced.
Sequential CIFAR10. The CIFAR10 dataset (Krizhevsky et al., 2009) consists of 60K real-world 32 × 32 RGB images uniformly drawn from 10 classes divided into training and test sets of 50K and 10K samples, respectively. Analogously to the sMNIST dataset, the sequential CIFAR10 (sCIFAR10) dataset presents CIFAR10 images as a sequence of 1024 pixels for image classification. This dataset is more difficult than sMNIST, as (i) even larger memory horizons are required to successfully solve the task, and (ii) more complex structures as well as intra-class variations are present in the images (Trinh et al., 2018).
CharacterTrajectories. The CharacterTrajectories dataset is part of the UEA time series classification archive (Bagnall et al., 2018). It consists of 2858 time series of different lengths and 3 channels representing the x, y positions and the pen tip force while writing a Latin alphabet character in a single stroke The goal is to classify which of the different 20 characters was written using the time series data. The maximum length of the time-series is 182.
Speech Commands. The Speech Commands dataset (Warden, 2018) consists of 105809 one-second audio recordings of 35 spoken words sampled at 16kHz. Following Kidger et al. (2020), we extract 34975 recordings from ten spoken words to construct a balanced classification problem. We refer to this dataset as SC_raw. In addition, we utilize the preprocessing steps of Kidger et al. (2020) and extract mel-frequency cepstrum coefficients from the raw data. The resulting dataset, named SC, consists of time series of length 161 and 20 channels.

heading: D ABLATION STUDIES
text: In this section, we perform an ablative study of our approach. Specifically, we analyze the effect of multiple components of our network, and provide additional comparisons with alternative architectures. Specifications on the architectures and hyperparameters used are given in Appx. E.
Table 6: Test accuracies of CKCNNs with multiple MLPψ nonlinearities. Model size = 100K.
NON-LINEARITY DATASET
SMNIST PMNIST SC SC_RAW
RELU 81.21 59.15 94.97 49.15 LEAKYRELU 80.57 55.85 95.03 38.67
SWISH 85.20 61.77 93.43 62.23 SINE 99.31 98.00 95.27 71.66
Table 7: Test accuracy of CKCNNs for various depths and widths.
PMNIST
DEPTH FIXED WIDTH FIXED SIZESIZE ACC.(%) SIZE ACC.(%)
2 Blocks 98k 99.21 98k 99.21 4 Blocks 225k 99.26 95k 99.19 8 Blocks 480k 99.29 105k 99.12 16 Blocks 990k 99.19 107k 99.02

heading: D.1 USING SINE NON-LINEARITIES OVER POPULAR ALTERNATIVES
text: As shown in Sec. 4.2, Sine nonlinearities provide astonishing improvements over equivalent networks with ReLU nonlinearities for function reconstruction. In this section, we provide additional experiments to highlight the suitability of Sine nonlinearities over other popular alternatives both for function approximation and the rest of the tasks considered in this work. The same architectures are used across all experiments and vary only in the nonlinearity used in MLPψ . We find that nonlinearities other than Sine benefit from layer normalization and thus we incorporate it in these variants.
Case I: Function Approximation via MLPψ . First, we evaluate the problem of function approximation in Sec. 4.2, Fig. 4, for nonlinearities other than ReLU and Sine. In particular, we approximate several functions with a MLPψ network which varies only in the type of nonlinearity used: ReLU (Nair & Hinton, 2010), LeakyReLU (Xu et al., 2015), Swish (Ramachandran et al., 2017), and Sine (Sitzmann et al., 2020).
Our results (Fig. 7), illustrate that Sine provides astonishing approximation capabilities over all other nonlinearities considered. In particular, we observe that Sine is the only nonlinearity able to reconstruct very nonlinear and very non-smooth functions, while all other alternatives fail poorly.
Case II: CKCNNs with nonlinearities other than Sine. Next, we consider the case in which CKCNNs with nonlinearities other than Sine are used to solve the tasks considered in Sec. 5. In particular, we train CKCNNs on sMNIST, pMNIST, SC and SC_raw for four different nonlinearities: ReLU, LeakyReLU, Swish, Sine. We utilize the same backbone architecture used in the main text for the corresponding dataset.
Our results (Tab. 6) indicate that Sine outperform CKCNNs using any of the other nonlinearities.
Analysis of the results. Our findings indicate Sine is much better suited to describe continuous spatial functions via neural networks than all other nonlinearities considered. This result motivates replacing popular nonlinearities by Sine for applications in which neural networks are used to describe continuous positional functions. This family of models encompass –but is not restricted to– continuous types of convolutions, e.g., Schütt et al. (2017); Thomas et al. (2018); Finzi et al. (2020); Fuchs et al. (2020), as well as positional encodings in transformers, e.g., Dai et al. (2019); Ramachandran et al. (2019); Romero & Cordonnier (2020), and graph neural networks, e.g., Defferrard et al. (2020). We consider this result to be of large relevance to the deep learning community.

heading: D.2 GOING DEEPER WITH CKCNNS
text: The experimental results shown in Sec. 5 are obtained with shallow CKCNNs composed of 2 residual blocks only. An interesting question is whether going deeper can be used to improve the performance of CKCNNs. To analyze this, we compare deep and shallow CKCNNs with the same architecture for equal width, and equal number of parameters.
Our results (Tab. 7) indicate that deep CKCNNs do not provide improvements over shallow CKCNNs. In fact, deep CKCNNs of fixed size underperform their shallow counterparts. This is an interesting results as shallow CKCNNs do not strongly rely on deep-wise compositionality of features, which is largely considered indispensable in deep learning.
Analysis of the results. The dynamics governing these results are not yet fully understood. However, our findings may lead to two different conclusions, both of which we consider important for the development and understanding of CKCNNs and deep learning in general:
Outcome I: Deep CKCNNs. The first possible outcome is that our current parameterization does not correctly leverage depth. In this case, efforts to construct proper deep CKCNNs will likely lead to performance improvements over the current architectures, and thus have the potential to advance the state-of-the-art further.
Outcome II: Depth is not needed when global memory horizons are provided with shallow networks. The second possible outcome is that depth is used mainly as a means to construct global memory horizons. Consequently, neural networks do not have to be very deep at all provided that global memory horizons are defined by shallow neural networks. Interestingly, this conclusion is in line with the predominant design of recurrent architectures, for which a moderate number of layers are used, e.g., Pascanu et al. (2013a); Graves et al. (2013); Gu et al. (2020b;a). This possible outcome is very exciting as depth is largely considered indispensable in the deep learning community.

heading: E EXPERIMENTAL DETAILS
text: In this section, we provide extended details over our implementation as well as the exact architectures and optimization schemes used in our experiments.

heading: E.1 GENERAL REMARKS
text: Our models follow the structure shown in Fig. 8 and vary only in the number of channels. We use layer normalization (Ba et al., 2016) in our backbone network, and use the Adam optimizer (Kingma & Ba, 2014) across all our experiments. Our code is implemented in PyTorch and is publicly available at link removed for the sake of the double-blind review. We utilize wandb (Biewald, 2020) to log our results, and use NVIDIA TITAN RTX GPUs throughout our experiments.
Continuous Convolutional Kernel MLPψ . All our convolutional kernels are parameterized by a vector-valued 3-layer neural network with 32 hidden units and Sine nonlinearities:
1→ 32→ 32→ Nout ×Nin,
where Nin, NCout are the number of input and output channels of the convolutional layer. We utilize weight normalization (Salimans & Kingma, 2016) in our MLPψ networks, and select a hidden size of 32 based on empirical evidence and findings from previous works, e.g., Finzi et al. (2020).
Normalized relative positions. The MLPs parameterizing our convolutional kenels receive relative positions as input. However, considering unitary step-wise relative positions, i.e., 0, 1, 2, ... , N, can be problematic from a numerical stability perspective as N may grow very large, e.g., N=16000 for the SC_raw dataset. Consequently, we follow good practices from works modelling continuous functions with neural networks, and map the largest unitary step-wise relative positions seen during training [0,N] to a uniform linear space in [−1,1]. Hyperparameter tuning. We tune the hyperparameters of our models via the bayes method given in wandb Sweeps, which selects hyperparameter values via a Gaussian process over the results obtained so far. We perform tuning on a validation dataset until a predefined maximum number of runs of 100 is exhausted. Further improvements upon our results may be obtained by leveraging more sophisticated tuning methods as well as additional runs.
Selecting ω0. CKCNNs are very susceptible to the value of ω0. In order to obtain a reasonable ω0, we first perform a random search on a large interval ω0 ∈ [0,3000]. After a few runs, we stop the random search and select the subinterval in which the validation accuracy is most promising. Next, we restart the random search on this sub-interval and repeat the process until a ω0 value is obtained, for which the validation accuracy is sufficiently high. Surprisingly, we found optimal values of ω0 to be always enclosed in the interval [1,70] even for very long sequences as in SC_raw.

heading: E.2 ACCOUNTING FOR SPATIAL DISPLACEMENTS OF THE SAMPLED CONVOLUTIONAL KERNELS
text: We follow the sampling procedure of Gu et al. (2020a) throughout our test sampling rate discrepancy experiments. Specifically, for a sequence seq of length N, subsampling by a factor n is performed by running seq[::n]. That is, by taking the n-th element of the sequence starting from its first
element. For example, for a sequence of length N=182, different values of n would yield the following sequences:
(n = 1) → [1, 2, 3, ... , 180, 181, 182] (n = 2) → [1, 3, 5, ... , 177, 179, 181] (n = 4) → [1, 5, 9, ... , 173, 177, 181] (n = 8) → [1, 9, 17, ... , 161, 169, 177] Recall that MLPψ takes normalized relative positions in [−1,1] as input, which are computed based on the max input length seen during training. However, some of these subsampling transitions change the max value of the sequence, e.g., for (n = 8) the maximum is given by 177, whereas for (n = 1) this value corresponds to 182. Consequently, a naive approach would consider the last position in each subsampled sequence to correspond to the maximum normalized relative position 1. This effectively induces an spatial displacement, and a re-scaling of the sampled convolutional kernel used during training.
This misalignment is automatically handled under the hood in our CKConv implementation. Nevertheless, we highlight this subtle phenomenon to prevent it in future applications.

heading: E.3 DEALING WITH HIGH-FREQUENCY COMPONENTS
text: Interestingly, our experiments revealed that our continuous kernels often contain frequency components of frequency higher than the resolution of the sampling grid used during training (Fig. 9). As these high-frequency components are not observed during training, we observe that they hurt performance when evaluated at higher resolutions.
In order to neutralize their influence, we filter these components before performing the convolution by means of blurring. This is performed by applying a convolution upon the convolutional kernel with a Gaussian filter G of length 2 srtest
srtrain + 1 and parameters µ=0, σ=0.5:
[G( − srtest srtrain ),G( − srtest srtrain + 1), ...,G(0), ...,G( srtest srtrain − 1),G( srtest srtrain )]
Note that blurring is only used when the test sampling rate is higher than the train sampling rate, as opposed to the normalization factor srtest
srtrain discussed in Eq. 5, Appx. A.2, which is applied whenever
the sampling rates differ.

heading: E.4 HYPERPARAMETERS AND EXPERIMENTAL DETAILS
text: In this section, we provide further specifications of the hyperparameter configurations with with our models are trained. An overview of these hyperparameters is provided in Tab. 8.
Copy Memory. We set the number of channels of our CKCNN as to roughly match the number of parameters of the GRU and TCN networks of Bai et al. (2018a). This is obtained with 10 hidden channels at every layer. We observe that the time to convergence grew proportional to the length of the sequence considered. Whereas for sequences of length 100 convergence was shown after as few as 10 epochs, for sequences of length 6000 approximately 250 epochs were required. The maximum number of epochs is set to 50, 50, 100, 200 and 300 for sequences of size 100, 200, 1000, 3000 and 6000. We observe that different values of ω0 are optimal for different sequence lengths. The optimal ω0 values found are 19.20, 34.71, 68.69, 43.65 and 69.97 for the corresponding sequence lengths.
Adding Problem. We set the number of channels of our CKCNN as to roughly match the number of parameters of the GRU and TCN networks of Bai et al. (2018a). This is obtained with 25 hidden channels at every layer. Similarly to the Copy Memory task, we observe that the time to convergence grew proportional to the length of the sequence considered. Interestingly, this task was much easier to solve for our models, with convergence for sequences of length 6000 observed after 38 epochs. The maximum number of epochs is set to 20, 20, 30, 50 and 50 for sequences of size 100, 200, 1000, 3000 and 6000. We observe that different values of ω0 are optimal for different sequence lengths. The optimal ω0 values found are 14.55, 18.19, 2.03, 2.23 and 4.3 for the corresponding sequence lengths.
sMNIST, pMNIST and sCIFAR10. We construct two models of different sizes for these datasets: CKCNN and CKCNN-Big. The first is constructed to obtain a parameter count close to 100K. The second model, is constructed to obtain a parameter count close to 1M. The parameters utilized for these datasets are summarized in Tab. 8. Despite our efforts, we observed that our models heavily overfitted sCIFAR10. Combinations of weight decay, dropout and weight dropout were not enough to counteract overfitting.
CT, SC and SC_raw. The parameters utilized for classification on these datasets are summarized in Tab. 8. For hyperparameters regarding experiments with irregularly-sampled data please refer to Tab. 9. Any non-specified parameter value in Tab. 9 can be safely consider to be the one listed for corresponding dataset in Tab. 8.

Review summaries:
The paper introduces convolutions with continuously parametrized kernels for sequential data. Continuous kernel convolutions, CKConv for short, parametrize the kernel associated to a convolutional layer as a continuous mapping, $\psi: \mathbb{R}^+ \rightarrow \mathbb{R}^{N_{out} \times N_{in}}$, from the relative time offset $\Delta\tau \in \mathbb{R}^+$ to a weights matrix $\mathbf{W} \in \mathbb{R}^{N_{out} \times N_{in}}$, i.e. $\psi(\Delta \tau) = \mathbf{W}$. This is in contrast to traditional CNN approaches, which represent the convolution kernel as an explicitly learnable set of weights over an a-priori fixed receptive field.  The corresponding CKConv operation can be applied 1) over arbitrarily long receptive fields, including a global one over the history of the input sequence, without the heavy memory burden incurred by representing the convolution kernel explicitly as a set of weights along the horizon; 2) to sequences that are sampled irregularly or asynchronously, since the admissible set of relative positions are not fixed in advance.  A relation is given, which shows approximately how the CKConv changes under resampling the input sequence (the output gets rescaled by the ratio of the sampling rates), which is suggested by the authors to support well-behavedness under varying input resolutions during training and/or between training and testing. Afterwards, a discussion is given about the parametrization of the kernel. Specifically, various MLP models are considered with different activation functions, such as, ReLU, LeakyReLU and Swish, and it is demonstrated that these MLPs are unable to learn functions with high-frequency oscillations, while a recently introduced Sine activation is able to.   The experimental evaluation includes 1) classic stress tests for RNNs, 2) discrete sequence tasks (sMNIST, pMNIST, sCIFAR10), 3) time series modelling (CharacterTrajectories, Speech Commands) , where additionally robustness to missing/irregularly-sampled data and resolution changes are also investigated. The proposed model overall performs favourably to common alternatives; and it seems robust with respect to missing data, and moderate resolution changes.

The paper proposes continuous convolutional kernels parametrized in the form of an MLP. The MLP gets the relative time as input and outputs the column of the convolutional kernel at the given relative time. The authors show that Sine-based (SIREN) non-linearity is best suited for the kernel generating MLP, and an experimental evaluation is performed.

This contribution adapts to sequential data the paradigm of continuous convolution kernels in convolutional neural networks (CNNs). While usual kernels are an extensive list of weights (one for each position), the introduced CKConv frames these kernels as continuous 1D functions, parameterized by a small multilayer perceptron. This paradigms allows for wider kernels with fewer parameters, making it possible to deal with long-range dependencies without recurrence. Several experiments are provided and show that the resulting CNN achieves state of the art performances on a variety of tasks, and in particular is able to deal with irregularly sampled data. 

The paper considers convolutional networks for sequence processing and suggests to parameterize convolutional kernels in this setting by small neural networks. These neural networks are similar to implicit networks in the sense that its input represents a time difference to the currently evaluated time point in a sequence. For each time difference, the associated weights of the convolutional kernel are computed.  This allows to handle arbitrarily large convolutional kernels (albeit the desired size needs to be determined somewhat deterministically apriori), to handle irregularly sampled data and to handle data at different resolutions. Good results on a set of experiments show the practical validity of the idea. The paper does a larger study into the importance of the type of activation functions.