CKConv: Continuous Kernel Convolution For Sequential Data
Authors: David W. Romero
Anna Kuzina
Erik J Bekkers
Jakub Mikolaj Tomczak
Mark Hoogendoorn
Venue: ICLR 2022 Poster (2022)
Keywords: Convolutional Networks
Continuous kernel Convolutions
Continuous Convolutional Kernels
Implicit Neural Representations
Sequential Data.

Abstract: Conventional neural architectures for sequential data present important limitations. Recurrent neural networks suffer from exploding and vanishing gradients, small effective memory horizons, and must be trained sequentially. Convolutional neural networks cannot handle sequences of unknown size and their memory horizon must be defined a priori. In this work, we show that these problems can be solved by formulating the convolutional kernels of CNNs as continuous functions. The resulting Continuous Kernel Convolution (CKConv) handles arbitrarily long sequences in a parallel manner, within a single operation, and without relying on any form of recurrence. We show that Continuous Kernel Convolutional Networks (CKCNNs) obtain state-of-the-art results in multiple datasets, e.g., permuted MNIST, and, thanks to their continuous nature, are able to handle non-uniformly sampled datasets and irregularly-sampled data natively. CKCNNs match or perform better than neural ODEs designed for these purposes in a faster and simpler manner.

Published as a conference paper at ICLR 2022
CKCONV: CONTINUOUS KERNEL CONVOLUTION FOR SEQUENTIAL DATA
David W. Romero', Anna Kuzina', Erik J. Bekkers" Jakub M. Tomezak', Mark Hoogendoorn'
Vrije Universiteit Amsterdam 2 Universityy of Amsterdam
The Netherlands
(d. w. romeroquzman, a. kukina] @vu.n1
ABSTRACT
Conventional neural architectures for sequential data present important limitations. Recurrent neural networks suffer from exploding and vanishing gradients, small effective memory horizons, and must be trained sequentially. Convolutional neural networks cannot handle sequences of unknown size and their memory horizon must be defined a priori. In this work, show that these problems be solved by formulating the convolutional kernels of CNNs as continuous functions. The resulting Continuous Kernel Convolution (CKConv) handles arbitrarily long quences in a parallel manner, within a single operation, and without relying on any form of recurrence. We show that Continuous Kernel Convolutional Networks (CK- CNNs) obtain state-of-the-an results in multiple datasets, e.g., permuted MNIST, and, thanks to their continuous nature, able handle non-uniformly sampled datasets and prregularlyssamplee data natively. CKCNNs match or perform better than neural ODEs designed for these purposes faster and simpler manner.
INTRODUCTION
Recurrent Neural Networks (RNNs) have long governed tasks with sequential data (Rumelhart et al., 1985; Hochreiter & Schmidhuber, 1997; Chung et al., 2014). Their main ingredient are recurrent units: network components with recurrence formulation which grants RNNs the ability to be unrolled for arbitrarily many steps and handle sequences of arbitrary size. In practice, however, the effective memory horizon of RNNs, i.e., the number of steps the network retains information from, has proven to be surprisingly small, most notably due to the vanishing gradients problem (Hochreiter, 1991; Bengio al., 1994). Interstingly, it the very recurrent nature of RNNs that allows them be unrolled for arbitrarily many steps which is responsible for vanishing gradients (Pascanu et al., 2013b). This, in turn, hinders learning from the far past and induces small effective memory horizon.
Convolutional Neural Networks (CNNs) (LeCun et al., 1998) have proven strong alternative to recurrent architectures as long as relevant input dependencies fall within their memory horizon, e.g.. Comneau et al. (2016); Oord et al. (2016); Dai et al. (2017); Dauphin et al. (2017); Bai et al. (2018a). CNNs avoid the training instability and vanishing exploding gradients characteristic of RNNs by avoiding back- pockppropagation through time (Werbos, 1990) altogether. However, these architectures model convolutional kernels as a sequence of independent weights. As a result, their memory horizon must be defined a a-priori, and larger memory horizons induce a proportional growth of the model size.
In this work, we provide a solution to these limitations. We propose to view a convolutional kernel as continuous function parameterized by small neural network instead of sequence of independent weights. The resulting Continuous Kernel Convolution (CKConv) enjoys the following properties:
CKConvs can define arbitrarily large memory horizons within single operation. Consequently. Continuous Kernel Convolutional Neural Networks (CKCNNs) detach their memory horizon from (i) the depth of the network, (ii) the dilation factors used, and (ūi) the size of the network.
CKConvs do not rely on any form of recurrence. As result, CKCNNs (i) can be trained in parallel, and (ii) do not suffer from vanishing exploding gradients or small effective memory horizons.
Continuous convolutional kernels can be evaluated at arbitrary positions. Consequently, CKConvs and CKCNNs can be readily used on irregularly sampled data, and data at different resolutions.

---

Published as a conference paper at ICLR 2022
Nay Ma
b49 k00
etdml
M(An) ((And) v(An) 6(ām)
((An))
(7(m))
((An) ((lm))
9(ars) ((dr)) sidn)) o(are)
X00)
207 351
Figure 1: Continuous Kernel Convolution (CKConv). CKConv views a convolutional kernel as a vector-valued continuous function sb: R RNousNNa parameterized by small neural network MLPV MLP" receives a time-step and outputs the value of the convolutional kernel at that position. We sample convolutional kernels by passing set of relative positions {Ar:) to MLP and perform convolution with the sampled kernel next. Since MLP is continuous function, CKConvs can (i) construct arbitrarily large kernels, (ii) generate kernels at different resolutions, and (iii) handle irregular data.
We observe that continuous kernel parameterizations previously used to handle irregular data locally, e.g., Schütt al. (2017); Wu et al. (2019), not adequate model long-term dependencies. This is due to the inability of their kernels to model long spatial complex functions (Sec. 4.2). Contrarily, CKConvs perfectly describe long complex non-linear, non-smooth functions by parameterizing their kernels as SIRENs (Sitzmann et al., 2020): implicit neural representations with Sine nonlincarities. Shallow CKCNNs match or outperform state-of-the-an
approaches on several tasks comprising stress tests, continuous, discrete and irregular data, as well as resolution changes. To the best of our knowledge, we are first to observe the potential of continuous convolutional kernels to model long-term dependencies, and to provide an useful parameterization to this end.
2 RELATED WORK
Continuous kernel formulation. Continuous formulations for convolutional kernels were introduced to handle irregularly sampled 3D data locally (Schütt et al., 2017; Simonovsky & Komodakis, 2017; Wang et al., 2018; Wu et al., 2019). As discrete convolutions learn independent weights for specific relative positions, they cannot handle irregularly sampled data effectively. Following work focuses on point-cloud applications (Fuchs al., 2020; Hu et al., 2020; Shi al., 2019; Thomas et al., 2018). Other approaches include Monte Carlo approximations of continuous operations (Finzi et al., 2020). Our work proposes a new broad flavor of applications for which continuous kernels are advantageous.
Implicit neural representations. Implicit neural representations construct continuous data repre- sentations by encoding the input in the weights of neural network (Mescheder et al., 2019; Park et al., 2019; Sitzmann et al., 2020). This leads to numerous advantages over conventional (discrete) data representations, e.g., memory efficiency, analytic differentiabilityy with interesting properties for several applications, e.g., generative modelling (Dupont et al., 2021; Schwarz et al., 2020).
Since we model convolutional kernels as continuous functions and parameterize them via neural networks, approach can be understood as implicitly representing the convolutional kernels of a conventional CNN. Different is the fact that these convolutional kernels are not known a- a-priori, but learned as a part of the optimization task of the CNN. Making the connection between implicit neural representations and continuous kernel formulations explicitly brings substantial insights for the construction of these kernels. In particular, it motivates the use of Sine nonlinearities (Sitzmann et al., 2020) to parameterize them, which leads to significant improvements over the ReL.U, Leaky ReUU, and Swish nonlinearities used far for this purpose (Sec. 4.2).
THE CONVOLUTION AND COMMON KERNEL PARAMETERIZAATIONS
Notation. [n] denotes the set {0, (.......; n). Bold capital and lowercase letters depict vectors and matrices, e.g., x, W, sub-indices index vectors, e.g., x= Nr. parentheses index time, e.g., x(r) is the value of x at time-step t. and calligraphic letters depict sequences, e.g., [x(() Nx
N be a vector valued signal Centered and causal convolutions. Let x RR + R.. and æb IR + RN,
and kernel on R, such that x= (ac) Nu and (i-
No, The convolution is defined as:
N
(x * (1))
Σ
J
0ll
2

---

Published as a conference paper at ICLR 2022
1000)
M(1) ME50
hx00)
Me0 B(3)
b(9)
(N2)) M00
(62)
MMP3 19(1) 6000
1000)
MM22)
(00)
M03
(X()
018)
4000
(62)
M03
M12) ME3
000
x(0)
x(1)
xt2)
XX8)
x00)
x11) x(2)
x00) x(1) x(3) x()
(a)
(b)
Figure 2: Discrete centered, causal, and dilated causal convolutions.
In practice, the input signal x is gathered via some sampling procedure. Resultantly, the convolution is effectively performed between the sampled input signal described as sequence of finite length X' [x=(x(T) Nx and convolutional kernel (=((<(T)))
described the same way:
NEw
Xx/2
(x +1) - Σ Σ
xe (e(()()((-T>)
T=-Nx/2 Xx/2
Values x(r) falling outside of are padded by a constant value often defined as zero (Fig. 2a).
The convolutional kernel is commonly centered around the point of calculation t. For sequence modeling this can be undesirable as future input values (x(t-T))==_ are considered during the operation. This is solved by providing a causal formulation to the convolution: a formulation in which the convolution at time-step & only depends input values time-steps (t-r) s &.Fi. 2b):
Nip
(x (1) ME=xE xrc())(c(1-))
c1
In practice, causal convolutions are easily implemented via asymmetrical padding. In this work, we consider causal convolutions as default. Nevertheless, our analyses are also valid for centered ones.
Discrete convolutional kernels. By a large margin, most convolutional kernels up in literature are parameterized as finite sequence of Nk 1 independent learnable weights {=(+>(r) (Fig. 2). As these weights are independent of one another, NK must be kept small to keep the parameter count of the model tractable. Hence, the kernel size often much smaller than the input length: Nk < Nx. This parameterization presents important limitations:
The memory horizon Nk must be defined priori.
Since NK & Nx, this parameterization implicitly assumes that the convolution (x 1b) at position t only depends on imput values at positions up to T=NK steps in the past. Consequently, no functions depending on inputs x(t -T) T) for r > Nk can be modeled.
The most general selection of Nk is given by global memory horizon: NK=Nx. Unfortunately, as discrete convolutional kernels are modeled as a sequence of independent weights, this incurs an extreme growth of the model size and rapidly becomes statistically unfeasible.
Dilated convolutional kernels. To alleviate these limitations, previous works propose to interleave kernel weights with zeros in order to cover larger memory horizons without additional weights (Fig. 2c). This formulation alleviates some of the previous limitations, but introduces additional ones:
Dilated kernels unable model dependencies of input values falling in the interleaved regions.
Several authors use dilated convolutions with varying dilation factors as a function of depth, e.g.. (Bai et al., 2018a; Dai et al., 2017; Oord et al., 2016; Romero et al., 2020). By carefully selecting layer-wise dilation factors, one can assure that some kernel hits each input within the memory horizon of the network. However, due to the extreme sparsity of the formulation, it is difficult to estimate the effective amount of processing applied to the input. In addition, this layout ties together (i) the memory horizon, (ii) the depth, and (iii) the layer-wise dilation factors of the network, which effectively constraints the flexibility of the neural architecture design.
In contrast to the (dilated) discrete convolutions presented in this section, our proposed formulation allows handling arbitrarily long sequences with arbitrarily large, dense memory horizons in a single layer and under a fixed parameter budget.
3

---

Published as a conference paper at ICLR 2022
(a) Recurrent unit (As1) (b) Recurrent unit ((A21) (c) Discrete kernel
(d) Continuous kernel
Figure 3: Functional family of recurrent units, discrete convolutions and CKConvs. For max. eigenvalues of W, X+1, recurrent units are restricted to exponentially decreasing (As1) or increasing (A21) functions (Figs. 3a, 3b). Discrete convolutions can describe arbitrary functions within their memory horizon but zero otherwise (Fig. 3c). Conversely, CKConvs define arbitrary long memory horizons, and thus are able to describe arbitrary functions upon the entire input sequence (Fig. 3d).
4 CONTINUOUS KERNEL CONVOLUTION
In this section, we introduce our approach. First, we define formally, analyze its properties, illustrate its connection recurrent units, and elaborate the functional family they can describe. Next, discuss concrete parameterizations of continuous convolutional kernels, illustrate their connection to implicit neural representations, and show that our final kernels able fit complex functions.
4.1 FORMULATION AND PROPERTIES
Arbitrarily large convolutional kernels. We formulate the convolutional kernel as a continuous vector-valued function parameterized by small neural network MLP IR - IR Noi (Fig. 1, left). MLP receives a relative position (t-r) and outputs the value of the convolutional kernel at that position ((-r)) As result, an arbitrarily large convolutional kernel K (<<<>(t->)) Nx can be constructed by providing an equally large sequence of relative positions {t-r), 7-0 to MLP For Nk=Nx, the size of the resulting kernel is equal to that of the input sequence X,, and thus it is able to model (global) long-term dependencies. The Continuous Kernel Convolution (CKConv) is given by:
Nuc
(T(MM) (x * + (t)= = (x<))_L= (t- (t- c-1 7-0
(4)
Irregularly sampled data. CKConvs are able to handle pregulaaly-sampllee and partially observed data. To this end, it is sufficient to sample MLP at positions for which the input signal is known and perform the convolution operation with the sampled kernel. For very non-uniformly sampled inputs, an inverse density function over the samples can be incorporated in order provide unbiased estimation of the convolution response (see Appx. A.1, Wu et al. (2019) for details).
Data at different resolutions. CKConvs can also process data at different resolutions. Consider the convolution (x *)),,, between an input signal x and a continuous convolutional kernel zp sampled sampling rate ST1- Now, if the convolution receives the same input signal sampled at different sampling rate Sr2, it is sufficient to sample the convolutional kernel at the sampling rate sr2 in order perform an "equivalent" operation: (x )/barg. As shown in Appx. A.2, it holds that:
(x +b) srg (1)
(x +>) )br, (1).
(5)
STi
That is, convolutions calculated different resolutions sT, and ST2 approximately equal up to factor given by the resolution change. As a result, CKCNNs (i) can be trained in datasets with data at varying resolutions, and (ii) can be deployed resolutions other than those seen during training.
We note that the previous features are hardly attainable by regular architectures, with an exception being RNNs with continuous- time interpretations, c.g., Gu et al. (2020a); Kidger al. (2020).
(Linear) recurrent units are continuous kernel convolutions. Consider a recurrent unit of the form:
h(()) (WW((T 1) Ux(r)) = = softmax(((()))
where U, W, V depict the hnput-tohidden. hidden--to-hidden and hidden-to-output connections of the unit, h( ((7). j(r) the hidden representation and the output at time-step т, and a a pointwise nonlinearity. As shown in Appx. A.3, can express the hidden representation h of linear recurrent unit, i.c., with o=Id, as a convolution between the input x and a convolutional kernel +)(r== of size equal to the input. That is, continuous kernel convolution with exponentially increasing
4

---

Published as a conference paper at ICLR 2022
Handooo
Saetooohh
Sine Chirp
Gozslan
Step
Ground Thunt
BaLU
Swid
She
Figure 4: Approximation quality of MLPs with ReLU, LeakyReLU Swish, and Sine nonlincarities. Networks with (smooth) piece-wise nonlinearities are unable to approximate non-smooth, non- linear functions. Sine networks, on the other hand, quickly approximate all target functions to near perfection. All networks share the same structure and vary only in the nonlinearity used.
or decreasing kernel (Fig. 3). Different authors show that nonlinear recurrent units are also restricted to the same functional family (Pascanu al., 2013b; Arjovsky et al., 2016; Zhao et al., 2020).
The functional family of continuous kernel convolutions. From the previous observation, we can conclude that CKConvs are not only more general than discrete convolutions, but that the functional family they describe is also more general than that of (linear) recurrent units (Fig. 3).
4.2 THE CONTINUOUS CONVOLUTIONAL KERNEL MLPV
Convolutional kernels point-wise MLPs. Let (A.;=(t-T:)) be sequence of relative positions. The continuous vector-valued convolutional kernel x/> R N NNi. is parameterized as neural network MLP which maps each relative position AT, to the value the convolutional kernel at that position (Fig. 1, left). We refer to the nonlincarity used in MLP as
What kind of kernels can MLP generate? Our method relies on the assumption that the neural network is able to model complex dependencies densely among all elements within the memory horizon. That is, assumes that *L able to generate arbitrary convolutional kernels.
To test this hypothesis, we fit existing MLP parameterizations, i.e., with ReLU, LeakyReLU and Swish nonlinearities, to long target functions of varying level of smoothness and non-linearity (Fig. 5). We observe that existing parameterizations functions, but for increasing levels of non- non-linearity and mon-smotthness, they fail by large margin. For our analysis, this means that CKConvs with ReLU, LeakyReLU and Swish parameterizations are not able to represent complex input dependencies. In our ablation studies (Appx. D) we verify that CKCNNs with these kernels consistently perform worse than our proposed parameterization.
approximate simple functions, e.g., Gaussian, step
Convolutional kernels as implicit neural representations. We notice that parameterizing a convo- lutional kernel with neural network is equivalent constructing an implicit neural representation of the kernel, with the subtle difference that our target objective is not known a-priori, but learned as part of the optimization task of the CNN. Implicit neural representations study generie ways to represent data living in diweensiiional spaces, e.g., R via neural networks, and thus, despite this difference, constitute an interesting starting point for the parameterization of continuous convolutional kernels. In particular, recent works noticed that neural networks with piece-wise activation functions unable to model high-rrequencis.. To alleviate this limitation, they introduce random Fourier features (Tancik et al., 2020) and Sine nonlinearities (Sitzmann et al., 2020).
Based on these observations, we repeat the fitting experiment for a SIREN (Sitzmann et al., 2020): MLP with hidden layers of the form y Sine(wol Wx b)). That is with Sine nonlinearities,
5

---

Published as a conference paper at ICLR 2022
and non-leamable value wy that serves as prior for the oscillations of the output. We observe that a SIREN quickly approximates all target functions to near perfection regardless of their grade of smoothness or nonlinearity. Even sequence of random noise. This implies that, contrary to other parameterizations, CKConvs with SIREN kernels have the ability to model complex input dependencies across large memory horizons. Our experimental results verify this statement. Our ablation studies in Appx. D show that SIREN kernels consistently outperform all other variants. In addition, our experimental results in Sec. 5 show that shallow CKCNNs with SIREN kernels achieve state-of-the-arr across datasets of different nature, i.e., with continuous and discrete data.
The success of Sine nonlinearites: A spline basis interpretation. Sitzmann al. (2020) motivates the usage of Sine nonlinearities for implicit neural representations. However, there is no clear understanding as of why Sine nonlinearities are better suited for this task than (smooth) piece-wiss wise nonlinearities. For the interested reader, we provide an interpretation to this phenomenon from spline function approximation perspective in Appx. B.
Of most practical relevance from this analysis is the observation that proper initialization of the network parameters, particularly of the bias term {b(),, is important to create a well-spread set of basis functions suited for function approximation. For SIRENs, this is achieved by initializing the bias term uniformly across the period of each of the Sine components: b; ~ ([<x]] Wi,: [ We observe that this initialization leads to better results and faster convergence for all tasks considered.
EXPERIMENTS
We validate our approach against several existing models and across several tasks selected from the corresponding papers. Specifically, we benchmark its ability handle long-term dependencies, data at different resolutions and sregularly-sample data. A complete description of the datasets used as well as additional experiments and ablation studies can be found in the Appendix (Appx. C, D).
Network details. We parameterize our convolutional kernels as 3-layer SIRENs.
Weight normaliza- tion (Salimans Kingma, 2016) leads to better and faster convergence when applied to the layers in MLP, and we use it across all experiments. All our CKCNNs follow the structure shown in Fig. 8 and vary only in the number of blocks and channels. We use two residual blocks for all experiments reported in this section. Specifications on the architectures and hyperparameters used are given in Appx. E. We speed up the convolution operations in our networks via the convolution theorem: (f (f*))= (3 [W]), with I the Fourier transform.
Stress experiments. First, we validate that CKCNNs can readily model memory horizons of different lengths. To this end, we evaluate if a shallow CKCNN is able to solve the Copy Memory and Adding Problem tasks (Hochreiter & Schmidhuber, 1997) for sequences of sizes in the range [100, 6000]. Success is achieved if 100% accuracy, or a loss S Ic-4 are obtained, for the copy memory and adding problem, respectively. Random predictions for the adding problem lead loss of approx. 0.17.
Our results show that a shallow CKCNN is able to solve both problems for all sequence lengths considered without requiring structural modifications (Tab. 2). Recurrent architectures are not able not solve the copy problem at all and could solve the sum problem up to 200 steps. TCNs with k=7, n=7 were able to solve both tasks for up to 1000 steps. However, larger lengths were out of reach as their memory horizon is constrained a priori. To handle larger sequences, TCNs must modify their network structure based on prior knowledge regarding the expected length of the input sequence.
Discrete sequences. The continuous nature of our kernels might give the impression that CKCNNs are only suited for continuous data, i.e., time-series. However, Sine nonlinearities allow our convolu- tional kernels to model complex non-smooth non-linear functions (Fig. 4). Consequently, we validate whether CKCNNs can be applied for discrete sequence modeling on the following tasks: sMNIST, PMNIST (Le et al., 2015), SCIFARIO (Trinh et al., 2018) and Char-level PTB (Marcinkiewicz, 1994).
Shallow CKCNNs outperform recurrent, self-attention and convolutional models on sMNIST and PMNIST (Tab. 1). On SMNIST, a small CKCNN (100K params.) achieves of results with a model 80x smaller than the current state-of-the-art. A wider CKCNN (1M params.) slightly increases this result further. On PMNIST, we see an improvement of 0.8% over the best model of size S 1100K, and our wider shallow CKCNN achieves state-of-the-art on this dataset. For SCIFARIO, 10,
Our code is publicly available at httpp:://githab...omm/

---

Published as a conference paper at ICLR 2022
Table 1: Test results on discrete sequential datasets.
SMNIST
PMNIST
SCIFARIO
CHAR- PTB bpc
MODEL.
SIzE
Acc (%)
Acc (%)
Acc (%)
TCN (Bai et al., 2018a)
70K
99.0
97.2
1.31'
LSTM (Bai al., 2018s)
70k
87.2
85.7
1.36
GRU (Bai al. 2018a)
2018)))
70k
96.2
87.3
1.37'
IndRNN (Li al. 2018)
838
99.0
96.0
DiRNN (Chang et al,, 2017)
44K
98.0
96.1
HiPPO al., 20200s
0.5M
98.30
r-LSTM (Trinh et al., 2018)
0.5M
98.4
95.2
72.2
Self-Att. (Trinh et al., 2018)
0.5M
98.9
97.9
62.2
TrellisNet (Bai et al., 201
8M
99.20
73.42
1.158'
CKCNN
98K
99.31
98.00
62.25
CKCNN-Big
1M
99.32
98.54
63.74
1.045'
Model sizes sizes are 3M for TCN, LSTM and GRU, 13.4M for TrellisNet and 1.8M for CKCNN Big.
Table 2: Evaluation on stress tasks. / marks if the problem has been solved.
Table 3: Test accuracies on CT, SC and SC_ C__www
MODEL
SIZE
CT
SC
SC_
MODEL
SIZE
SEQ. LENGTII
(De Browwer et al., 2019)
GRU-ODE
89K
10.0
100
200
1000
3000
6000
GRU-At
COPY MEMORY
(Kidger et al., 2020)
89K
97.8
20.0
10.0
GRU
TCN
16K
16K
/
GRU-D Che et al. (2018)
89K
95.9
23.9
10.0
CKCNN
16K
/
ODE-RNN
88K
97.1
93.2
10.0
ADDING PROBLEM (LoSs)
(Rubanova al., 2019)
NCDE
GRU
70K
/
(Kidger et al., 2020)
89K
98.8
88.5
10.0
TCN
70K
/
/
CKCNN
70K
/
/
/
7
CKCNN
100X
99.53
95.27
71.66
our small CKCNN obtains similar results self-attention model 5x bigger, and wider variant improves performance by an additional 1%. Our best results are obtained with an even wider model (2.5M params) with which an accuracy of 65.59% is obtained. On Char-level PTB a CKCNN with 3M parameters outperforms all models considered as well as the state-of-ther afttte-ar: Mogrifier LSTMs (Melis et al., 2019), while being 13.3x smaller.
Time-series modeling. Next, we evaluate CKCNNs on time-series data. To this end, we consider the CharacterTVajeecoriiss (CT) (Bagnall al., 2018) and the Speech Commands (SC) (Warden, 2018) datasets. We follow Kidger et al. (2020) to obtain a balanced classification dataset with precomputed mel-frequency cepstrum coefficients. In addition, we evaluate the ability of CKCNNs to model long-term dependencies by training on the raw SC dataset (SC_raw), whose records have length 16k.
We compare CKCNNs with representative sequential models with continuous-time interpretations: GRU-ODE (De Brouwer et al., 2019), GRU-At (Kidger et al., 2020), ODE-RNN (Rubanova et al., 2019), and NCDE (Kidger et al., 2020). Continuous- time sequential models were selected as they are only sequential methods also able to handle pregularlyssapple data, and data at different resolutions. Our results show that shallow CKCNNs outperform all continuous-time models considered for both the CT and SC datasets (Tab. 3). In addition, CKCNNs obtain promising results on SC_raw which validates their ability to handle very-long-teng dependencies. In fact, CKCNNs trained on SC, raw are able outperform several Neural ODE models trained on the preprocessed data (SC).
In addition, we observed that neural ODE methods considered in Tab. 3 were prohibitively slow for long sequences. For instance, NCDEs were 228x slower than CKCNN of equivalent size on SC raw, taking 17 hours per epoch to train. Consequently, training a NCDE on SC_raw raw for matching number of epochs would take more than 212 days conclude. In order to provide results for these models, we train them under the same computational budget than CKCNNs. This is enough to train them for a single epoch. All obtained results are best only marginally better than random.
Testing at different sampling rates. We now consider the case where a network is trained with data at a sampling rate sr, and tested with data at a different sampling rate sr2. Our results show that the performances of CKCNNs remains stable for large sampling rate fluctuations (Tab. 5). This behaviour contrasts with most previous continuous -iime models, whose performance rapidly decays upon these changes. CKCNNs outperform HiPPO (Gu et al., 2020a) and set new sfate-of-ther -o--he--rttttt in this setting. Importantly, depending on the sampling, additional care may be needed to account for spatial displacements and figh-freeueencis of our kernels (see Appx. E.2 for details).
7

---

Published as a conference paper at ICLR 2022
Table 4: Test results on irregular data.
PHYSIONET
CHARACTERTRRJECCORIEEI
SPEDCHCOMMANDS RRW
MODEL
AUC
(0%)
(30%)
(50%)
(70%)
(0%)
(30%)
(50%)
(70%)
GRU-ODE
0,852
96.2
92.6
86.7
89.9
~ 10.0
10.0
~ 10.0
10.0
GRU-At
0.878
97.8
93.6
91.3
90.4
GRU-D
0.871
95.9
94.2
90.2
91.9
ODE-RNN
0.874
97.1
93.4
96.0
95.3
NCDE
0.880
98.8
98.7
98.8
98.6
CKCNN
0.895
99.53
99.30
98.83
98.14
71.66
63.46
60.55
57,50
Table Results for different train and test resolutions. Fractions depict resolutions proportional to the original one of the dataset. The accuracy of all models on the original resolution surpasses 90%.
CKCNN SiZE=
TEST FREQ.
DATASET
TRAIN FREQ
1/2
1/4
1/4
1/10
99.53
99. 30
99.30
95.88
76 45
1/3
98.83
99.07
98.37
96.97
96.74
96.97
99.30
98.83
84.85
4/8
96.97
97.44
97.20
99.30
73.43
71.66
65.96
52.11
40.33
30.87
SC_ RAW
1/2
72.09
72.06
69.03
63.00
29.67
1/4
68.25
68.40
69.47
67.09
37.91
3/6
40.48
42.00
54.91
66.44
22.29
MODEL COMPARISON CHARACTER TRAJECTORIES
MODEL
GRU-D
ODE-RNN
LMU
NCDE
HIPPO
CKCNN
1/2
23.1
41.8
99.30
1/2 1
25.5
31,5
11.3
13.1
90.1
98.83
crregularlysaample data. To conclude, we validate CKCNNs for irregularly-sapple data. To this end, consider the PhysioNet sepsis challenge (Reyna et al., 2019) as well as the CT dataset with drops provide results under the
of 30%, 50% and 70% of the data as in Kidger et al. (2020). In addition, same methodology for the SC_ Jaw dataset. As Kidger et al. (2020) we add an additional channel to the input to indicate whether the value at that position is known.
Our results show that CKCNNs outperform NCDEs and obtain state-of-thed of-thc-art performance on the PhysioNet dataset. In addition, CKCNNs exhibit stable performance for varying quantities of missing data, and perform better than several models explicitly developed to this end (Tab. 4). On the CT dataset, NCDEs perform slightly better than CKCNNs for large data drop rates. However, argue that our method is still advantageous due to the gains in training speed -see Section for details-.
6 DISCUSSION AND LIMITATIONS
Parameter-efficient large convolutional kernels. CKConvs construct large complex kernels with a fixed parameter budget. For large input sequences, this results in large savings in the number of parameters required to construct global kernels with conventional CNNs. For sequences from the PMNIST (length 784) and SC. raw (length 16000) datasets, a conventional CNN with global kernels would require 2.14M and 46.68M of parameters, respectively. for a model equivalent to our CKCNN (100K). In other words, our kernel parameterization allows us to construct CKCNNs that are 21, 84 and 445, 71 times smaller than corresponding conventional CNNs for these datasets. Detailed exploration on the effect of our efficient continuous kernel parameterizations in optimization, overfitting and generalization is an interesting direction for future research.
Is depth important? Shallow global memory horizons. Our results are obtained with CKCNNs built with two residual blocks only. Additional experiments (Appx. D.2) indicate that our models do not benefit from larger depth, and suggest that CKCNNs do not rely very deep features. Though further analysis is required to draw consistent conclusions, it intriguing to explore it is sufficient to equip neural networks with global memory horizons even if this happens in shallow manner.
High-frequency components. Interestingly, our kernels often contain frequency components higher than the resolution of the grid used during training (Fig. 9). As result, transitions to finer resolutions benefit from smoothing (see Appx. E.3). Nevertheless, we believe that, if tuned properly, these high- frequency components might prove advantageous for tasks such super-resolution and compression.
8

---

Published as a conference paper at ICLR 2022
Faster continuous-time models. CKCNNs rely on convolutions, and thus can be executed in parallel. As a result, CKCNNs can be trained faster than recurrent architectures. This difference becomes more pronounced with concurrent continuous-time models for sequential data, which are based on neural ODEs and require at least 5x slower than RNNs (Kidger et al., 2020). At the cost of larger memory costs, CKCNNs be further sped up by using the convolution theorem.
Neural networks parameterizing spatial functions should be able to model high-frequencies. Our findings indicate that, common nonlinearities do not provide MLPs modelling spatial continuous functions the ability to model high-frequencies. Consequently, architectures that model continuous spatial functions via neural networks should transition towards models endowed with this ability, e.g.. MLPs with Sine nonlinearities. These models encompass convolutional networks with continuous kernels, e.g., Schütt et al. (2017); Thomas et al. (2018); Wu et al. (2019), positional encodings in transformers, e.g., Romero & Cordonnier (2020); Hutchinson et al. (2020), and graph neural networks, e.g.. Defferrard et al. (2020). Sine nonlinearities can be used to reduce the number of parameters needed to model local functions, to extend the receptive field of the operations efficiently.
Memory requirements. Although, CKCNNs can be deployed and trained in parallel, CKCNNs must store the convolution responses at each layer and for all input positions. This induces linear memory complexity with regard to the sequence length, and largely contrasts with recurrent continuous time models, whose memory complexity is constant. The memory consumption of the operation is further incremented the convolution theorem is applied because it requires multiplying the Fourier transform of the convolution and the kemel,, and taking them back to the temporal representation. On the other hand, large convolutional kernels seem to allow CNNs to perform well without using many layers, which has a positive effect on memory consumption.
Selection of wo- We observe that CKCNNs are very susceptible to the selection of wp. For instance, performance on PMNIST may vary from 98.54 65. 22 for values of wo in [1, 100]. Consequently, finding a good value of wo induces an important cost in hyperpararameter search (see Appx. E.4), wo acts as prior on the variability of the target function. However, it is not obvious which value of wo is optimal for the internal (unknown) features of network. Learning layer-wise wp values yielded sub-optimal results, and better results were obtained by using a predefined we value across all layers.
7 CONCLUSION AND FUTURE WORK
We introduced the Continuous Kernel Convolution (CKConv), a simple, yet powerful approach able model global long-term dependencies effectively in a parameter-efficient manner. Aside from the ability to get good accuracy, CKConvs are readily able to handle different resolutions. CKCNNs achieve state-of thetheerrt results on multiple datasets, and often surpass neural architectures designed for particular settings, e.g., for irregularly-samplee data.
data, and data at
We are intrigued about the potential of CKCNNs for tasks in which (global) long-term dependencies play crucial role, e.g., audio, video, reinforcement learning, (autoregressive) generative modeling. The usage of CKConvs to model long-term interactions in images also very promising. addition, CKConvs provide convenient way to study the effect of the receptive field size of convolutional architectures, as no network modifications are required for different sizes. Our findings may also be useful for specific problems with irregularly-sanee sampled data, e.g., medical, point clouds. We are also excited about structural advances of CKConvs. For instance, attentive versions of CKCNNs, or formulations that further improve computation and parameter efficiency
Alleviating limitations. Reducing the memory consumption of CKConvs is vital for its application on broad range of scenarios, e.g., embedded devices. Moreover, finding kernel parameterizations more stable to hyperparameter changes is desirable to reduce the need for hyperparameter search.
What is the best implicit kernel parameterization for convolutional kernels? Despite the success of SIRENS, we believe that better kernel parameterizations might still be constructed, e.g., with Random Fourier Features (Tancik et al.. 2020). Aside from improvements in implicit neural represen- tations. which are directly transferable to CKConvs, we consider important to analyze the effect that having unknown, changing target objectives has on the approximation. A thorough empirical study of possible kernel parameterizations is an important direction for future research. A parameterization with which additional desiderata, e.g., smoothness, can be imposed is also desirable.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We believe in reproducibility. In order to make our paper reproducible, we have release the source code used in our experiments to the public. addition to the code, our repository includes the explicit command lines used to execute each of our experiments, as well as the corresponding pretrained models. Appx. E provides the experimental details of our approach. This section includes details regarding the hardware used, the specification of neural architecture as well as the inputs of MLP It also states the method used for hyperparameter tuning and the hyperparameters of our final models. Details regarding the smoothing of tegh-frequency artifacts are also provided in this section. Details regarding the datasets and any preprocessing steps used are provided in Appx. C. The proofs of our claims can be found in Appx. A.
ACKNOWLEDGEMENTS
We gratefully acknowledge Gabriel Dernbach for interesting analyses on the knot distribution of ReLU networks. We thank Emiel van Krieken and Ali el Hasouni as well for interesting questions and motivating comments at the beginning of this project.
David W. Romero is financed part of the Efficient Deep Learning (EDL) programme (grant number P16-25), partly funded by the Dutch Research Council (NWO) and Semiotic Labs. Anna Kuzina is funded by the Hybrid Intelligence Center, 10-year programme funded by the Dutch Ministry of Education, Culture and Science through the Netherlands Organisation for Scientific Research. Erik J, Bekkers is financed by the research programme VENI (grant number 17290) funded by the Dutch Research Council. All authors sincerely thank everyone involved in funding this work.
This work was carried out on the Dutch national cinfrastructure with the support of SURF Cooperative
REFERENCES
Martin Arjovsky, Amar Shah, and Yoshua Bengio. Unitary evolution recurrent neural networks. In International Conference on Machine Learning, pp. 1120-1128, 2016.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv: 2016.
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The uea multivariate time series classification archive, 2018. arXiv preprint arXiv:1811.00075. 2018.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling, arXiv preprint arXiv:1803.0127), 2018a.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Trellis networks for sequence modeling. arXiv preprint arXiv:1810.0660 2018b.
Randall Balestriero and Richard Baraniuk. Mad max: Affine spline insights into deep learning, arXiv preprint wwXXiv/805.065777 2018
Yoshua Bengio. Patrice Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE transactions on neural networks, 5(2):157-166, 1994.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL http:://wwwwwwwa/ com/. Software available from wandb.com.
Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Kasegawa-JJhnson and Thomas S Huang. Dilated recurrent neural networks. In Advances in neural information processing systems, pp. 77-87, 2017.
Zhengping Che, Sanjay Purushotham, Kyunghyun Cho, David Sontag, and Yan Liu. Recurrent neural networks for multivariate time series with missing values. Scientific reports, 8(1):1112, 2018.
10

---

Published as a conference paper at ICLR 2022
Junyoung Chung. Caglar Gulcehre, KyungHyun Cho, and Yoshua Bengio. Empirical evaluation of gated recurrent neural networks on sequence modeling. arXiv preprint arXiv:1412.3555, 2014.
Alexis Conneau, Holger Schwenk, Loie Barrault, and Yann Lecun. Very deep convolutional networks for text classification. arXiv preprint arXiv:1606.01780 2016.
Wei Dai, Chia Dai, Shuhui Qu, Juncheng Li, and Samarjit Das. Very deep convolutional neural networks for raw waveforms. In 2017 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 421-425 IEEE, 2017.
Zihang Dai, Zhilin Yang, Yiming Yang, Jaime Carbonell, Quoc Le, and Ruslan Salakhutdi- Transformer-xl: Attentive language models beyond fixed-length context. arXiv preprini arXiv:1901.02860 2019.
Yann N Dauphin, Angela Fan, Michael Auli, and David Grangier. Language modeling with gated convolutional networks, In International conference on machine learning, pp. 933-941, 2017.
Edward De Brouwer, Jaak Simm, Adam Arany, and Yves Moreau. Gru-ode-bayes: Continuous modeling of sporadically time series. In Advances in Neural Information Processing Systems, pp. 7379-7390, 2019.
Michaël Defferrard, Martino Milani, Frédérick Gusset, and Nathanaël Perraudin. Deepsphere: graph-based spherical cnn. arXiv preprint arXiv:2012. /5000, 2020.
Emilien Dupont, Yee Whye Teh, and Arnaud Doucet. Generative models distributions of functions. arXiv preprint arXiv:2102.04776 2021.
Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. arXiv preprint arXiv:2002. /2880, 2020.
J Fourier. Mémoire sur la propagation de chaleur dans les corps solides, présenté le 21 décembre 1807 à l'institut national nouveau bulletin des sciences par la société philomatique de paris. i. In Paris: First European Conference on Signal Analysis and Prediction, pp. 17-21, 1807.
Fabian B Fuchs, Daniel E Worrall, Volker Fischer, and Max Welling. Se (3)-transformm:s:: 3d roto-translation equivariant attention networks. arXiv preprint arXiv:2006. 10503, 2020.
Ary L Goldberger, Luis AN Amaral, Leon Glass, Jeffrey M Hausdorff, Plamen Ch Ivanov, Roger G Mark, Joseph E Mietus, George B Moody, Chung-Kang Kang Peng, and H Eugene Stanley. Physiobank, physiotoolkit, and physionet: components of new research resource for complex physiologic signals. circulation, 101((3):e215 e220, 2000.
Alex Graves, Abdel-rahman Mohamed, and Geoffrey Hinton. Speech recognition with deep recurrent neural networks. In 2013 IEEE international conference on acoustics, speech and signal processing. pp. 6645-66499 leee, 2013.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. arXiv preprint arXiv:2008.07669 2020a.
Albert Gu, Caglar Gulcehre, Thomas Paine, Matt Hoffman, and Razvan Pascanu. Improving the gating mechanism of recurrent neural networks. In International Conference on Machine Leurning, pp. 3100330000000 PMLR, 2020b.
Boris Hanin and David Rolnick. Complexity of linear regions deep networks. arXiv preprint arXiv:1901.09221 2019.
Sepp Hochreiter. Untersuchungen zu dynamischen neuronalen netzen. Diploma, Technische Univer- sität Münnchen 91(1), 1991.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
11

---

Published as a conference paper at ICLR 2022
Qingyong Hu, Bo Yang, Linhai Xie, Stefano Rosa, Yulan Guo, Zhihua Wang, Niki Trigoni, and Andrew Markham. Randla-net: Efficient semantic segmentation of large-scale point clouds. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11108-11117, 2020.
Michael Hutchinson, Charline Le Lan, Sheheryar Zaidi, Emilien Dupont, Yee Whye Teh, and Hyunjik Kim. Lietransformer: Equivariant self-attentio for lie groups. arXiv preprint arXiv:2012. 2020.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons Neural controlled differential equations for irregular time series. arXiv preprint (xXii:2005008922. 2020.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: 6980, 2014.
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Quoe V Navdeep Jaitly, and Geoffrey E Hinton. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint arXiv:1504.0094 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, (6(11))2278-222 1998.
Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indrnn): Building a longer and deeper rnn. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5457-5466, 2018.
Mary Ann Marcinkiewicz. Building large annotated corpus of english: The penn treebank. Using Large Corpora, pp. 273, 1994.
Gábor Melis, Tomás Kočiský, and Phil Blunsom. Mogrifier Istm. arXiv preprint arXiv:1909.01792 2019.
Lars Mescheder, Michael Oechsle, Michael Niemeyer, Sebastian Nowozin, and Andreas Geiger. Occupancy networks: Learning 3d reconstruction in function space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4460-4470, 2019.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes neural radiance fields for view synthesis. arXiv preprint arXiv:2003.08934, 2020.
Guido F Montufar, Razvan Pascanu, Kyunghyun Cho, and Yoshua Bengio. On the number of linear regions of deep neural networks. In Advances in neural information processing systems, pp- 2932- 2014.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In Icml, 2010.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv:1609.03499 2016.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 165-174 174, 2019.
Razvan Pascanu, Caglar Gulcehre, Kyunghyun Cho, and Yoshua Bengio. How to construct deep recurrent neural networks. arXiv preprint 131iv:1312.600 2013a.
Razvan Pascanu, Tomas Mikolov, and Yoshua Bengio. On the difficulty of training recurrent neural networks. In International conference on machine learning, pp. 1310-1318, 2013b.
Prajit Ramachandran, Barret Zoph, and Quoc V Le. Searching for activation functions. arXiv preprint arXiv: 2017.
12

---

Published as a conference paper at ICLR 2022
Prajit Ramachandran, Niki Panmar, Ashish Vaswani, Irwan Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. arXiv preprint UrXiv:196.00599, 2019.
Matthew A Reyna, Chris Josef, Salman Seyedi, Russell Jeter, Supreeth P Shashikumar, M Brandon Westover, Ashish Sharma, Shamim Nemati, and Gari D Clifford. Early prediction of sepsis from clinical data: the physionet/compuring in cardiology challenge 2019. In 2019 Computing in Cardiology (CinC), pp. Page-1. IEEE, 2019.
David W Romero and Jean-Baptiste Cordonnier. Group equivariant stand-alone self-attention for vision. arXiv preprini urXiv:2010.00997, 2020.
David W Romero, Erik J Bekkers, Jakub M Tomezak, and Mark Hoogendoorn. Wavelet networks:
Scale equivariant learning from raw waveforms, arXiv preprint arXiv:2006.05259 2020.
Yulia Rubanova, Ricky TQ Chen, and David Duvenaud. Latent ordinary differential equations for ireeuulrly-samplle time series. In Advances in Neural Information Processing Systems, pp. 5320-5330, 2019.
David E. Rumelhart, Geoffrey E Hinton, and Ronald J Williams. Learning internal representations by error propagation. Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
Tim Salimans and Diederik P Kingma. Weight normalization: A simple reparameterization to accelerate training deep neural networks. arXiv preprint
2016.
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoe Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnet: continuous-filte convolutional neural network for modeling quantum interactions. In Advances in neural information processing systems, pp. 991-1001, 2017.
Katja Schwarz, Yiyi Liao, Michael Niemeyer, and Andreas Geiger. Graf: Generative radiance fields for 3d-aware image synthesis. arXiv preprint arXiv:2007.02442. 2020.
Thiago Serra, Christian Tjandraatmadja, and Srikumar Ramalingam. Bounding and counting linear regions of deep neural networks. In International Conference Machine Learning, pp. 3558-4566. PMLR, 2018.
Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. arXiv preprint arXiv:1907.03670, 2019.
Martin Simonovsky and Nikos Komodakis. Dynamic edge-conditioned filters in convolutional neural networks on graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3693-3702, 2017.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. arXiv preprint arXiv:2006.1077 2020.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-an and translation -qquvvvainnn neural networks for 3d point clouds. arXiv preprint arXiv:1802.088 08200. 2018
Trieu H Trinh, Andrew M Dai, Minh-Thang Luong, and Quoc V Le. Learning longer-term dependen- 2018.
cies in rnns with auxiliary losses. arXiv preprint
Shenlong Wang. Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky, and Raquel Urtasun. Deep parametric continuous convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2589-2597, 2018
13

---

Published as a conference paper at ICLR 2022
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03209, 2018.
Paul J Werbos. Backpropagation through time: what it does and how to do it. Proceedings of the IEEE, 18510): 1990.
Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9621-9630, 2019.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. Empirical evaluation of rectified activations in convolutional network. arXiv preprint arXiv: 1505:0005500555 2015.
Jingyu Zhao, Feiqing Huang, Jia Lv, Yanjie Duan, Zhen Qin, Guodong Li, and Guangjian Tian. Do mn and lstm have long memory? In International Conference on Machine Learning, pp. 11365-11375. PMLR, 2020.
14

---

Published as a conference paper at ICLR 2022
APPENDIX
A PROPERTIES OF CKCONVS
A.1 VERY IRREGULARLY SAMPLED DATA
CKConvs can readily handle irreuularly-samplee and partially observed data. This is result of the convolutional kernel MLP being able to be sampled at arbitrary positions. For very non-uniformed sampled inputs, however, the corresponding sampling of the convolutional kernel provide biased estimation the operation. To overcome this, one can follow the strategy proposed by Wu al. (2019), which summarize here for completeness.
For very non-uniformly sampled inputs, the continuous convolution (t)= (()) dr, must account for the distribution of samples in the input. Specifically, it is reformulated
(x (t) f ((T))(r))((--))
(8)
where s(T) depicts the inverse sample density of the input at point т. Intuitively, s(T) controls the contribution of points x(r) to the output response. If multiple points are close to one another, their contribution should be smaller than the contribution of points in regions where the sample distribution is much sparser. This provides Monte Carlo estimate of (x v)) from biased samples. In particular, one has that:
[ f
P(T) ~ ((,)) for p(T).
f(T.)
With s(T)= 1 p(-), Eq. 8 provides an unbiased estimation of (x * 1/).
A.2 DATA SAMPLED AT DIFFERENT SAMPLING RATES
In addition, CKConvs are readily able to handle data at different resolutions. In particular, the continuous kernel convolution between an input signal and continuous convolutional kernel tb calculated at sampling rates sr1: (x v/) sr1 and s12: (x v/) s02 are approximately equal up to
normalization factor given by SI2:
ST1
ST2
(x * 2 822 (t)
(x wbb sxr
Sr
Consequently, CKCNNs (i) can be deployed at sampling rates different than those seen during training, and (ii) can trained on data with varying spatial resolutions. The later is important for tasks in which data can be given at different resolutions such super-resolution and segmentation.
Proof. To prove the previous statement, we start with the continuous definition the convolution:
* (+))) = fc x(T))((( 7) dr,
where we assume for simplicity and without loss of generality that the functions x, 1/ are scalar-valued.
In practice, an integral on a continuous function f R + IR cannot be computed on finite time. Consequently, it is approximated via Riemann integral defined on a finite grid (r by sampling r at sampling rate sr:
N obtained
Nor f [(F) Σ
where A.r = 1 depicts the distance between sampled points. For two sampling rates sr SI2, the convolution can be approximated through the corresponding Riemann integrals:
Nor. Σ
f (((r)((t-7) -t v
x ((Ts - Tsr. Den.
Σ
x (TTr..)))((( -Tac. Dan
15

---

Published as a conference paper at ICLR 2022
As result, we have that both approximations approximately equal to the continuous integral at positions defined on both discrete grids. By equating both approximations, we obtain that:
Nary
2 (Tarz.) (t( Tar2
Σ x w) Ter
Nazz
Nary
Σ
x (Tarz ( )b/(t Tarzir) 1
Σ x (Tar )abb
Ters ari
(x+x))c)(1)
(x*v))r,
SI2
(x (b),))
(x aa (t)
sT1
which concludes the proof.
A.3 LINEAR RECURRENT UNITS ARE CKCONVS
Interesting insights can be obtained by drawing connections between convolutions and recurrent units. In partieular,
show that linear recurrent units are equal to CKConv with particular family of convolutional kernels: exponential functions. Besides providing a generalization to recurrent units, this equality provides fresh and intuitive view to the analysis of vanishing and exploding gradients.
Recurrent unit. Given input sequence X = (x(+)) recurrent unit is constructed as:
h(T) o (Wh(T-- Ux(T))
(9)
ý(7) softmax(Vh(7)).
(10)
where U, W, V parameterize the input-to-hidder
and hidden-to-outpe connec- tions of the unit. h(T). ((T) depict the hidden representation and the output at time-step т, and o represents a point-wise non-linearity.
The hidden representation h of a linear recurrent unit, i.e., with ø=Id, can be written as a convolution. To see this, consider the hidden representation of the unit unrolled for steps:
h(t)= hw-+)((1)) Σ W'U(tt-))
(11)
Here, h(-1) is the initial state of the hidden representation. We see that in fact it corresponds to convolution between an input signal x and a convolutional kernel zbb given by:"
x= [x(0), x(1),
x))x((
(12)
Φ= UU WU, W'
(13)
h(()= Σ x(7))((1 - - x(t-
(14)
Drawing this equality yields some important insights:
The cause of the exploding and vanishing gradients. Eqs. 12-14 intuitively depict the root of the exploding and vanishing gradient problem. It stems from sequence elements x(t ((f-r) -) t steps back in the past being multiplied with an effective convolutional weight (T))=W'U. For eigenvalues of W, X, other than one, the resulting convolutional kernel vp can only represent functions that either grow (A21) decrease (As1) exponentially as function of the sequence length (Figs. 3a, 3b). As result, the contribution of input values in the past either rapidly fades away or governs the updates of the model parameters. As exponentially growing gradients lead divergence, the eigenvalues of W for converging architectures are often smaller than 1. This explains the effective small effective memory horizon of recurrent networks.
Linear recurrent units are subclass of CKConvs. Linear recurrent units can be described as convolution between the input and a very specific class of convolutional kernels: exponential functions (Eq. 13). In general, however, convolutional kernels are not restricted to this functional class. This can be seen in conventional (discrete) convolutions, whose kernels are able to model complex functions within their memory horizon. Unfortunately, discrete convolutions use predefined, small kernel size,
We discard 1-1)) as it only deseribes the initialization of h.
16

---

Published as a conference paper at ICLR 2022
and thus possess restricted memory horizon. This is equivalent to imposing an effective magnitude of zero to all input values outside the memory horizon (Fig. 3c). CKConvs, on the other hand, are able to define arbitrary large memory horizons. For memory horizons of size equal to the input length, CKConvs are able to model complex functions upon the entire input (Fig. 3d).
In conclusion, we illustrate that CKConvs also generalization of (linear) recurrent architectures which allows for parallel training and enhanced expressivity.
B AN SPLINE INTERPRETATION OF RELU AND SINE NETWORKS
Sitzmann et al. (2020) motivates the usage of Sine nonlinearities for implicit neural representations. However, there is no clear understanding as of why Sine nonlinearities are better suited for this task than (smooth) piece-wise nonlinearities. Here, we provide an interpretation to this phenomenon from a spline function approximation perspective.
B. KERNEL PARAMETERIZATION VIA ReLU NETWORKS
The importance of initialization. There is an important distinction between implicit neural rep- resentations and conventional neural applications regarding the assumed distribution of the input. Conventional applications assume the distribution of the input features to be centered around the origin. This orthogonal to implicit neural representations, where the spatial distribution of the output, i..., the value of the function being implicitly represented, is uniformly distributed.
For ReLU networks, function approximation is equivalent spline basis (Balestriero & Baraniuk, 2018), and its expressiveness is determined by the number of knots the basis provides, i.e., places where non-linearity bends the space. Nat- urally, the better the placing of these knots at initialization, the faster the approximation may converge. For applications in which the data is centered around zero, initializing the knots around zero is a good inductive bias.3 However, for spatially uniform distributed in- puts, the knots should be uniformly distributed (Fig. 5). As a result, conventional initializations lead to very poor reconstructions (ReLU O-Init, Fig. 4), and explicitly aggregating positional encodings to the mappings leads to important improvements, e.g, Mildenhall et al. (2020).
approximation via max-
An step function approximated via spline basis (left) and a periodic basis (right). As the target function is defined uniformly terval, uniformly initializing the knots of the spline basis provides faster and better approximations. Pe- riodic bases, on the other hand, periodically bend space, and thus can be tuned easier approximate the target function at arbitrary points in space.
Figure 5:
given in-
For ReLU layers y=max (0, Wx + b) knots appear at the point where O= + b. To place the knots at x=0, it is sufficient to set the bias to zero: b=0. For uniformly distributed knots in range [Xmin: Xmax J. however, one must solve the ReLU equation for uniformly distributed points in that range: 0- Wxmii + b. It results that b Wxunt. for arbitrary values of W.
In multilayered networks, the approximation problem can be understood as reconstructing the target function in terms of basis (L-1), Consequently, the expressivity of the network is determined by the number of knots in (L-1), In theory, each ReLU layer is able divide the linear regions of the previous layer in exponentially many sub-regions (Montufar et al., 2014; Serra et al., 2018), or equivalently, to induce an exponential layer-wise increase in the number of knots. For the first layer, the positions of the knots are described by the bias term, and for subsequent layers, these positions also depend on ( Unfortunately, as depicted by Hanin & Rolnick (2019), slight modifications of (W()) ( can strongly simplify the landscape of the linear regions, and thus the knots (Fig. 6). More importantly, Hanin & Rolnick (2019) showed that the number of linear regions initialization is actually equal to a constant times the number of neurons in the network (with a constant very close
Thiis is why b=O is common regular initialization schemes.
17

---

Published as a conference paper at ICLR 2022
Input
Sawtooth(X)
Figure 6: The sensitivity of networks with layer-wise exponential growing slight changes. Taken from Hanin & Rolnick (2019). The sawtooth function with 2" teeth (left) can be easily expressed via ReLU network with 3n neurons (bottom). However, slight perturbation of the network parameters -Gaussian noise with standard deviation 0.1- greatly simplifies the linear regions captured by the network, and thus the distribution of the knots the basis (right).
to one in their experiments). addition, they show that this behavior harely changes throughout training.
An improved initialization scheme. Following the previous reasoning, we explore inducing a uniformly distributed initialization of the knots. However, observe that finding initialization with an exponential number of knots is cumbersome and unstable procedure. In fact, it is not always possible, and, whenever possible, it strongly restricts the values the weights (()) can assume.
Following the findings of Hanin & Rolnick (2019), we instead employ initialization procedure with which the total number of knots is equal to the number of neurons of the network. This is obtained by replicating the initialization procedure of the first layer throughout the network: For randomly initialized weights W(), the bias term (()) is given by the equality b()= W (()
(xunit). (xxmir)) where Xunif is a vector of uniformly distributed points in [Xmin> Xmax Interestingly, we observe that this initialization strategy consistently outperforms the standard initialization for large range of target functions (ReLU Unif-Init, Fig. 7). Unfortunately however, we note that RcLU networks still show large difficulties in representing very nonlinear and non-smooth functions. In Fig. 7, we illustrate that LeakyReLU, Swish, exhibit the same behavior.
other popular nonlinearities:
B.2 KERNEL PARAMETERIZATION VIA Sine NETWORKS
Recently, Sitzmann et al. (2020) proposed to replace ReLU nonlinearities by Sine for the task of implicit neural representation learning. Based on their relation with implicit neural representations, we explore using Sine networks to parameterize our continuous kernels. Intriguingly, we observe that this slight modification allows our kernels to approximate any provided function near perfection, and leads to consistent improvement for all tasks considered in this paper (Appx. D.1, Fig. 7).
possible explanation to these astonishing results can be provided via our prior analysis:
Periodic bending of the space. A Sine layer is given by: y Sin(wo [x+ bx+b]), where wo works as prior on the variability of the target function. Orthogonal ReLU layers, Sine layers periodically bend the space. As result, the same value is obtained for all bias values b/-b, n2x / W Vn E Z. This is important from spline approximation perspective. While for ReLU layers a unique value of b exists that bends the space at a desired position, infinitely many values of b do so for Sine ones. Resultantly, Sine layers are much more robust to parameter selection, and can be tuned to benefit pattern approximation at arbitrary conjecture that this behavior leads to much more reliable approximations and faster convergence.
multiple- positions in space (Fig. 5, right). We
An exponentially big Fourier basis. It is not surprising for (large) basis of phase-shifted sinusoidal functions to be able to approximate arbitrary functions with high fidelity. This result was first observed two centuries ago by Fourier (1807) and lies at the core of the well-known Fourier transform: any integrable function can be described as a linear combination of a (possibly) infinite basis of shase-shifted sinusoidal functions. Sitzmann et al. (2020) proposed an initialization of (W()) that allows for the construction of deep Sine networks able to periodically divide the space into
18

---

Published as a conference paper at ICLR 2022
exponentially many regions as a function of depth. Intuitively, approximations via Sine networks can be seen in terms of an exponentially large Fourier-like basis. We conjecture that this exponential growth combined with the periodicity of sine is what allows for astonishingly good approximations: the more terms in a Fourier transform, the better the approximation becomes.
Interestingly, we find that uniformly distributed initialization of the bias term b; u(- W. a 1. (
also leads to better and faster convergence for Sinc networks.
C DATASET DESCRIPTION
The copy memory task consists of sequences of length T+20, for which the first 10 values chosen randomly among the digits {1...., 8), the subsequent T-1 digits are set to zero, and the last 11 entries are filled with the digit 9. The goal is to generate an output of the same size of the input filled with zeros everywhere except for the last 10 values, for which the model is expected to predict the first 10 elements of the input sequence.
Copy Memory Problem.
The Adding Problem. The adding problem consists of input sequences of length T and depth 2. The first dimension is filled with random values in [0, 1], whereas the second dimension is set to zeros except for two elements marked by 1. The objective is to sum the random values for which the second dimension is equal to 1. Simply predicting the sum to be results in a MSE of about 0. 0767
Sequential and Permuted MNIST. The MNIST dataset (LeCun al., 1998) consists of 70K gray- scale 28x 28 handwritten digits divided into training and test sets of 60k and 10K samples, respectively. The sequential MNIST dataset (sMNIST) presents MNIST images sequence of 784 pixels for digit classification. Consequently, good predictions require preserving long-term dependencies up to 784 steps in the past: much longer than most language modelling tasks (Bai et al., 2018b).
The permuted MNIST dataset (PMNIST) additionally permutes the order or the sMNIST sequences at random. Consequently, models can no longer rely on on local features to perform elassification. As a result, the classification problem becomes more difficult and the importance of long-term dependencies more pronounced.
Sequential CIFAR10. The CIFARIO dataset (Krizhevsky et al., 2009) consists of 60k real-world 32 x 32 RGB images uniformly drawn from 10 classes divided into training and test sets of 50K and 10K samples, respectively. Analogously to the sMNIST dataset, the sequential CIFARIO (SCIFARIA) 10) dataset presents CIFAR10 10 images as a sequence of 1024 pixels for image classification. This dataset is more difficult than sMNIST, (i) even larger memory horizons required to solve the task, and (ii) more complex structures and intra-class variations are present in the images (Trinh et al., 2018).
Character Trajectories. The Character/Tajectoriie dataset is part of the UEA time series classifica- tion archive (Bagnall et al., 2). It consists of 2858 time series of different lengths and 3 channels representing the x; y positions and the pen tip force while writing a Latin alphabet character in single stroke The goal is to classify which of the different 20 characters was written using the time series data. The maximum length of the time-series is 182.
Speech Commands. The Speech Commands dataset (Warden, 2018) consists of 105809 one-second audio recordings of 35 spoken words sampled at 16kHz. Following Kidger et al. (2020), we extract 34975 recordings from ten spoken words to construct a balanced classification problem. We refer to this dataset as SC_ raw. In addition, we utilize the preprocessing steps of Kidger et al. (2020) and extract mel- frequecyyy cepstrum coefficients from the raw data. The resulting dataset, named SC, consists of time series of length 161 and 20 channels.
PhysioNet. The PhysioNet 2019 challenge on sepsis prediction (Goldberger et al., 2000; Reyna et al., 2019) is irregularly sampled, partially observed dataset consisting of 40335 time series of variable length describing the stay of patients within an ICU, Time-series are made out of 5 static features, e.g., age, and 34 time-dependent features, e.g., respiration rate, creatinine blood concentration, and 10.3% of the values are observed. We follow Kidger et al. (2020) and consider the first 72 hours of patient's 's stay to predict whether sepsis is developed over the course of their entire stay wwhich can extend for month for some patients-.
Penn BrenBa...a The PennTreeBank (PTB) (Marcinkiewicz, 1994) is a language corpus which consists of 5,095K characters for training, 396K for validation and 446K for testing. On a char
19

---

Published as a conference paper at ICLR 2022
Table Test accuracies of CKCNNs with multi- ple MLP nonlinearities. Model size 100k.
Table 7: Test accuracy of CKCNNs for various depths and widths.
PMNIST
DATASET
NON-LINEARIE
FIXED WIDTH ACC.(S)
FIXED SiZE
SMNIST
PMNIST
SC
SC_RAW
DEPTH
S1ZE
S1ZE
AcC. (6)
RELU
59.15
94.97
49.15
Blocks
98k
99.21
98k
99.21
LEAKYRELU
80.57
55.85
95.03
38.67
Blocks
225k
99.26
95k
99.19
SWISH
85,20
61.77
93.43
62.23
Blocks
480k
99.29
105k
99.12
SINE
99.31
98.00
95.27
71.66
Blucks
990L
lever that we use in our experiment the vocabulary size is 50 characters (or the size of the alphabet, including end-of-string char). We follow Bai et al. (2018a) in performing character-level language modeling task on this dataset.
D ABLATION STUDIES
In this section, we perform an ablative study of our approach. Specifically, we analyze the ef- fect of multiple components of our network, and provide additional comparisons with alternative architectures. Specifications on the architectures and hyperparameters used are given in Appx. E.
D.1 USING SINE NON-LINEARITIES OVER POPULAR ALTERNATIVES
As shown in Sec. 4.2, Sine nonlinearities provide astonishing improvements over equivalent net- works with ReLU nonlinearities for function reconstruction. In this section, we provide additional experiments to highlight the suitability of Sine nonlinearities over other popular alternatives both for function approximation and the rest of the tasks considered in this work. The same architectures are used across all experiments and vary only in the nonlinearity used in MLP We find that nonlinearities other than Sine benefit from layer normalization and thus we incorporate it in these variants.
Case I: Function Approximation via MLP" First, we evaluate the problem of function approxima- tion in Sec. 4.2, Fig. 4, for nonlinearities other than ReLU and Sine. In particular, we approximate several functions with a MLP network which varies only in the type of nonlinearity used: ReLU (Nair& & Hinton, 2010), LeakyReLU (Xu et al., 2015), Swish (Ramachandran et al., 2017). and Sine (Sitzmann al., 2020).
Our results (Fig. 7), illustrate that Sine provides astonishing approximation capabilities over all other nonlinearities considered. In particular, we observe that Sine the only nonlinearity able to reconstruct very nonlinear and very non-smooth functions, while all other alternatives fail poorly.
Case I: CKCNNs with nonlinearities other than Sine. Next, we consider the case in which CKCNNs with nonlinearities other than Sine are used to solve the tasks considered in Sec. 5. In partieular, we train CKCNNs on sMNIST, PMNIST, SC and SC_raw for four different nonlinearities: ReLU, LeakyReLU, Swish, Sine. We utilize the same backbone architecture used in the main text for the corresponding dataset.
Our results (Tab. 6) indicate that Sine outperform CKCNNs using any of the other nonlinearities.
Analysis of the results. Our findings indicate Sine much better suited to describe continuous spatial functions via neural networks than all other nonlincarities considered. This result motivates replacing popular nonlincaritics by Sine for applications in which neural networks are used to describe continuous positional functions. This family of models encompass but is not restricted to- continuous types of convolutions, e.g. Schüit et al. (2017): Thomas et al. (2018); Finzi et al. (2020); Fuchs et al. (2020), as well as positional encodings in transformers, e.g., Dai et al. (2019); Ramachandran et al. (2019); Romero & Cordonnier (2020), and graph neural networks, e.g., Defferrard et al. (2020). We consider this result be of large relevance the deep learning community.
D.2 GOING DEEPER WITH CKCNNS
The experimental results shown in Sec. 5 are obtained with shallow CKCNNs composed of 2 residual blocks only. An interesting question is whether going deeper can be used improve the performance
20

---

Published as a conference paper at ICLR 2022
Random
Sowtooth
Sine Cairp
Gaussian
Constaut
Ground Truth
ReLU 0-Inil
RoLU Unit-nni
Leakyftel.U O-Init
Leakeited.U Uni-Ihil
Swiish
O-Init
Unif- Init
Swish
Sine
Figure 7: Function approximation via ReLU, LeakyReLU, Swish and Sine networks. All network variants perform a decent job in approximating simple functions. However, for non-linear, non- smooth functions, all networks using nonlinearities other than Sine provide very poor approximations. Interestingly, the uniform knot initialization proposed Sec. 4.2 provides consistent improvements for all network variants. However, despite this improvement, the approximation results remain insufficient. Contrarily, Sine networks quickly and seamlessly approximate functions. network configurations equal up the non- non-lincarities used.
21

---

Published as a conference paper at ICLR 2022
of CKCNNs. To analyze this, compare deep and shallow CKCNNs with the same architecture for equal width, and equal number of parameters.
Our results (Tab. 7) indicate that deep CKCNNs do not provide improvements over shallow CKCNNs. In fact, deep CKCNNs of fixed size underperform their shallow counterparts. This is an interesting results as shallow CKCNNs do not strongly rely deep-wise compositionality of features, which is largely considered indispensable in deep learning.
Analysis of the results. The dynamics governing these results are not yet fully understood. However, our findings may lead to two different conclusions, both of which we consider important for the development and understanding of CKCNNs and deep learning in general:
Outcome I: Deep CKCNNs. The first possible outcome that our current parameterization does not correctly leverage depth. In this case, efforts construct proper deep CKCNNs will likely lead to performance improvements over the current architectures, and thus have the potential to advance the ofate-of-the-ar further.
Outcome II:
Depth is not needed when global memory horizons are provided with shallow net- works. The second possible outcome is that depth is used mainly means to construct global memory horizons. Consequently, neural networks do not have to be very deep at all provided that global memory horizons are defined by shallow neural networks. Interestingly, this conclusion is in line with the predominant design of recurrent architectures, for which a moderate number of layers are used, e.g., Pascanu et al. (2013a); Graves et al. (2013); Gu et al. (2020b;a). This possible outcome is very exciting depth is largely considered indispensable in the deep learning community.
E EXPERIMENTAL DETAILS
In this section, we provide extended details over our implementation as well as the exact architectures and optimization schemes used in our experiments.
E.1 GENERAL REMARKS
Our models follow the structure shown in Fig. 8 and vary only in the number of channels. We use layer normalization (Ba et al., 2016) in our backbone network, and use the Adam optimizer (Kingma Ba, 2014) across all our experiments. Our code is implemented in PyTorch and is publicly available at link removed for the sake of the double-blind review. We utilize wandb (Biewald, 2020) to log our results, and use NVIDIA TITAN RTX GPUs throughout our experiments.
Continuous Convolutional Kernel MLP*, All our convolutional kernels are parameterized by a vector-valued 3-layer neural network with 32 hidden units and Sine nonlinearities:
32 + 32 Nout x Nin-
where Nin. NCont are the number of input and output channels of the convolutional layer. We utilize weight normalization (Salimans & Kingma, 2016) in our MLP networks, and select a hidden size of 32 based on empirical evidence and findings from previous works, e.g., Finzi et al. (2020).
Normalized relative positions. The MLPs parameterizing our convolutional kenels receive relative positions as input. However, considering unitary step-wise relative positions, i.c., 0, 1, 2, N, can be problematic from numerical stability perspective N may grow very large, e.g., N=16000 for the SC_raw dataset. Consequently, we follow good practices from works modelling continuous functions with neural networks, and map the largest unitary step-wise relative positions seen during training [O,N] to a uniform linear space in [-
Hyperparameter tuning. We tune the hyperparameters of our models via the bayes method given in wandb Sweeps, which selects hyperparameter values via Gaussian process over the results obtained so far. We perform tuning on a validation dataset until a predefined maximum number of runs of 100 is exhausted. Further improvements upon results may be obtained by leveraging more sophisticated tuning methods as well as additional runs.
Selecting Wp- CKCNNs are very susceptible to the value of wp. In order obtain reasonable wp. we first perform a random search on a large interval wn 3000]. After a few runs, we stop the random search and select the subinterval in which the validation accuracy is most promising. Next,
22

---

Published as a conference paper at ICLR 2022
input. input.lengtl
Input
CKCoav
Lay Lare
GetRelPositione
ReLU
RelPositions
xL
CKBlock
DropDut
Kerne
Linear
CKCoav
ConvKernel
LayerNorm
ReLU
FFTConv
Output
DropDut
RaLU
Figure 8: Graphical description of continuous kernel convolutional networks. Dot-lined blocks depict optional blocks, and blocks without borders depict variables. KernelNet blocks use Sine nonlinearities. We replace spatial convolutions by Fourier Convolutions (FFTConv), which leverages the convolution theorem to speed up computations.
we restart the random search on this sub-interval and repeat the process until Wy value is obtained, for which the validation accuracy is sufficiently high. Surprisingly, we found optimal values of wy to be always enclosed in the interval [1, 70] even for very long sequences as in SC_raw.
E.2 ACCOUNTING FOR SPATIAL DISPLACEMENTS OF THE SAMPLED CONVOLUTIONAL KERNELS
We follow the sampling procedure of Gu et (2020a) throughout our test sampling rate discrepancy experiments. Specifically, for sequence seq of length N, subsampling by factor is performed by running = :n]. That is, by taking n-th element the sequence starting from its first element. For example, for sequence of length 181 different values of would yield the following sequences:
(n 1) [1, 2, 3,
180, 181, 182]
nn 2) [1, 3, 5,
177, 179, 181]
(n 4) [1, 5, 9,
173, 177, 181]
(n 8) + [1, 9, 17,
161, 169, 177]
Recall that MLP takes normalized relative positions in [-1,1] input, which computed based on the max input length seen during training. However, some these subsampling transitions change the max value of the sequence, e.g., for 8) the maximum is given by 177, whereas for (n 1) this value corresponds to 182. Consequently, a naive approach would consider the last position in each subsampled sequence correspond to the maximum normalized relative position 1. This effectively induces an spatial displacement, and a re-scaling of the sampled convolutional kemnel used during training.
This misalignment is automatically handled under the hood in our CKConv implementation. Never- theless, we highlight this subtle phenomenon prevent future applications.
E.3 DEALING WITH HIGHI-PREQUENCY COMPONENTS
Interestingly, our experiments revealed that our continuous kernels often contain frequency com- ponents of frequency higher than the resolution of the sampling grid used during training (Fig. 9). As these high-free figh-frequency components are not observed during training, we observe that they hurt performance when evaluated higher resolutions.
23

---

Published as a conference paper at ICLR 2022
Table 8: Hyperparameter specifications of the best performing CKCNN models.
PARAMS.
CoPy MEMORY
ADDING PROBLEM
Small / Rig
SMNIST
PMNIST Small Big
SCIFARIO Small /Big
CT
SC
SC_RAW
PTB
Ppochs
See Appx.
See Appx,
200
200
300
200
Batch Size
32
32
64 Adum 0.001
64 Adam
64
32
64
32
24
Optimizer
Adam
Adam
Adam
Adam
Adam
Adam
Adam
Lesmiing Rate
0.001
0.001
0.001
0.901
0.001
0.001
0.002
# Blooks
2
2
2
2
2
2
2
Hidden Size
10
25
30/ 100
30/ 100
30/ 100
30
30
30
128
Li
See Appx. E.4
See Appx.
43.46 42.16
25.67
39.45
25.78
Dropout
0.2
0.2/0.3
0.1
0.2
Inpur Dropout
0.1
0.1
0.0/0.0
0.1
Emrbeeking Dropout
0.1
Weight Dropout
-70.1
Weight Decay
-/ le-4
le-4
le-6
Schedulor
Platcau
Plateuu
Plateau
Platean
Platcau
Platean 5
Patience
20
20
20
20
15
20
Scheduler Decay
5
5
5
5
Model
70.59%
98.29K 1.03M
1.05M
100.04K
118.24K
98.29K
Hy pepprrrateerr values for the classtication and varying sampling ralc tasks, For hyperporameters w.r.l inegularly sampled data please sec Talx 9.
In order to neutralize their influence, we filter these components before performing the convolution by means of blurring. This is performed by applying convolution upon the convolutional kernel with Gaussian filter G of length 2 glamt
and parameters μ=0, o=0.5:
stunin
[o(
Blire )
C.(((
SFice + +1)...,, G(),, G Stiee 1), G
Slice.
Note that blurring is only used when the test sampling rate is higher than the train sampling rate, as opposed to the normalization factor Shee discussed in Eq. 5, Appx. A.2, which is applied whenever the sampling rates differ.
E.4 HYPERPARAMETERS AND EXPERIMENTAL. DETAILS
In this section, we provide further specifications of the hyperparameter configurations with with our models are trained. An overview of these hyperparameters is provided in Tab. 8.
Copy Memory. We set the number of channels of our CKCNN as to roughly match the number of parameters of the GRU and TCN networks of Bai et al. (2018a) 18). This is obtained with 10 hidden channels at every layer. We observe that the time to convergence grew proportional to the length of the sequence considered. Whereas for sequences of length 100 convergence was shown after as few as 10 epochs, for sequences of length 6000 approximately 250 epochs were required. The maximum number of epochs is set to 50, 50, 100, 200 and 300 for sequences of size 100, 200, 1000, 3000 and 6000. We observe that different values of Wo are optimal for different sequence lengths. The optimal we values found are 19.20, 34.71, 68.69, 43.65 and 69.97 for the corresponding sequence lengths.
Adding Problem. We set the number of channels of our CKCNN as to roughly match the number of parameters of the GRU and TCN networks of Bai et al. (201 1a). This is obtained with 25 hidden channels at every layer. Similarly to the Copy Memory task, we observe that the time to convergence grew proportional to the length of the sequence considered. Interestingly, this task was much easier to solve for our models, with convergence for sequences of length 6000 observed after 38 epochs. The maximum number of epochs set to 20, 20, 30, 50 and 50 for sequences of size 100, 200, 1000, 3000 and 6000. We observe that different values of wp are optimal for different sequence lengths. The optimal we values found are 14.55, 18.19, 2.03, 2.23 and 4.3 for the corresponding sequence lengths.
SMNIST, PMNIST and SCIFAR10. We construct two models of different sizes for these datasets: CKCNN and CKCNN-Big. The first is constructed to obtain parameter count close to 100K. The second model, constructed obtain a parameter count close to 1M. The parameters utilized for these datasets are summarized in Tab. 8. Despite our efforts, observed that models heavily overfitted SCIFARIO. Combinations weight decay, dropout and weight dropout were not enough to counteract overfitting.
CT, SC and SC_raw. in Tab. 8. For hyperparameters regarding experiments with eeguularl-sampllle data please refer to Tab. 9. Any non-specified parameter value in Tab. can be safely consider to be the one listed for corresponding dataset in Tab. 8.
raw. The parameters utilized for classification on these datasets are summarized
24

---

Published as a conference paper at ICLR 2022
Table Hyperparameter values for experiments on irregularly sampled data. Non-listed parameters correspond to those in Tab. 8.
CT
SC, RAW
PARAMS.
PHYSIONET
(50%)
(70%)
(30%)
(50%)
(7016)
Wi
4,38
17.24
12.00
4.24
35.66
31,70
25.29
Dropout
0.D
0.2
0.2
0.0
0.1
Weight Decay
0.0
16-4
0.0
Size
Model Size
175.71K
101.75k
99 34K
Training Freq Joney
sralys
sew
sr=l
Maphlanl M
Figure 9: High-frees figh-frreuency components in Sine continuous kernels. We observe that continuous kernels parameterized by Sine networks often contain frequency components of frequency higher than the resolution of the grid used during training. Here, for instance, the kernel looks smooth on finer
the training grid. However, several high-frequency components appear when sampled grid. Though this may be problematic phenomenon, we believe that, if tuned properly, these high- frequency components can prove advantageous model fine details in tasks such super-resolution and compression cheaply.
Penn TreeBank For a character-level language modeling on PTB dataset we use hyperparameters specified in Tab. 8. We embedding of size 100 following the TCN model from Bai al. (2018a).
25


Reviews:
[{"review_id": "zuF5BNOgAf", "reviewer": "Reviewer_NgYq", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The contribution is clear and well written. I reviewed a previous\nversion for another venue and thank the authors for their effort in\nclarifying their manuscript, which I now find very didactic and\npleasant to read.\n\nI found the concept of continuous kernel interesting. The authors make\nit clear that they are adapting it from previous work on 3D data, but\nalso show how its transfer to sequential data is a useful contribution\nin particular for dealing with long-range dependencies without the\nlimitations of RNN or discrete CNN. In particular, they clearly show\nhow a discrete convolution kernel would require much more parameters\nthan a CKConv to model the same range of dependencies.\n\nThey also provide thorough experiments illustrating the superiority of\ntheir proposed continuous kernel CNN (CKCNN) against state of the art\napproaches on all targeted applications: long-range dependency\nmodelling, irregular sampling, varying sampling rate.\n\nI believed that this contribution could both be inspiring for new\nparadigms of networks (e.g. as an alternative or complement to\nattention), and lead to better performances in practice on the\ntargeted applications.\n\nOne point that could be further clarified is the choice of using a single MLP\nfor modelling the Nin x Nout convolution kernels. Among other\npossibilities, a separate network could be used for each of the Nout\nconvolution filters (i.e., one MLP instead of one discrete Nin x\nkernel_size filter). Using a single MLP likely creates dependencies\namong the Nout filters, is there a justification for this choice?\n", "summary_of_the_paper": "This contribution adapts to sequential data the paradigm of continuous\nconvolution kernels in convolutional neural networks (CNNs). While\nusual kernels are an extensive list of weights (one for each\nposition), the introduced CKConv frames these kernels as continuous 1D\nfunctions, parameterized by a small multilayer perceptron. This\nparadigms allows for wider kernels with fewer parameters, making it\npossible to deal with long-range dependencies without\nrecurrence. Several experiments are provided and show that the\nresulting CNN achieves state of the art performances on a variety of\ntasks, and in particular is able to deal with irregularly sampled\ndata.\n", "summary_of_the_review": "The contribution is original and could have a large impact both because it leads to better performances in several important situations (including long range dependencies) and because it could inspire new designs. The proposed method is thoroughly evaluated.\nThe submission is also clear and well written.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "sMa5pQXsu2L", "reviewer": "Reviewer_nLYG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "# Review\n## Clarity and novelty \nThis is overall a well-written paper, which clearly discusses the limitations of popular sequence models such as CNNs and RNNs, and explains how the proposed CKConv model is able to provide improvements in those aspects. The related work seems to be adequately cited. The main idea, which is to represent the convolution kernel implicitly via parametrizing it by a neural network, is not novel in the literature, but it has not been applied previously for sequential data modelling as far as I know. \n***\n## Strengths\nThe main value of the paper for me is in:\n1) adapting the continuous kernel convolution model to the sequential data context\n   + the idea of using a CKConv with a global memory horizon seems particularly useful, remedying the problem of not being able to learn long-range relationships, typically associated with RNNs and fixed horizon (shallow) CNNs\n   + the ability to generate kernel weights for arbitrary relative offsets naturally also lends itself to learning from irregularly sampled data, which is non-trivial to address with classic approaches\n2) the effort put into the empirical evaluation\n   + observing that convolution kernels might naturally contain high-frequency oscillations, which are difficult to learn via common activations, and identifying MLPs with Sine activations as a suitable candidate for representing the kernel\n   + carrying out the empirical evaluation across a range of tasks, which show strong performance of the proposed model\n***\n## Drawbacks\nMy criticism is regarding some of the claims and conclusions drawn. See these points below.\n\n**Insensitivity to resolution changes**\nIt seems that the relation in equation (5) does not exactly support that the model is robust with respect to larger resolution changes. This is also shown in Table 5, where the performance degrades significantly if there is an order of magnitude difference between training and testing resolutions. This is of course because the discretized convolution in eq.(2) is not a proper discretization of the integral in eq.(1). In particular, if we consider a high-frequency limit (that is, sampling a continuous signal at increasing frequencies), then the summation in (2) does not converge. I wonder why the authors chose to omit the normalization from eq.(2)? They could have normalized by the number samples (or equivalently, by the sampling rate) for equispaced samples, or for irregular data simply multiply the summand by the local step size to obtain Riemann sums that are actually convergent, and hence mapping that is continuous-time convergent and invariant to resolution changes, which it is not right now.\n\n**Asynchronous sampling**\nIt seems to me that regarding asynchronously sampled data, the model formulation is equivalent to combining zero imputation for the missing values with concatenating an observation mask to the input as extra channels. This is a valid approach, but not specific to the model at hand.\n\n**Activations in $\\texttt{MLP}^\\psi$**\nI found the discussion regarding the activation functions in the kernel parametrization quite intriguing. In particular, the experiments shown in Figure 4 coincide with the findings of \"On the Spectral Bias of Neural Networks\", Rahaman et al. 2019, where it is observed that NNs with ReLU activations trained under GD are biased towards learning low-frequency functions. It is also an interesting finding that the Sine activation seems to be able to overcome this limitation in this setting. But then one of the main points the paper tries to communicate seems to be that _Neural networks parameterizing spatial functions should use Sine nonlinearities_, which in my opinion paints too dark of a picture about ReLU and its companions. Especially so, as there have been approaches proposed in the literature to circumvent this limitation without changing the activation itself, see \"Fourier Features Let Networks Learn\nHigh Frequency Functions in Low Dimensional Domains\", Tancik et al. 2020. Without properly testing out if these techniques do help the usual activations, the aforementioned conclusion stands on shaky grounds. Especially so, since ReLUs are well-studied in the literature, while Sine nonlinearities in NNs (i.e. SIRENs) seem to be somewhat of a wildcard. In particular, by the discussion in App. E.3 I am left wondering if SIRENs might be, on the other hand, prone towards learning (unnecessarily) high-frequency functions? \n\nConcretely, I would be curious to know how the conclusions of the experiment in Figure 4, and the ablation study in App. D.1 change if the relative time offset $\\Delta \\tau$ is first passed through a positional encoding or Random Fourier Feature mapping as done in the previously referenced paper. Furthermore, in case any of these preprocessings do help the \"standard\" activations, it would be also good to know if the learned functions still contain the high-frequency oscillations beyond the input time grid as mentioned in App. E.3, or if these end up being smoother?\n\n**Modelling long-range relationships**\nThe authors have noted that dilated convolutions are able to model long-range interactions. I wonder why it wasn't included among the experiments? Clearly, to achieve a larger receptive field, dilated convolutions stack many layers, which means that the inputs are passed through several nonlinear activations before reaching the final representation the prediction layer receives. On the other hand, the CKConvs can increase the receptive field while keeping the model shallow. The benefits of this are not really clear at the moment, and whether the way the sequential information is processed in the dilated model (by passing it through \"extra depth\") has adversarial effects for the quality of the results.\nAdditionally, self-attention models are also able to capture long-range dependencies in their input sequences, so it would be worthwhile to include a discussion about them, and compare against it on more experiments.", "summary_of_the_paper": "The paper introduces convolutions with continuously parametrized kernels for sequential data. Continuous kernel convolutions, CKConv for short, parametrize the kernel associated to a convolutional layer as a continuous mapping, $\\psi: \\mathbb{R}^+ \\rightarrow \\mathbb{R}^{N_{out} \\times N_{in}}$, from the relative time offset $\\Delta\\tau \\in \\mathbb{R}^+$ to a weights matrix $\\mathbf{W} \\in \\mathbb{R}^{N_{out} \\times N_{in}}$, i.e. $\\psi(\\Delta \\tau) = \\mathbf{W}$. This is in contrast to traditional CNN approaches, which represent the convolution kernel as an explicitly learnable set of weights over an a-priori fixed receptive field.\n\nThe corresponding CKConv operation can be applied\n1) over arbitrarily long receptive fields, including a global one over the history of the input sequence, without the heavy memory burden incurred by representing the convolution kernel explicitly as a set of weights along the horizon;\n2) to sequences that are sampled irregularly or asynchronously, since the admissible set of relative positions are not fixed in advance.\n\nA relation is given, which shows approximately how the CKConv changes under resampling the input sequence (the output gets rescaled by the ratio of the sampling rates), which is suggested by the authors to support well-behavedness under varying input resolutions during training and/or between training and testing. Afterwards, a discussion is given about the parametrization of the kernel. Specifically, various MLP models are considered with different activation functions, such as, ReLU, LeakyReLU and Swish, and it is demonstrated that these MLPs are unable to learn functions with high-frequency oscillations, while a recently introduced Sine activation is able to. \n\nThe experimental evaluation includes 1) classic stress tests for RNNs, 2) discrete sequence tasks (sMNIST, pMNIST, sCIFAR10), 3) time series modelling (CharacterTrajectories, Speech Commands) , where additionally robustness to missing/irregularly-sampled data and resolution changes are also investigated. The proposed model overall performs favourably to common alternatives; and it seems robust with respect to missing data, and moderate resolution changes.", "summary_of_the_review": "I am in favour of the paper mainly due to the compelling empirical performance, which should be of interest to the community, although some additional baselines and experiments could have been included.  It is clearly written with figures and illustrations to aid the reader. The main idea is not novel, but it has not been adapted to handling sequential data before, and it seemingly has many benefits in this context.  My criticism is regarding some discussions in the paper, and that some properties of the model seem to be overstated.\n\nPOST-REBUTTAL: Given the comprehensive answers given by the authors, and the implemented changes in the paper, I am inclined to raise my rating to an 8 from the previous 6.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "p91R48lO0L", "reviewer": "Reviewer_DZHd", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:** \n- Interesting concept of representing arbitrary-sized convolutional kernels.\n- The paper is well written, and the method's limitations are outlined clearly\n- Great evaluation of how a SIREN-based parametrization is best suited for realizing the MLP. \n\n**Cons:**\n- No code is provided during the review (although the authors claim to release it)\n- The experimental evaluation mostly consists of equidistantly sampled fixed-length time series (sMNIST, etc.), i.e., data for which CKConv provides only little benefits over standard CNNs.\n- Experimental evaluation with attention-based architectures on irregularly sampled variable-length sequences datasets, e.g., PhysioNet, MIMIC-III, and the Human Activity dataset, from Shukla et al. 2021 would have demonstrated the advantages and limitations much better.\n\n**Questions**:\n- How was the sequence reduced to a single prediction in the sequence classification tasks (sMNIST, etc.)?\n- The paper Gu et al. 2020 \"HiPPO: Recurrent Memory with Optimal Polynomial Projections\" was published at NeurIPS. Please update the reference.\n\nShukla et al. Multi-Time Attention Networks for Irregularly Sampled Time Series. ICLR 2021", "summary_of_the_paper": "The paper proposes continuous convolutional kernels parametrized in the form of an MLP. The MLP gets the relative time as input and outputs the column of the convolutional kernel at the given relative time. The authors show that Sine-based (SIREN) non-linearity is best suited for the kernel generating MLP, and an experimental evaluation is performed.", "summary_of_the_review": "I am in favor of acceptance as the concept of CKConv provides enough contribution, although the experimental evaluation would have been stronger with standard irregularly time-series datasets (PhysioNet, MIMIC-III)", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "2cn0JbChIQv", "reviewer": "Reviewer_Lwkz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This is a nice paper with one clear idea, which is demonstrated to work well. I like your detailed look at activation functions -- I think you should have a look at \"Gegenbauer Polynomials\". A potential weakness of the paper is it's limited set of experiments: Compared to the 'baseline' paper Bai 2018a, several datasets are missing, and, given the progression of time, several larger-scale datasets e.g. from speech or the financial domain could be added. ", "summary_of_the_paper": "The paper considers convolutional networks for sequence processing and suggests to parameterize convolutional kernels in this setting by small neural networks. These neural networks are similar to implicit networks in the sense that its input represents a time difference to the currently evaluated time point in a sequence. For each time difference, the associated weights of the convolutional kernel are computed.\n\nThis allows to handle arbitrarily large convolutional kernels (albeit the desired size needs to be determined somewhat deterministically apriori), to handle irregularly sampled data and to handle data at different resolutions. Good results on a set of experiments show the practical validity of the idea. The paper does a larger study into the importance of the type of activation functions. \n\n", "summary_of_the_review": "The described idea is a nice transfer from existing work of implicit neural network representations and continuous kernel formulation. Showing the importance of sine activation functions is very helpful to direct attention to the role of activation functions, depending on the actual modeling task. The empirical validation should be larger, though.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper introduces a convolution where the kernel is parametrised continuously over time (in the context of recurrent networks) to address vanishing gradients issues, by using another neural network to generate the kernels.
This is a meaningful idea, addressing an important problem.
The paper is well written and clear. The idea is novel (parametrised kernel already exist, but the way it's used here is new).
The experimental section is solid, although some reviewers suggests it could be extended with more baselines.
All reviewers recommend to accept the paper, therefore I also recommend accept.
Decision: Accept (Poster)