Modelling Long Range Dependencies in $N$D: From Task-Specific to a General Purpose CNN
Authors: David M Knigge
David W. Romero
Albert Gu
Efstratios Gavves
Erik J Bekkers
Jakub Mikolaj Tomczak
Mark Hoogendoorn
Jan-jakob Sonke
Venue: ICLR 2023 poster (2023)
Keywords: convolutional neural networks
continuous convolutional kernels
CNNs
continuous parameterizations
sequential data
visual data
point-cloud data

Abstract: Performant Convolutional Neural Network (CNN) architectures must be tailored to specific tasks in order to consider the length, resolution, and dimensionality of the input data. In this work, we tackle the need for problem-specific CNN architectures. We present the Continuous Convolutional Neural Network (CCNN): a single CNN able to process data of arbitrary resolution, dimensionality and length without any structural changes.  Its key component are its continuous convolutional kernels which model long-range dependencies at every layer, and thus remove the need of current CNN architectures for task-dependent downsampling and depths. We showcase the generality of our method by using the same architecture for tasks on sequential ($1{\rm D}$), visual ($2{\rm D}$) and point-cloud ($3{\rm D}$) data. Our CCNN matches and often outperforms the current state-of-the-art across all tasks considered.

Published as a conference paper at ICLR 2023
MODELLING LONG RANGE DEPENDENCIES IN ND: FROM TASK-SPECIFIC TO A GENERAL PURPOSE CNN
David M. Knigge*: David W. Romero x: Albert Gu Efstratios Gavves', Erik J. Bekkers' Jakub M. Tomezak', Mark Hoogendoorn' Jan-Jakob Sonke* University of Amsterdam 2 Vrije Universiteit Amsterdam Stanford University
Netherlandds Cancer Institute
d. m. wn..kigggeeuv.nl, d. w. www.roguuuaaaaaammmooooo vu.nl n1
ABSTRACT
Performant Convolutional Neural Network (CNN) architectures must be tailored to specific tasks in order to consider the length, resolution, and dimensionality of the input data. this work, we tackle the need for problem-specine CNN architectures. We present the Continuous Convolutional Neural Nenwork (CCNN): single CNN able to process data arbitrary resolution, dimensionality and length without any structural changes. Its key component its continuous convolutional kernels which model long-range dependencies at every layer, and thus remove the need of current CNN architectures for task-dependent downsampling and depths. We showcase the generality of our method by using the same architecture for tasks on sequential (1D), visual (2D) and point-cloud (3D) data. Our CCNN matches and often outperforms the current state-of of-the-art across all tasks considered.
INTRODUCTION
The vast popularity of Convolutional Neural Networks (LeCun et al., 1998) (CNNs) is a result of their high performance and efficiency, which has led them to achieve state-of-the-ar in applications across sequential (Abdel-Hamee et al., 2014; Van Den Oord et al., 2016), visual (Krizhevsky et al., 2012; Simonyan & Zisserman, 2014) and high-dimensional data (Schütt al., 2017; Wu et al., 2019). Nevertheless, a major limitation of CNNs -nd other neural networks- that their architectures must be tailored to particular applications in order consider the length, resolution and dimensionality of the input data. This has led to a plethora of task-specific architectures (Oord et al., 2016; Bai et al., 2018; Simonyan & Zisserman, 2014; Szegedy al., 2015; Ronneberger al., 2015; He al., 2016; Qi et al., 2017; Wu et al., 2019) which (i) hampers the selection of the most appropriate architecture for particular task, and (ii) obscures the transfer and generalization of insights across applications. In this work, we tackle the need for problem-speciff CNN architectures and propose a generic CNN architecture that can be used independent of the length, resolution and dimensionality of the data.
CNN architectures are data dependent. Current CNN architectures are task-specific because they are tied to the length, resolution, and dimensionality of the input. The length of the data varies from task to task e.g. audio fragments may span milliseconds minutes. This requires carefully chosen
Equal contribution.
Our code is publicly available at g thub. com/ davi d-kn ogge/conn.
(x)) Kermal x
Discrete kernel
Kermel
weights
Continuous kernel
weights kermel values
mag
Tied to kernel size
Arbitrary kernel size
Tied to resolution
Resolution agnostic
Tied dimensionality
Applicable to any
dimensionality
Xi
X;
Figure 1: Discrete and continuous convolutional kernels. Discrete convolutional kernels assign weight w; out discrete set of weights W to a relative offset x. This ties the kernel to the length, resolution and dimensionality of the input, limiting the general applicability of the CNN architectures. Instead, our Continuous Convolutional Neural Network parameterizes kernel values as continuous function (Kerree the input domain IRd, which decouples it from data characteristies.

---

Published as a conference paper at ICLR 2023
Pioudd
3=3
Kernel
Viorre
(ka)
Pland
Phimelty)
(a) Kernel network.
(b) D sequences.
D-) images.
(d) D=3, volumes.
Figure 2: Continuous convolutional kernels: the key to unified CNN architecture. The continuous
parameterization of convolutional kernels used in this work consists of a small kernel network @kernel that receives coordinates as input and outputs the value of the convolutional kernel at that position (2a). By changing the dimensionality of the coordinates x,, the same kernel network can render convolutional kernels for sequential (2b), visual (2c), and higher dimensional data (2d).
strides and pooling to capture relevant dependencies across the entire input (Van Den Oord et al., 2016; Lee al., 2017). In addition, physical signals, e.g., audio, images, are continuous in nature. As such, their semantic meaning is independent of the resolution at which they are sampled, e.g., the same audio may be expressed at different resolutions. Nevertheless, current CNN architectures are resolution-bound and thus different resolutions require different CNNs. These limitations aggravate when considering mulii-dimensional data, Each input dimension can be defined at different lengths and resolutions, e.g., video, rectangular images, and each data modality brings its own conventions for each of these properties, c.g., the resolution a second of audio (16kHz) (Warden, 2018) strongly contrasts with that of images (32 x 32) (Krizhevsky et al., 2009).
Towards a unified CNN architecture. As discussed in Sec. 3, the core component that makes CNNs data-dependent are their discrete convolutional kernels. Convolutional kernels are implemented via one-to-one mapping between kernel values and model parameters (Fig. left), which (i) binds them to the input resolution and length, and (ii) makes them ill suited to model long-range dependencies. The latter results from the large number of parameters needed to construct large convolutional kernels. This is why standard CNNs favour using local kernels in combination with task-dependent depths and pooling layers to model long-range dependencies, at the cost of making them task lask-dependent.
The need for continuous parameterization. To overcome task-dependent architectures, it is crucial to define a kernel parameterization that decouples parameter count from kernel size. Following Schütt et al. (2017); Romero et al. (2022b), we use a small neural network define continuous mapping from positions to the value of the kernel at those positions. The resulting Continuous Convolutional Kernels (Fig. 2), allow for the construction of convolutional kernels of arbitrary size in parameter efficient manner. Consequently, the same convolutional layers -and thus the same CNN- can be used regardless of the input length, resolution and dimensionality. We leverage this formulation to construct the Continuous Convolutional Neural Network (CCNN): single CNN architecture that can be applied regardless of the input length, resolution and dimensionality.
Empirical results. To showcase the proposed CCNN, we deploy the same CCNN for several tasks on sequential (ID), visual (2D) and point-cloud (3D) data. Our CCNN matches and often outperforms the current state-of-the--e ahrt across all tasks considered. Importantly, the continuous parameterization of our CCNN allows it to handle irregularly sampled data natively. As a result, the CCNN is not restricted grid data, e.g., 3D voxels, and can be used on point-clouds directly.
Contributions:
We propose the Continuous Convolutional Neural Network: general purpose CNN architecture able to process data of arbitrary resolution, dimensionality and length without structural changes.
We study the layers of CNNs, and demonstrate that the ability to model long-term dependencies on ND without the need of input dependent downsampling and depth values necessary and sufficient for the construction of a general purpose CNN architecture.
order to model long-term dependencies on ND without input dependent downsampling and depth values, utilize and improve the Continuous Kernel Convolutions of Romero et al. (2022b). Our proposed improvements allow the proposed Continuous CNN to achieve good empirical results on the tasks considered in ID, 2D and 3D without structural changes.
2 RELATED WORK
An extended section with extended comparisons related works is provided in Appx. A.
2

---

Published as a conference paper at ICLR 2023
Local Operations
Globel Operations
Figure 3: Operation types: global, pointwise and local. Local op- crations are resolution dependent. Transferring local operation from (3a) to a lower resolution (3b) leads an increased receptive field.
Operations
(a)
(b)
General purpose architectures. To the best of our knowledge, the only existing method aiming for a general purpose architecture is the Perceiver (Jaegle et al., 2021), which uses a Transformer to lift restrictions regarding data characteristics and modalities. However, of their size- to a small latent representation to reduce the quadratic complexity of self-attention, (i) decouples the depth of the network from parameter count via recurrence, which requires tuning the number of unrolling steps per task, and (iii) must use absolute positional encodings to encode the data structure, which break the translation equivariance of the self-attention operation (Romero & Cordonnier, 2020). In contrast, CCNNs provide general purpose convolutional method that: (i) scales much more favorably than self-attention, (it) does not require a constant small latent representation, (iii) does not require task-dependent depths, and (iv) preserves translation equivariance.
must map inputs regardless
Long range dependencies in ND. Based on our analysis (Sec. 3), any architecture able to model long range dependencies in ND without the need of input-dependent pooling or depth could be used as a general purpose architecture. To our best knowledge, the only existing convolutional methods able construct global convolutional kernels are CKConvs (Romero al., 2022b;a) and state-spaces (Gu et al., 2021; 2022). However, state-spaces rely on complex dynamical systems that are not easily defined in ND aside from the combination of 1D systems, equivalent representing ND kernels as combinations of N independent 1D kernels- Nguyen et al. (2022). Consequently, we select CKConvs as the building block our approach given their advantages in terms of expressivity and simplicity.
Convolutional kernels as neural networks. Modelling convolution kernels with small neural net- works that map kernel positions kernel values showed promising results for small kernels (Jia et al., 2016; Schütt al., 2017; Wu et al., 2019). Subsequently, Romero et al. (2022b) realized that this pa- rameterization decouples the size the convolutional kernel from the number of parameters required to construct it, and thus can be used construct arbitrarily large convolutional kemels in parameter efficient manner. In this work, we show that this parameterization allows for the construction of a single CNN that can be used regardless of the input length, resolution and dimensionality.
Romero et al. (2022b) realized that the piece-wise wise MLPs used so far to parameterize convolutional kernels were unable to model complex long range dependencies due their spectral bias (Tancik et al., 2020), and showed that implicit neural representations -specifically SIRENS (Sitzmann et al., 2020) could be used to solve the issue. Subsequently, Romero et al. (2022a) parameterized their convolutional kernels with Multiplicative Anisotropic Gabor Nets (MAGNets), which provided them control over frequencies admitted in the convolutional kernel, thus preventing aliasing and aiding generalization across resolutions. In our work, we use FlexConvs parameterized by MAGNets. However, we observe that neural networks used to parameterize convolutional kernels are not correctly initialized for that purpose, and propose an initialization that solves the issue.
FROM DATA-DEPENDENT TO DATA-
CNN ARCHITECTURES
In this section, we study the components of CNN architectures and pinpoint the changes required in order to construct CNN architecture independent of input lengths, resolutions and dimensionalities.
3.1 POINTWISE OPERATIONS: LINEAR LAYERS, DROPOUT, POINTWISE NONLINEARITIES AND RESIDUAL CONNECTIONS
Pointwise operations are operations applied to each spatial element of the input separately (Fig. 3), e.g., pointwise linear layers, dropout and pointwise nonlincarities. As such pointwise operations do not depend on the input shape and model the same function regardless input length, resolution and dimensionality. Learnable parameters of pointwise operations, e.g., in pointwise linear layers and
3

---

Published as a conference paper at ICLR 2023
Parametrie ReLU (He et al., 2015), are shared over the spatial domain; the same set of parameters is applied to imputs of any spatial shape.
Conclusion, Based on the previous observations, used without changes in order to construct seneral-purposs CNN architecture.
conclude that pointwise operations can be
3.2 GLOBAL OPERATIONS: NORMALIZATION LAYERS AND GLOBAL POOLING
Global operations aggregate all spatial elements of the input for their processing, e.g., normalization layers, global pooling. As such, common global operations do not depend on the specific shape of the input signal and their effect is equivalent regardless of the input length, resolution and dimensionality. It is important to note, however, that common global CNN operations only define learnable parameters along the channel axes c.g., the scale and mean parameters of a normalization layer have shape Ni,, Nevertheless, one were to define global operation for which spatial positions are also assigned weights, the shape of the parameters would be dependent on the input length, resolution and dimensionalty, and thus the previous statement would no longer hold.
The previous observations indicate that global operations that only define channel learnable parameters can be used without changes in order to construct a unified CNN architecture. Luckily, this is the case for all common global operations used in CNN architectures."
Conclusion.
3.3 LOCAL OPERATIONS: CONVOLUTIONAL LAYERS AND SUBSAMPLING
Local operations are operations that rely on spatial portions of the input and are applied across its spatial dimensions, e.g., convolution, subsampling.
practice, the neighborhoods on which these operations are applied are hyperparameters, e.g., 3 3 kernels or pooling over 2 x 2 regions, and are selected based on the input size. Unfortunately, if the resolution of the input changes then the portion of the input that the operation considers changes and the effect of the operation changes (Fig. 3). Similarly, if the length of the input changes, then larger neighborhoods are required in order to model dependencies across the imput. These effects exacerbate if one considers multi-dimensional inputs, as different dimensions might require different neighborhood sizes. Consequently, we can say that local operations depend on the resolution, length and dimensionality of the imput.
possible solution would be to adjust the size of the operations proportional to resolution and length changes. Unfortunately, the local operation defines learnable parameters over its spatial dimensions -as in (discrete) convolutional kernels (Sec. 3.3.1), then increasing the size of the convolutional kernel is tied to a proportional increase in the number of parameters required construct it. This, in turn changes the learning dynamics of the network and easily becomes prohibitive.
The need for continuous parameterization. better solution results from using a parameterization in which the number of parameters is independent from the size of the kernel. By doing so, the same parameterization can be used independently from the input length, resolution and dimensionality.
3.3.1 FROM DATA-DEPENDENT TO DATA-INDEPENDEN CONVOLUTIONAL LAYERS
Conventional CNNs implement a discrete version of the convolution operation defined as:
(f (f+ S f(x f(x-ffxxx-ee (xxx))x(XX)) (x)dx o [o,..,,Nout)
where R:RP-R is a B-aD--a -immeeniional input signal with Nin channels, and k: -R.--NNNN is set of Nout convolutional kernels. For the types of data CNNs are commonly used for, e.g. images, the signal f is generally sampled on a discrete grid of equidistant points, and thus it can be described as a function ZD N.., In practice, the signal f non-zero only on a finite subset of the grid (2(f) ZD with limits given by the range of sampling, e.g., height and width of an image. Accordingly, the convolutional kernel k is defined over the same grid of coordinates, of which generally a subset 12(k) I(f) maps to nonzero values. Since both (2k) and P(f) are discrete and finite, the convolution can be implemented as the inner product of function and kernel values at each position:
(f (x)
Σ xx)))) xx
(2)
se((k)
Discrete convolutional kernels tie convolutional layers to data characteristics. Conventionally. the convolutional kernel & is implemented through discrete set of randomly initialized weights W. of which each entry w, R N NNo corresponds to point x, E ((k). Consequently, an increased
"Global discrete convolutional kernels could be interpreted as global operations for which parameters along the spatial dimensions of the input are defined. Consequently, these require special treatment (Sec. 3.3).
4

---

Published as a conference paper at ICLR 2023
fin
(1.0.05))
PKrrnll
Pitenalx (x-1)
( 0.5, 1.0]
PKemm((x-8) 2,)
fin Kerusl (x)
(a)
(c)
Figure 4: Applying a Continuous Kernel Convolution. Given a pixel-position in the input image x e
Q()). we obtain relative offsets to surrounding pixels (x-Xi x) SE((() (4a). Next, we pass each relative position to the kernel generator network in order to generate the kernel: {yKemel (x x)) x))())))))))))))))))))))()) (4b). Finally, we apply the convolution between the generated convolutional kernel and the input (4c).
kernel size is reflected in larger set (Kk), and thus in a correspondingly larger weight matrix W. This directly ties model's parameter count to its kernel size.
In order to model the long-range dependencies needed to extract high-level features in parameter- efficient way, we must then resort to pooling operations and the stacking of layers that implicitly increase the receptive field of the kernel. This turn, makes CNNs effective only on inputs of a certain size. For example, if we apply network created to model long-range dependencies over images of size 256 x 256 to images of size 32> x 32, then intermediary pooling layers would make the spatial extent of the feature maps collapse long before all convolutional layers applied. Similarly, we use a network designed to model long-range dependencies over images of size 32 x 32 to images of size 256x x 256, then the network will not be able to model long-range dependencies in the input.
Additionally, the definition of the kernel & through a discrete set of weights W ties the model to a given input resolution. Yet, for many applications, the input f is discretization of an underlying continuous function. Therefore, we would like our model to provide the same responses regardless of the resolution which is provided. As discrete convolutional kernels live in discrete domain, they cannot be easily represented at other resolutions. In fact, one can show that discrete CNNs do not generalize unseen resolutions Romero et al. (2022a); Nguyen al. (2022). Consequently, is not possible to reliably apply trained discrete CNNs across resolutions.
In addition, note that in Eq. 2, the same discrete convolutional kernel & can be used every location x é S)) only because the input signal f is defined over an equidistant grid, and the values of the discrete kernel k(x) align with the features f (x). Vx 9)f)). This not the case for irregular data, Consequently, discrete kernels are ill-suited to handle irregular data. With weights fixed to relative positions, an infinite number of weights would be needed to cover any continuous domain.
These limitations suggest better approach to model convolutional kernels: using the model weights to parameterize as a continuous function over the data domain Rd.
A data independent parameterization. To obtain a formulation for a CNN applicable to arbitrary resolutions and sizes, we require a parameterization for convolutional layers that is invariant to the set {(f) over which f is sampled. In other words, we must find a parameterization with which the kernel can be modelled over the underlying continuous domain of the input signal, i.e., Rd. Moreover, to avoid models with different parameter count for different resolutions, it necessary that the parameterization of the kernel decouples its parameter count from the size of the kernel.
Such a parameterization provided by Continuous Kernel Convolutions (CKConvs) (Romero et al., 2022b;a). CKConvs provide a continuous parameterization for convolutional kernels by using small neural network Keernll RDD Now N as a kernel generator network. This network maps coordinates in the domain of the kernel x; TR. to the values of the convolutional kernel at that position: (xx) E IR.- Nour (Fig. 1). A Continuous Kernel Convolution is illustrated in Fig. 4.
Since the parameter count of the kernel generator network is independent from the number of points in the neighborhood that determines the size of the kernel, CKConvs allow for construction of arbitrarily large kernels without increasing the parameter count of the layer. Romero et al. (2022b) shows that
5

---

Published as a conference paper at ICLR 2023
large kernels allow CNNs to model long range spatial dependencies at every layer. thus removing the need for downsampling and stacking of layers to increase receptive fields. This in turn allows us to build architecture which does not contain resolution-, dimensionality.. and size-dependent layers.
Conclusion. The previous observations indicate that local operations equipped with existing pa- rameterizations are tied to the length, resolution and dimensionality of the input. Nevertheless, this limitation can be lifted if an alternative parameterization is used that detaches the neighborhood of action of the operation from the length, resolution and dimensionality of the input.
A GENERAL PURPOSE CNN ARCHITECTURE
Starting point. In principle, the CNN architectures with CKConv (Romero al., 2022b) and FlexConv (Romero et al., 2022a) introduced previously fulfill the require- ments posited in Sec. 3. Nevertheless, these papers, these architectures still must be tailored to specific applications and domains in order perform well, c.g., TCN (Bai et al., 2018) and ResNet (He et al., 2016) backbones for sequential and visual tasks, respec- tively in Romero et al. (2022a).
CCHNBlock
SepFLexCone
BatchNorn
depicted in
Batchhorn
GELU
SepPlexCon
GELU
xL
CCNNBlock
Dropout
BatchNorn
Pointxisel.inear
In order construct single performant general purpose architecture that works well across all tasks considered, we start from a FlexNet (Romero al., 2022a), and pro- pose several structural changes (Sec. 4.1). The resulting CCNN architecture is shown Fig. 5.
GlobalAvgPoo1
Linear
Figure 5: The CCNN architecture.
4.1 MODIFICATIONS AND IMPROVEMENTS
Proper initialization the kernel generator network (eKmmll-- First, we analize the kernel generator network (Kernll, and observe that it is not initialized properly in previous works (Schütt et al., 2017; Wu et al., 2019; Romero et al., 2022b;a) for the purpose parameterizing convolutional kernels.
Recall that, in order ensure training stability it is desirable to retain constant (unitary) variance throughout the activations of neural network (Glorot & Bengio, 2010). Hence, the discrete weights W conventionally used to construct convolutional kernel are initialized have variance inversely proportional to the number of elements over which the convolution is computed, i.e., the number of pixels [[((k) times the number of channels Nin For instance, He initialization (He et al., 2015) initializes W s.t. Var (W))="")>/NN (NNnn((((()()))))) with a gain that depends on the nonlinearity used.
In related works, the networks used to parame- terize convolutional kernels are themselves ini- tialized to preserve a constant (unitary) vari- ance throughout the network. Consequently, when used as kernel generator network, stan- dard initializations lead the generated kernel to have unitary variance, i.e., Var(k)=1. This in turn, make CNNs using neural networks to parameterize their convolutional kernels expe- rience layer-wiss growth in the variance of the feature maps proportional to Ni, |[((k) This growth is of particular importance for kernel generator networks that generate large convolutional ker- nels, i.e., with large [.=)). For instance, we observe that the logits of CNNs with CKConvs (Romero et al., 2022b) and FlexConvs (Romero et al., 2022a) lie in the order of le upon initialization: an undersirable property that might lead unstable training and a need for low learning rates.
Figure 6: Histogram of the output of a CCNN with and without the proposed kernel initialization.
To address this issue, we must ensure that the variance at the output of the kernel generator network inversely proportional to N,, |((k)). Inspired by Chang al. (2020), we therefore re- re-weight the last linear layer of the kernel generator network @Krrnll by g²/ //NNu [[2(k)] With this modification, we observe that the variance of the generated convolutional kernels satisfies the desired constrains, and consequently, the logits of our CCNN show unitary variance upon initialization (Fig. 6).
Depthwise Separable Continuous Convolutions. Separable convolutions have long been used to improve the parameter and computational efficiency of CNNs (Rigamonti et al., 2013; Sifre & Mallat, 2014). Recent architectures have leveraged separability, and found CNNs with separable kernels to

---

Published as a conference paper at ICLR 2023
perform better than CNNs with conventional convolutions (Knigge al., 2021; Liu et al., 2022). This phenomenon results from the separation of spatial and channel dimensions, which allows for wider networks without additional computational and parameter complexity.
Based on these observations, we construct a depth-wise separable version of FlexConv (Romero et al., 2022a), in which channel-wise convolution is computed with kernel generated by kernel generator network (Kernel IRD - R Nia, followed by a pointwise linear layer from Njn to Nout dimensions. Separable FlexConvs allow for the construction of a much wider CCNN -from 30 to 140 hidden channels- with the same parameter and computation complexity.
An improved residual block. Residual connections (He et al., 2016) provide training stability and improved performance. A residual block OR (f) (f) f is defined as the sum between the input and so-called residual connection & composed of multiple layers: convolutional, normalization, etc.
+
Recent studies found improvements over the residual block of He et al. (2016) by changing the nonlincarities as well as the position and type of normalization layers within the blocks (Xiong et al., 2020; Liu et al., 2022). Based on these advances and empirical evidence, blocks of Romero et al. (2022a) with resid- ual blocks similar to those of S4 (Gu al., 2022) complemented with a nonlinearity at the end of the block. A comparison of our residual block and those in Gu et al. (2022) and Romero et al. (2022a) is given in Fig. 7.
PlassWiStock
FlexsDesy
StBlock
CCHBBlckk
Layertorm
BeLU
Bateialora
Trogowr
SALayer
Sep Fll Succonn
GEUJ
modify the residual
FlexDonr
Gropout
Drepout
LayezHorm
Pulntwlectiinea
ReLL7
GSLD
Dropant
Figure 7: Residual blocks used in FlexCCNNs (Romero et al., 2022a), S4 (Gu et al., 2022) and CCNNs (ours).
L2 regularization of continuous kernels. Weight decay penalizes the Le norm of learned kernel values by adding the norm of the weights W an additional loss term (Krogh & Hertz, 1991). Since continuous kernels are not directly parameterized by weights W, applying La regularization on the parameters of Keruel directly would not have the intended effect. Consequently, in order to produce the same effect, we amend the L2 regularization term to penalize the generated convolution kernel instead. Given PPkkerel the set of generated kernel for all channels at layer I, Lobj the objective loss function, and À the regularizing parameter, we define a regularized loss as:
1 Σ
L= = = Lreg Lobs + A
l1
(3)
2 1-
EXPERIMENTS
We aim to define an architecture that can be applied regardless of the specific data characteristics. To this end, we construct a CCNN and validate it on sequential (ID), visual (2D) and point-cloud (3D)
datasets -see Appx. B for a detailed description of each dataset- We show that the same CCNN obtains state-of-the-ar results several sequential tasks, competitive performance and surpasses the Perceiver on point-cloud processing. In addition, we show that learned CCNNs generalize from regular to irregular data by transferring trained CCNN from voxels to point-clouds.
image tasks,
Architecture specifications. To study the scalability of our method, we create two CCNNs with differing numbers of hidden layers sizes: CCNN4,140 (4 blocks, 140 channels, 200K params) and CC0N6 (6 blocks, 380 channels, 2M params). The kernel network @eeerel is chosen to be a 3-layer MAGNet (Romero et al., 2022a), with 32 hidden units for CCNN and 64 hidden units for the larger CCNN6,350. In each task, the input dimension of the kernel network Kerrel corresponds to the dimensionality of the data 1 for sequences, 2 for images, and 3 for point-clouds- Additional details regarding hyperparameters, training regimes and experimental settings are given in Appx. C. An empirical assessment of the computational efficiency of our architectures is given in Appx D.
CCNNS on sequential datasets. In 1D we consider Sequential MNIST, Permuted MNIST (LeCun et al., 1998), Sequential CIFARIO 10 (Krizhevsky et al., 2009), the Long Range Arena (Tay et al., 2021) and the Speech Commands dataset (Warden, 2018). Our results (Tabs. 1, 2), demonstrate that CCNNs obtain state-of-the-an thettrte across several tasks, surpassing the performance of tailored architectures such as Recurrent Neural Networks and Transformers. The performance of CCNNs is explained by their ability to model long term dependencies with higher capabilities than tailored models. Interestingly, observe that on tasks defined over flattened 2-dimensional signals, e.g., SMNIST, SCIFAR 10,
7

---

Published as a conference paper at ICLR 2023
Table I: Experimental results on sequence, image and point-cloud datasets. x unable to apply, either due the model's 's inability to handle the specified data, or computational complexiry. N.A. not available.
SEQUENEE
IHAGE
POUN: CLOUD
MFGCH--FCC
SPEnCIL
SP
Traesforer
500s.
53.50
$7.50
62.30
90.75
LSSL.
7.5M4
$9:53
58.76
34.05
2.534
9953
98.10
91.13
54
300k
S.A.
N.A.
N.A.
9195
98.52
91.30
CXCNNN-See
95k
99:31
99:00
62.25
95.30
71.66
65.95
MCNN-Sco-lll
1M
99:32
400C
T1IS
Rocha-11
11:2M
16
TEA0
REM
6.15
26.54
N.A
Sein-DD
27.5M
N.A
CNN
58.5
Harilbanoon CNN
2645
59.3
64D
76.3
CRCNS
630S.
86.8
N.
N.A.
92.3
N.A.
HALES
925
pGL20
1.54M
890
PhaneNee
3-48M
1.45M
2002
99:22
58.93
00.30
05.03
96. 54
57.33
92. 72
36.86
$1.00
$1.44
DCNN
24
99:22
98.94
93/08
91:95
90.44
96.44
98.20
73.15
83.00
$5.30
CCNNs learn to construct very large periodic kernels that model flattened local dependencies in 2D (Fig. 12). This showcases the ability of CCNNs to model meaningful long range dependencies.
Additionally, we evaluate the capacity of CCNNs to classify speech (Warden, 2018) both from prepossessed (Speech-MFCC state-of-the of-te-of-hheart results on both preprocessed and raw signals -of length 16000-, thus demonstrating the ability of CCNNs to process very heterogeneous data types with a unified architecture
and raw signals (Speech-Rave Raw). Our results (Tab. 1) demonstrate
Generalization across resolutions, CCNNs are defined on continuous space. Consequently, it is possible to train a CCNN at one resolution and deploy it at other resolutions -a feat not achievable with discrete models (Romero et al., 2022a; Nguyen et al., 2022). To showcase this ability, we train CCNNs on Speech-Raw, and test them on a subsampled version of the dataset (Speech-0.5x) CCNNs not only generalize, but outperform existing models on zero-shot prediction over resolution changes.
CCNNs on image datasets. In 2D, we consider the CIFARIO, CIFAR 100 (Krizhevsky et al., 2009) and the STL10 datasets (Coates et al., 2011). CCNNs outperform existing continuous convolutional vision architectures (Ruthotto & Haber, 2020; Tomen et al., 2021; Romero et al., 2022b;a) and are competitive with existing large scale discrete models, while being (much) smaller (Tab. 1).
The importance of modelling long range dependencies in ND. In principle, all tasks can be treated as sequential tasks ignoring the ND structure as done in S4 (Gu et al., 2022) for images due to the complexity of defining state spaces on 2D-, but this sacrifices structural information. Contrarily, CCNNs can be casily defined on multidimensional spaces simply by changing the dimension of the input coordinates of the kernel generator networks. We observe that by considering the 21) structure of the Image and Pathfinder tasks of the LRA benchmark, much better results can be obtained (Tab. 2, right). In PathFinder with 2D images, the CCNNG, 3,00 obtains accuracy of 96.00, outperforming the previous state-of-the-ar by almost 10% points and performing remarkably better than on flattened images. Additionally, we observe that models trained on the original 2D data converge faster than their sequential counterparts (Fig. 9). Finally. we remark that discrete 2D CNNs with small convolutional kernels, e.g., R8 (He et al., 2016), are unable to solve Pathfinder due the lack of fine- fine-grained global context resulting from intermediate pooling layers. This was also seen by Gu et al. (2022).
CCNNs on point-cloud datasets. An additional advantage that comes from the continuous nature of CCNNs, is that -onttrary to discrete CNNs- CCNNs can be seamlessly applied on irregular data, e.g., point-clouds. p.g.toonttsl To assess the performance of CCNNs in 3D, we consider the ModelNet40 dataset (Wu et al., 2015), which consists of 3D meshes of objects often treated as point-clouds (Wu et al., 2019). Our results show that CCNNs are able to achieve a decent overall classification accuracy in comparison to point-cloud specific architectures with relatively low parameter count (200 k). In particular, CCNNs outperform the Perceiver (Jaegle et al., 2021) while being much smaller (Fig. 8).
use CCNNs trained regular data to process irregular data. We showcase this ability by training CCNN on a version of ModelNet10 (Wu et al., 2015) voxelized onto a grid of 40x40x40 voxels -ig. 1 llb shows an example of the point-cloud and voxelized datasets-. After trained, deploy the CCNN on the original (point-cloud) and voxelized test sets and observe respective test accuracies of 90.900.00 and 90.64. This result shows that CCNNs learn representations that reflect the continuous
From regular irregular data. Interestingly, the continuous nature of CCNNs allows
8

---

Published as a conference paper at ICLR 2023
Table 2: Experimental results on the Long Range Arena benchmark. wabll to apply due to the
model's inability to handle the specified data.
LisTÖps
TEXT
IMAGE
PATHPINDER
AVG.
2DIMAGE
2DPATHRINDRR
Transformer
Reformer
37.27
56.10
38.07
50
30.56
BigBird
36.05
64,02
40.83
74.87
57.17
Linear Trans,
16.13
65.90
42.34
75.30
50.46
Performer
42.77
77.00
FNet
35.33
65.11
38.67
77 30
54.52
Nystromfirmer
37.15
65.52
41.58
70.94
57.46
Luna-256
37.25
59.37
44.85
63.59
87.62
91 36
76.86
89, 89.48
94,
43.60
84,08
88.90
91.51
77.02
91.12
96.00
0.9
0.950
PepuurFUU
Validation Accuracy
0.925
0.5
0.900
PointNet+
0.075
CCNN aB
DGCNI
PointNet
0.7
CCNNN CCNN,
CCNN
0.050
CCNN,
Percelve
0.6
Sequence
0.825
10k
250c
309
No: Model Parameterss (oo)
150
Figure 8: Parameter count versus performance on ModelNet40. Very small CCNN models ob- tain remarkable results.
Figure Performance of the CCNN6.380 on the LRA Image task, where samples are either input as sequence or with the original image structure.
nature of the data, showcasing CCNNs truly general- purposs architecture even able generalize across different data representations - feat hardly obtainable by existing architectures-.
Large CCNNS overfit on point-clouds, but tiny CCNNs perform surprisingly well. In contrast to 1D and 2D, we observe that larger CCNNs perform worse than smaller ones on 3D point-clouds (Fig. 8). We hypothesize that this is result of the sparsity of the task. Continuous convolutional kernels define kernel function over the entire domain even if the number of kernel values sampled is sparse. As result, sparse supervision makes it hard for the network to learn kernel function that generalizes to unseen data, as sparsity exposes the kernel function to aliasing -high frequencies between sampled points-. To corroborate this hypothesis, we train extremely small CCNNs wwith 6K, 14K and 48k parameters- on ModeINet4O. We observe that these models achieve surprising results and even outper- form the 200K and 2M CCNNs (Fig. 8). These results supports our previous hypothesis, but also show that the continuous kernel paradigm can be used to construct extremely small, performant CNNs.
6 LIMITATIONS AND FUTURE WORK
Computational efficiency. Although convolutions with large convolutional kernels scale better than self-attention, e.g., Perceiver (Jaegle et al., 2021), they can still be expensive. Luckily, the Fourier convolution be used strongly reduce their cost, e.g., Romero et al. (2022b;a); Gu et al. (2022). Nevertheless, the Fourier convolutions by themselves are not sufficient to scale CCNNs to very large inputs. This problem is exacerbated if irregular data is considered, as it (i) prevents the usage Fourier convolutions, and (ii) requires rendering a different convolutional kernel for each spatial position of each sample in the batch. An important avenue for future research may look into reducing computational requirements, either via separability or self-adjusting architectures, e.g., downsampling (Riad al., 2022), for regular data, and "gridifying" techniques for irregular data.
Larger models on point-cloud data. Although CCNNs performs remarkably well on point-clouds with extremely small models, the current CCNN formulation unable to achieve better performance with larger models. We hypothesize that this is due to increasing aliasing in the learned kernel generator networks. Although anti-aliasing techniques for kernel generator networks exist (Romero al., 2022a), this assume an underlying grid and a corresponding Nyquist frequency. An additional avenue for future research may look into the aliasing issue on irregular data either by generalizations of Romero et al. (2022a), or properly-defince smoothing techniques.
Cross-modal training and data fusion. CCNNs present a potential solution for cross modal training -currently challenging due to dissimilar per-modality architectures-. Additionally, the continuous properties of CCNNS allow them to be trained on a wide range of data sources, e.g., multiple resolutions, regular, irregular data, etc, which is interesting for several machine learning application.

---

Published as a conference paper at ICLR 2023
REFERENCES
Ossama Abdel-Hamid, Abdel-rahman Mohamed, Hui Jiang, Li Deng, Gerald Penn, and Dong Yu. Convolutional neural networks for speech recognition. IEEE/ACM Transactions on andio, speech, and language processing, 22(10):1533-144 2014.
Shaojie Bai, J Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.0127 2018.
Erik Bekkers. B-spline cnns on lie groups. arXiv preprint arXiv:1909.12033 2019.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL com/. Software available from wandb.com.
Oscar Chang. Lampros Flokas, and Hod Lipson. Principled weight initialization for hypernetworks. In International Conference on Learning Representations, 2020. URL https::/ net/ / Eorum? id=#1 1ma24 P....
Adam Coates, Andrew Ng. and Honglak Lee. An analysis single- networks in unsupervised feature learning. In Proceedings of the fourteenth international conference artificial intelligence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 764-773, 2017.
William Falcon et al. Pytorch lightning. GitHub, Note: http:///hhha...mmmm//////iiiaagn lightning, 3, 2019.
Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and J Zico Kolter. Multiplicative filter networks. In International Conference Learning Representations, 2021. URL httpps://ppezeeiieeet net formm? 1ee00mmmcPkkhhh
Marc Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. International Conference on Machine Learning, pp. 3165-3176 PMLR, 2020.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272. PMLR, 2017.
Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249-256. JMLR Workshop and Conference Proceedings, 2010.
Albert Gu, Isys Johnson, Karan Goel, Khaled Saab, Tri Dao, Atri Rudra, and Christopher Ré. Combining recurrent, convolutional, and continuous-time models with linear state space layers. Advances in Neural Information Processing Systems, 34, 2021.
Albert Gu, Karan Goel, and Christopher Re. Efficiently modeling long sequences with structured state spaces. In International Conference on Learning Representations, 2022. URL https: net. f (ww.earrerieeeeeee.foooooodooade
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pp. 1026-1034, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Binh-Son Son Hua, Minh-Khop Tran, and Sai-Ke Kit Yeung. Pointwise convolutional neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 984-993, 2018.
10

---

Published as a conference paper at ICLR 2023
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. International conference on machine learning, pp. 4651-4664. PMLR, 2021.
Yunho Jeon and Junmo Kim. Active convolution: Learning the shape of convolution for image classification.. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.
pp.
Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. Dynamic filter networks, Advances in neural information processing systems, 29, 2016.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. Advances in Neural Information Processing Systems, 33:6696-6707, 2020.
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv: (601::669..2200 (097002077 2016.
David M Knigge, David W Romero, and Erik Bekkers. Exploiting redundancy: Separable group convolutional networks on lie groups arXiv preprint vrXiv:2110. 13059, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. Advances in neural information processing systems, 25, 2012.
Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in neural information processing systems, 4, 1991.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-2324, 1998.
Jongpil Lee, Tacjun Kim, Jiyoung Park, and Juhan Nam. Raw waveform-based audio classification using sample-level enn architectures. arXiv preprint 17 2017.
Zhuang Liu, Hanzi Mao, Chao Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. arXiv preprint arXiv:2201.0354 2022.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.0399 2016.
lya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017.
Eric Nguyen, Karan Goel, Albert Gu, Gordon W Downs, Preey Shah, Tri Dao, Stephen Baccus, and Christopher Ré. S4nd: Modeling images and videos as multidimensional signals using state spaces. arXiv preprint arXiv:2210.0653 2022.
Aaron van den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. arXiv preprint arXiv: 1609: 03499, 2016.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 652-660, 2017.
Rachid Riad, Olivier Teboul, David Grangier, and Neil Zeghidour. Learning strides in convolutional neural networks. arXiv preprint arXiv:2202.01653. 2022.
Roberto Rigamonti, Amos Sironi, Vincent Lepetit, and Pascal Fua. Learning separable filters. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2754-2761, 2013.
David W Romero and Jean-Baptiste Cordonnier. Group equivariant stand-alone self-attention for vision. arXiv preprint arXiv:2010.00977 2020.

---

Published as a conference paper at ICLR 2023
David W. Romero, Robert-Jan Bruintjes, Jakub Mikolaj Tomczak, Erik Bekkers, Mark Hoogendoom, and Jan van Gemert. Flexconv: Continuous kernel convolutions with differentiable kernel sizes. In International Conference Learning Representations, 2022a. URL MRRLtttps///opeeeeiiee net/ fert/formm i d= jooF --
David W. Romero, Anna Kuzina, Erik J Bekkers, Jakub Mikolaj Tomezak, and Mark Hoogendoorn. CKConv: Continuous kernel convolution for sequential data. In International Conference on Learn- ing Representations, 2022b. URL http:::pp/. http:///wpeeeeeeeeeeeeee. wwt///ooounaneee forum? dFF O.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In International Conference on Medical image computing and computer- assisted intervention, pp. 234-241. Springer. 2015.
Lars Ruthotto and Eldad Haber. Deep neural networks motivated by partial differential equations. Journal of Mathematical Imaging and Vision, 62(3):352-366 364, 2020.
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre
Tkatchenko, and Klaus- Roburt Müller. Schnet: A continuous- filter convolutional neural network for modeling quantum interactions. Advances in neural information processing systems, 30, 2017.
Laurent Sifre and Stéphane Mallat. Rigid-motion scattering for texture classification. arXiv preprint arXiv: 1687, 2014.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 140:.14099.56 2014.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33:7462-7473, 2020.
Ivan Sosnovik, Michal Szmaja, and Arnold Smeulders. Scale-equivariant stcerable networks. arXiv preprint arXi: 1900 2019.
Ivan Sosnovik, Artem Moskalev, and Arnold Smeulders. Disco: accurate discrete scale convolutions. arXiv preprint 10Xiv:2//660223333 2021.
Bing Su and Ji-Rong Wen. Log-polar space convolution for convolutional neural networks. arXiv preprint arXiv:2 107. 1943, 2021.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1-9, 2015.
Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil. Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng. Fourier features let networks learn high frequency functions in low dimensional domains. Advances in Neural Information Processing Systems, 33:7537-7547, 2020.
Yi Tay, Mostafa Dehghani. Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler. Long range arena : A benchmark for efficient transformers. In International Conference Learning Representations, 2021. URL https: //openreview..o/ net /tor/ooomm///// wobormm?idgoee wetmmiiddiVeWggeeeeeec grC2k.
Hugues Thomas, Charles R Qi, Famanell Deschaud, Beatriz Marcotegui, François Goulette, and Leonidas Guibas. Kpconv: Flexible and deformable convolution for point clouds. In Proceedings of the IEEE/CVF international conference computer vision, pp. 6411-6420, 2019.
Nergis Tomen, Silvia-Laura Pintea, and Jan Van Gemert. Deep continuous networks. In International Conference on Machine Learning, pp. 10324-10335. PMLR, 2021.
Aäron Van Den Oord, Sander Dieleman, Heiga Zen, Karen Simonyan, Oriol Vinyals, Alex Graves, Nal Kalchbrenner, Andrew Senior, and Koray Kavukcuoglu. Wavenet: A generative model for raw audio. SSW, 125:2, 2016.
12

---

Published as a conference paper at ICLR 2023
Shenlong Wang. Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky, and Raquel Urtasun. Deep parametric continuous convolutional neural networks. In Proceedings of the IEEE conference on computer viston and pattern recognition, pp. 2589- 2589-2597, , 2018.
Pete Warden. Speech commands: dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1804.03229, 2018.
Maurice Weiler and Gabriele Cesa. General e (2)-equivariant steerable cnns. Advances in Neural Information Processing Systems, 32, 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. Advances in Neural Information Processing Systems, 31, 2018.
Daniel Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. Advances in Neural Information Processing Systems, 32, 2019.
Daniel E Worrall, Stephan Garbin, Daniyar Turmukhambetov and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 5028-5037, 2017.
Wenxuan Wu, Zhongang Qi, and Li Fuxin. Pointconv: Deep convolutional networks on 3d point clouds. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 9621-9630, 2019.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: deep representation for volumetric shapes. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Ruibin Xiong, Yunchang Yang, Di He, Kai Zheng, Shuxin Zheng, Chen Xing, Huishuai Zhang, Yanyan Lan, Liwei Wang, and Tieyan Liu. On layer normalization the transformer architecture. In International Conference on Machine Learning, pp. 10524-10533, PMLR, 2020.
Yifan Xu, Tianqi Fan, Mingye Xu, Long Zeng, and Yu Qiao. Spiderenn: Deep learning point sets with parameterized convolutional filters. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 87-102, 2018.
Onry Yadan. Hydra a framework for elegantly configuring complex applications. Github, 2019. URL httphttpp///// //gt//tthhbbb wwt.c//..aaaa..ccccca http:////cccccoooeeeeeea////////
13

---

Published as a conference paper at ICLR 2023
APPENDIX
MODELLING LONG RANGE DEPENDENCIES IN ND: FROM TASK-SPECIFIC To A GENERAL PURPOSE CNN
A EXTENDED RELATED WORK
In this section, we provide a more extensive treatment of similar methods related continuous repa- rameterizations of convolutional kernels and their limitations in use for general purpose architectures.
Rethinking convolutional kernels. Several previous works investigate a reformulation of CNNs based on modifications to the classical convolutional layer. These works provide formulations of continuous convolutional kernels pursuing several different motivations. Several works apply specifically to point-cloud data -for which discrete kernels are not appropriate- by interpolating a set of weights to obtain a definition over the continuous input space (Hua et al., 2018; Thomas et al., 2019) or expressing the kernels in polynomial (Xu et al., 2018) neural network basis (Jia et al., 2016; Schütt et al., 2017; Wang et al., 2018; Wu et al., 2019). These architectures often rely on sophisticated data augmentation and feature engineering schemes, perform pooling or downsampling operations, and have not been shown to perform well on regular data, e.g. images (Wu et al., 2019).
Other work focuses continuous kernel formulations to address specific architectural considerations, for example to increase receptive fields efficiently (Su & Wen, 2021), or to learn more appropriate kernel geometries (Dai et al., 2017; Jeon & Kim, 2017). In these examples, fixed set of weights is interpolated over the kernel domain, which implicitly defines low-resolution kernel that impedes modeling long term dependencies in a dense manner (Romero al., 2022a).
A different line of work incorporating continuous kernel definitions focuses exploiting desir- able properties of spatially structured kernels for implementing convolutions equivariant to roto- translations (Sifre& & Mallat, 2014; Worrall et al., 2017; Weiler et al., 2018; Weiler & Cesa, 2019; Bekkers, 2019; Finzi et al., 2020), or tilasiiontrannslation transformations (Sosnovik et al., 2019; Worrall & Welling, 2019; Sosnovik et al., 2021). Although these formulations also in principle give us handles for a continuous definition of convolutional kernels over the spatial domain, we choose not restrict the kernel functions learned in our models to be equivariant, as this limits applicability to settings in which such priors are not warranted.
Note that graph convolutions (Kipf & Welling, 2016) may similarly be viewed providing convolution operation invariant to arbitrary permutation symmetries on the input. Since this operation disregards positional information, it essentially shares the kernel function over the domain. In practice, the bias to permutation invariance needlessly impedes network expressivity for data where such assumptions do not hold, such sequences, images point-clouds.
Continuous Kernel Convolutions (Romero et al., 2022b) may be seen as a special case of neural message passing (Gilmer et al., 2017) a fully connected graph over the input, with messages conditioned on relative positions of nodes. In the case of FlexConv (Romero et al., 2022a), graph connectivity is dynamic, conditioned on distance.
B DATASET DESCRIPTION
Sequential and Permuted MNIST. The MNIST dataset LeCun et al. (1998) consists of 70K gray- scale 28x28 handwritten digits divided into training validation and test sets of 60K and 10K samples, respectively. For validation purposes, the training dataset further divided into training and validation sets of 55K and 5K samples, respectively.
The sequential MNIST dataset (sMNIST) presents MNIST images as a sequence of 784 pixels for digit classification. Consequently, good predictions require the model to preserve long-term dependencies up to 784 steps in the past. The permuted MNIST dataset (pMNIST) incorporates an additional level of difficulty by permuting the order of all sMNIST sequences with random permutation. Resultantly, models can no longer rely on local information for the construction of their features and the importance of modelling long-term dependencies becomes more pronounced.
14

---

Published as a conference paper at ICLR 2023
CIFARIO, CIFARI00 and Sequential CIFAR1O. The CIFARIO dataset Krizhevsky et al. (2009) consists of 60k real-world 32x32 RGB images uniformly drawn from 10 classes divided into training and test sets of 50K and 10K samples, respectively. The CIFARIO0 100 dataset Krizhevsky et al. (2009)
is similar to the CIFAR10 10 dataset, with the difference that the images are now uniformly drawn from 100 different classes. For validation purposes, the training dataset of both CIFAR10 10 and CIFAR100 are further divided into training and validation sets of 45K and 5K samples, respectively.
Analogously the sMNIST dataset, the sequential CIFARIO (SCIFARIO) 10) dataset presents CIFARIO images as a sequence of 1024 pixels for image classification. This dataset is more difficult than sMNIST, as (i) larger memory horizons are required to successfully solve the task, and (ii) more complex structures and intra-class variations are present in the images.
Speech Commands. The Speech Commands dataset Warden (2018) consists of 105809 one-second audio recordings of 35 spoken words sampled at 16kHz. Following Kidger et al. (2020), we extract 34975 recordings from ten spoken words to construct balanced classification problem. We refer to this dataset as Raw Speech Commands. In addition, we use the preprocessing steps of Kidger et al. (2020) and extract mel-frequency cepstrum coefficients from the raw data. The resulting dataset, referred to as MFCC Speech Commands, consists of time series of length 161 and 20 channels.
Long Range Arena. The Long Range Arena benchmark (Tay et al., 2021) consists of 6 tasks with lengths 1K-16K steps encompassing modalities and objectives that require similarity, structural, and visuospatial reasoning. The P aathfinder, Path-X X and Image tasks are similar in nature to the sMNIST and sCIFARIO tasks. These tasks consists of classification tasks performed on images that are treated as sequences.
The Image task corresponds to the sequential CIFARIO 10 dataset with the only difference that the CIFAR10 images treated gray-scale images. The Pathfinder and Path-X tasks are binary tasks in which binary images are pro- vided and the model must predict whether the two points in the images are connected with a line or not see Fig. 10 for an example-. The dif- ference between both datasets their resolution. Whereas Pathf f i nnde has images of size 32x32, Path-X has images of size 128x 128. It is impor- tant to mention that these tasks are so difficult that even if treated as 2D signals, CNNs without global receptive fields cant solve them (Gu et al., 2022).
(b) Figure 10: Positive and negative samples from
(a)
the Path-X X dataset
10L-10. The STL-10 dataset (Coates al., 20 is subset of the ImageNet dataset (Krizhevsky et al., 2012) consisting of 13,000 96x96 real-world RGB images uniformly drawn from 10 classes divided into training and test sets of 5K and 8K images, respectively. For validation purposes, the training dataset is further divided into training and validation sets of 4,500 and 500 samples, respectively.
ModelNet40. The ModelNet40 dataset (Wu et al., 2015) contains 12,311 3D meshes of objects belonging 40 classes. We use the official split with 9,843 training samples and 2,468 validation samples. In contrast with previous works which sample 1,024 points from these meshes, (e.g. Wu al. (2019)), we sample 512 points uniformly from the faces of each mesh, along with the normal vectors at each of these positions. These position and normal vectors serve as input to the model.
ModelNet10 Voxelized. ModelNet10 is a sub- set of ModelNet40 containing orientation-aligned samples from 10 classes of objects. We vox- clize ModelNet10 based on subsampling of 4096 points from the meshes. First, we normalize these points to fall within the interval [-1, 1J. Next, we bin these points into a grid of 40 x 40 x 40 vox- els. All nonzero voxels get assigned their location as feature value. Afterwards, to align with our point-cloud configuration, we mask out the all but 512 nonzero voxels. Throughout the network, we
(a)
(b)
Figure 11: An example of a point-cloud sampled from ModelNet10 (a), and corresponding voxel 15 representation of the same sample (b).

---

Published as a conference paper at ICLR 2023
only apply convolutions at the locations of each nonzero voxel (effectively masking out activations at all other voxel locations). Like with the point- cloud configuration, to limit computational complexity, we limit the convolutions to integrate over the closest 256 points.
C EXPERIMENTAL DETAILS
C.I GENERAL REMARKS
Code repository and logging. Our code is written in PyTorch. We utilize wandb (Biewald, 2020) hydra (Yadan, 2019) and wyterch-ligghniin
(Falcon et al., 2019) for logging and code structuring. Our experiments are performed on NVIDIA TITAN RTX, A6000 and A A100 GPUs, depending on the size the datasets and inputs considered. Our code is publicly available at ur! hidden for the sake of the double blind review process.
Normalized relative positions. The kernel network (Keeell can, in principle, receive arbitrary coordinates input. However, considering unitary step-wise relative positions, i.e., 0, 1, 2, ... .N can be problematic from numerical stability perspective as N may grow very large, e.g., N- 1600 for the Speech Commands dataset. Consequently, based insights from the Implicit Neural Representations, e.g., Sitzmann et al. (2020); Fathony et al. (2021), we normalize the coordinates such that they lie in the space [-1, 1]D for D-dimensional kernels. To this end, we map largest unitary positions seen during training [o, _V] to a uniform linear space in [-1, 1]. Note that any possible relative kernel positions outside of the trained kernel domain which may be encountered during inference are masked out.
C.2 HYPERPARAMETERS AND TRAINING DETAILS
Optimizer and learning rate scheduler. All our models are optimized with Adam (Loshchilov Hutter, 2017) in combination with cosine annealing learning rate scheduler Loshchilov & Hutter (2016) and a linear learning rate warm-up stage of 10 epochs.
Best hyperparameters found. We perform hyperparameter search on the learning rate, dropout rate, weight decay, and w/ of our CCNNs for each task considered." The best hyperparameters found are reported in Tables 3 and 4.
efficiency experiments on ModelNet40. To further assess parameter efficiency of the CCNN on ModelNet40, run number of experiments with smaller model sizes. Results for the following models are shown in Fig. 8: CCNN4,16 with residual blocks, a channel size of 16 and a kernel network hidden size of 8. with 6,545 total parameters. CCNN with 4 residual blocks, channel size of 32 and a kernel network hidden size of 8, with 14,401 total parameters. CCNN with residual blocks, a channel size of 48 and kernel network hidden size of with 26,353 total parameters. We use a learning rate of 2e 2 no weight decay, and an wp of 50.
Parameter
D ADDITIONAL EXPERIMENTS AND DETAILS
D.1 COMPUTATIONAL EFFICIENCY
D.1.1 EFFICIENCY OF CONTINUOUS CONVOLUTIONAL KERNELS
Experimental setup. We experimentally asses the computational complexity of our model. We measure the time of a single forward/backward pass for the two architectures used in our experiments: CCNN4,140 (4 blocks, 140 channels) and CCNNG (6 blocks, 380 channels). It is important to note that we assume the "worst-case" scenario for CCNNs in which the learned kernel size equals the input length, i.e., the model performs convolutions with global kernels at each layer. FlexConvs (Romero et al., 2022a) learn the kernel size during training and -as shown exemplarily Fig 12- having global kernels at each layer is not something that we observe in practice.
wg serves prior the variance of the data that is fitted with several types of implicit neural representa- tions, e.g., SIRENs Sitzmann al. (2020), MFNs Fathony et al. (2021),
16

---

Published as a conference paper at ICLR 2023
Table 3: Best hyperparameters found for CCNN 4,140 on all tasks considered.
Dropout
Learning
Weight Decay
Butch
Epochs
SMNIST
2976.49
0.1
0.01
le-6
100
210
2985.63
SCIFAR 1D
2386.49
0.0
0.02
0
50
210
SPEECH CUMMANDS (RAW)
1295.61
0.02
Ic-6
20
SPEECH COMMANDS (MFCC)
750.18
0.2
0.02
le-6
100
110
LiSTOPS
784.66
0.001
Ic-6
50
TEXT
2966 60
0.2
0.001
le-5
50
60
PATHEINDER
2272.56
0.2
0.01
0
100
210
CIFARIO
1435.77
0.0001
CIFARI00
3521.55
0.1
0.02
0.0001
50
210
STL10
954.28
0.02
0
64
2DIMAGE
2085.43
0.02
le-6
50
210
2DPATHEENDER
1239.14
0.01
0
100
210
Nor00
50
0.0
0.02
lc-8
16
200
Table 4: Best hyperparameters found for CCNN on all tasks considered.
100
Drupout
Learning Rate
Weight Decay
Batch Size
Epochs
SMNIST
2976.49
0.1
0.01
100
210
PMNIST
2985.63
0.2
0.02
100
210
SCIFARIO
4005.15
0.25
0.01
210
SPERCH COMMANDS (RAW)
1295.61
0.2
0.02
le-6
160
SPEECH COMMANDS (MFCC)
750.18
0.2
0.02
Ic-6
100
110
LISTOPS
784.66
0.25
0.001
50
60
TEXT
2966.60
0.3
IMAGE
4005.11
0.1
0.01
210
PATHFINDER
2272.56
0.1
0.01
1o-6
100
210
CIFARIO
1435.77
0.15
0.02
210
CI
679.14
0.2
0.02
50
210
STL.10
2DIMAGE
2306.08
0.2
0.02
50
210
2PPATRRNNDER
3908.32
0.2
0.01
100
210
MODEL NET40
50
0.0
0.02
1e-7
32
200
17

---

Published as a conference paper at ICLR 2023
(a) First block.
(b) Second block.
(c) Third block.
(d) Fourth block.
Figure 12: Learned (causal) convolutional kernels in CCNN4, 140 trained on sequential CIFAR1O. The vertical axis indexes the channels, and the horizontal axis the length of the kernels in a single layer. Interstingly, all layers -a period that corresponds to the width of the underlying 32x32 CIFAR 10 images-. This illustrates that CCNNs in fact learn represent 2D structures the flattened 1D space which sequential CIFAR10 defined. Despite these important capabilities, is important to note that modelling ND signals flattened ID signals poses unnecessary burden the model, and modelling the signal in the original 2D space leads to faster convergence and better accuracy (Fig. 9).
observe clear periodic pattern of 32 steps along the spatial dimension across
18

---

Published as a conference paper at ICLR 2023
Table 5: Training time (ms). unavailable due to memory issues. n number of blocks tuned per input length.
SIZE
1024
4096
8192
16000
Local CNN,,.100
24
32
43
65
Global 4N
13
25
53
122
CCNN 4,140
52
56
77
145
Local CNN
24
38
93
178
Global CNN
31
102
220
x
CCNN6,330
71
136
255
537
We compare (i) discrete convolutional networks which match in architecture ours, with global depthwise separable convolutional kernels; Global CNN,,140- and Global CNNo,380 (ii) more traditional CNN architectures with downsampling after each block (pooling of size 2), local depthwise separable kernels (kernels of size 5), and depth tuned to the input size such that the receptive field of the final residual block covers the entire input. We take depth [d] with given by: input length
2 kernel size.
pooling size"
These architectures are denoted Local CNN4,140 and Local CNN6,3so For input lengths of 1024, 4096, 8192 and 16000 this results in depths of 8, 10, 11 and 12 blocks respectively.
We measure training time (forward and backward pass) for sequence data of different lengths between 1024 -the length of SCIFARIO-, and 16000 -the length of Speech-Raw-, the largest sequence length in our experiments. We measure the time of training over 1000 batches of size 8 on a RTX 3090.
Results. Results are summarized in Tab 5. Note that for the Global CNNo,380. we were unable to complete the experiment on data of lengths exceeding 8192 because of insufficient memory during the backward pass. This highlights another benefit of the continuous convolutional layer: its relative memory efficiency in computing weight updates, which results from the fact that gradients for individual kernel values can be discarded. As only the kernel network weights need be updated, gradient values for the kernel weights do not need to be stored during the backward pass.
Our results show that our approach requires additional computation compared to more traditional discrete CNNs. We note that although the use of a kernel network to infer kernel values brings computational overhead, this overhead is not a limiting factor in practice. The global convolutional kernels used in network architectures have impact computational efficiency, but the use of convolutions in the Fourier domain help address this issue.
1.1.2 EFFICIENCY OF CCNNS WITH REGARD TO OTHER GLOBAL METHODS
Experimental Setup. Next, we benchmark the computational efficiency of CCNNs compared to other input-global methods: $4 (Gu al., 2022) and LSSL (Gu et al., 2021).
We reproduce the efficiency benchmark in Gu et al. (2022). That is, we provide the runtime for single layer of forward and backward pass for different hidden dimensionalities, measured for different input lengths. The kernel network used in these experiments is MAGNet with 3 layers and hidden dimensionality
32. Again, we assume "worst-casse setting in which the learned kernel size equals the input length, i.e. the layer performs convolutions with global kernels. Note that in this setting, we perform convolutions in the spatial domain. We average runtime over 10000 steps. Experiments are performed on RTX 3090.
Results. Results are summarized in Tab. We show that, up to a sequence length of 16000, runtimes in comparison with S4 are marginally faster, but remain in the same order of magnitude. Differences in memory utilization become more pronounced for increases in hidden dimensionality and sequence length, but remain constant factor. Next, note that in both memory and speed, we show that CCNN layer is asymptotically more efficient than LSSL. For LSSL, only results present in Gu et al. (2022) unable to successfully reproduce the LSSL. layer performance.
shown,
19

---

Published as a conference paper at ICLR 2023
Table 6: Forward and backward for single layer of CCNN in comparison with S4 (Gu al., 2022)
and LSSL (Gu et al., 2021) in speed (ms). result not available.
INPUT LENGTH
128
1024
4096
16000
HIDDEN SIZE
256
512
1024
256
512
1024
256
512
1024
256
512
1024
LSSL
9.32
20.60
140.70
x
8
X
S4
4.16
3.41
4.16
4.11
6.39
9.06
6.86
10.57
25.20
18.10
35.72
96.64
CONN
3,99
3.71
4.12
3.90
5.07
7.25
5.91
9.67
22.39
20.64
42.49
102.57
Table 7: Forward and backward for a single layer of CCNN in comparison with S4 (Gu et al., 2022)
and LSSL (Gu et al., 2021) memory usage (Mb). x result not available.
INPUT LENGTH
128
16000
HIDDEN SIZE
256
512
1024
256
512
1024
2.56
512
1024
256
512
1024
LSSL
222.1
1685
13140
S4
5.6
14,46
42.64
33,65
70.47
153.69
129.69
262.51
537.72
508.93
1014,08
2033.82
CONN
14:46
28.94
60.38
113.13
222.97
444.17
451.39
888.24
1763.43
1764.46
3467.34
6878.26
D.2 ABLATION: PERFORMANCE FLEXCNN, S4 AND CCNN RESIDUAL BLOCKS
Experimental setup. We provide an ablation over the impact on performance of the proposed residual CCNNBlock compared to the FlexCNNBlock used in the architectures Romero al. (2022a) and the $4Block used in Gu et al. (2022) (Fig. 7).
To this end, we run experiments on a select number of datasets covering sequence and image data with version of our architecture that includes the FlexCNNBlock (F-CCNN4,140 with 4 blocks, 140 channels, 233K params and F-CCNN,,350 with blocks, 380 channels, 2.24M params), and a version of our architecture that includes the S4Block (S4-CCNN,,14 4,400 with 4 blocks, 140 channels, 200K params and S4-CCNN6, with 6 blocks, 380 channels, 2M params). We compare performance against the two architectures used throughout the experiments in Sec. 5, which uses the CCNNBlock (CCNN4,140 with 4 blocks, 140 chamnels, 200K params and CCNN with 6 blocks, 380 channels, 2M params). To isolate the impact of the residual block architecture, we replace the FlexConv layers the original formulation of the FlexCNNBlock with our proposed depthwise separable implementation SepFlexConv. Note that do not parameter-match these models, which results in the architectures with Flex having more parameters compared to ones with the CCNNBlock, due the FlexCNNBlock having two convolutional layers instead of the one convolutional layer and one pointwise linear layer of the CCNNBlock. Training regimes and hyperparameters for all architectures are as per Sec. C.
Results. Results are summarized in table 8. First, note that without the nonlinearity added add the end of the block, the S4Block performs poorly over tasks. Next, as shown, the CCNNBlock formulation improves performance of the architecture compared to FlexCNNBlock, in some cases by a significant margin. Note that architectures with the S4Block and CCNNBlock contain fewer convolutional layers and parameters compared to architecturally matched ones with FlexCNNBlocks, indicating higher parameter-efficiency as well as computational efficiency of CCNNBlocks (e.g. 8h 55m runtime for F-CCNN 6,30 8h 12 12m runtime for CCNN6, on SCIFAR10). Because of the shown performance benefits, we use the CCNNBlock throughout our experiments.
Table 8: Test accuracy of models with FlexCNNBlock (F-CCNN4, 140. F-CCNN, 6.380.. models with S4Block (S4-CCNN,,140. S4-CCNNN,380) and models with CCNNBlock (CCNN CCNN6,350).
SPRECHCCOMANNS--MFCC
SCIFAR10
CIFAR10
F-CCNN 1.140
94.38
85.54
86.34
S4-CCNN 4,140
73,60
53.25
60.06
CCNN4,140
95.01
90.30
92.78
F-CCNN
94.92
84,68
92.24
S4-CCNN6.380 6,380
60.82
51.50
67.01
CCNN6.381
97.98
93.08
95.20
20


Reviews:
[{"review_id": "vxSFzWieUj", "reviewer": "Reviewer_RGNG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors present a Convolutional Neural Network (CNN) architecture based on pointwise operations, global operations, and local operations, i.e., continuous convolutional kernels to handle the input with different lengths and resolutions. They show that this setting is data independent, which models long-range dependencies at every layer, with a fixed number of parameters. The experimental results show that the proposed Continuous Convolutional Neural Network (CCNN) performs well on the Long Range Arena dataset given sequence data. ", "summary_of_the_review": "Overall, my major concern is the real novelty of this paper. Besides, considering that the empirical results on several datasets are relatively lower than the baseline results, the effectiveness of the proposed method remains unclear. The authors are encouraged to solve the weaknesses above.", "strength_and_weaknesses": "# Strength:\n- The idea of \"a CNN architecture that can be used across input resolutions, lengths and dimensionalities (1D, 2D, 3D) showing its viability across several 1D, 2D and 3D tasks\" seems interesting, which may facilitate the cross-modal pre-training to further enhance the performance of large neural networks. \n- The experimental results seem encouraging that CCNN tends to outperform strong baselines on sequence data, e.g., the competitive results on the Long Range Arena dataset. \n\n# Weaknesses:\n- ### Major concerns:\n  - I politely disagree that \"Convolutional Neural Network (CNN) architectures must be tailored to specific tasks in order to consider the length, resolution, and dimensionality of the input data\" since the input data size can be inconsistent with the training dataset. Though CNN can not be directly applied to the 1D/3D input when pre-trained on 2D images, the descriptions of CNN as \"current CNN architectures are resolution-bounded and thus different resolutions require different CNNs\" and \"there is no trivial way to obtain equivalent kernel values on upsampled data, and hence, no way to apply a trained model directly\" seem inappropriate.\n  - The authors claim that \"our Continuous Convolutional Neural Network parameterizes kernel values as a continuous function $\\varphi$ over the input domain $R^d$, which decouples it from data characteristics\". It would be better to further discuss the \"data characteristics\" decoupled by the continuous function. Besides, it is true that a continuous kernel is compatible with arbitrary size/resolution/dimensionality, however, given the limited computing resources, a sampling operation is required during pre-processing (also pointed out by the authors in **Computational efficiency**). Is \"informed sampling\" against the motivation of a data-independent network?\n  - Though the authors elaborate on the \"data-independent\" architecture, I am not fully convinced that this setting should contribute to higher performance. Notice that the popular Transformer architecture still requires conventional convolution operations to extract features and then conducts multi-head self-attention. From my point of view, convolution operations that introduce inductive bias could be more suitable for images. Then, the authors may further discuss the motivation of this paper.  \n  - It would be better to report the real runtime speed instead of the model parameters to make a fair comparison with S4 [1].\n\n- ### Minor Comments:\n  - **Parameter scaling**: the description of \"CCNN formulation lacks in parameter scaling capabilities; increased parameter counts yield strongly\ndiminishing returns\" seems unclear.\n  - **Separable Continuous Convolutions**: How to determine the input channel number and the feature map size of the continuous \"spatial\" separable convolutional layer? It would be better to explain the meaning of $D$, $N_{in}$, and $N_{out}$.\n  - **Table 1**: As claimed in *Empirical results*, \"the CCNN is not restricted to grid data, e.g., 3D voxels, and can be used on point-clouds directly\", I expect promising results on ModelNet40. However, the $CCNN_{4,110}$ seems far from satisfactory.\n  - I am curious why the authors did not increase the weight decay $\\lambda$ given that \"we observed strong overfitting to the training set, which\nmay indicate the need for stronger regularization\"?\n  - **Normalized relative positions**: What if the largest unitary position is larger than $N$ at inference time?\n  - **Code repository and logging**: It seems that the code link is missing.\n  - **Section 3.1**: \"Pointwise operations are applied independently to each spatial element of the input signal\". However, each spatial element shares the same parameter for 1x1 convolutions.\n  - Please fix the \"Sec.??\" on page 4.\n  - I expect an ablation study on the effect of the S4 Block [1].\n \n# Reference:\n[1] Efficiently modeling long sequences with structured state spaces. ICLR2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "iVL9ewWCyIS", "reviewer": "Reviewer_jCEb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents Continuous Convolutional Neural Networks, which can process the input of arbitrary resolution, length, and dimensionality. Compared to conventional convolutional neural networks which adopts fixed-size kernels, the proposed continuous convolutional networks adopt dynamic and continuous kernels, which removes the dependencies of the input data. Based on the presented continuous convolution operation, this paper designs a deep continuous convolutional network and apply it to several tasks with different input shapes and resolutions. ", "summary_of_the_review": "Considering the limited technical novelty and contribution of this paper, I think this paper is not qualified as a conference paper. This paper is highly coincident with previous works.", "strength_and_weaknesses": "### Strength\n1. This paper presents a data-independent operation, i.e., continuous convolution operator, which is adaptive to the shape, resolution, and dimension. The continuous convolution based on Continuous Kernels maps the spatial (relative) locations to the parameters.\n2. This paper builds a continuous convolutional networks based on the continuous kernel and depth-wise continuous convolutions, namely CCNN, and provides a series of optimization strategies, including the initialization, regularization, and etc.\n3. This paper adopts the same architecture on several tasks with different input resolutions or shapes, e.g., 2D images and 3D point clouds.\n4. Experimental results of this paper on varieties of benchmarks of various input shapes are good.\n\n### Weakness\n1. The novelty of this paper is limited. The proposed continuous convolution has been explored in the most relevant papers [1,2], which share the same motivation and similar approach with this submission. The authors SHOULD clearly state the contribution of this paper and the differences compared to others’ works. The technical novelty of the proposed CCNN based on Continuous Kernels and FlexConv is limited for me.\n2. This paper lacks detailed illustration and description about the proposed architecture, which is the main contribution in my opinion since the continuous convolution can not be regarded as a novel contribution of this paper.\n3.  I don’t agree that data-dependent architectures are terrible or limited by shapes, resolutions, or dimensions. Firstly, data-dependent architectures bring more inducive bias for specific tasks, which has better characteristics compared to the so-called universal architectures. Secondly, data-dependent or hand-crafted architectures can perform better on specific data/tasks in terms of both speed and accuracy, and most practical applications are driven by data-dependent architectures. Moreover, we now have massive versatile transformers and neural architecture search mechanisms, and I’m concerned about the predominance of the proposed CCNN.\n4. This paper lacks the exact inference speed, training speed, or latency on the given devices, e.g., NVIDIA GPU. It matters for me.\n5. In Tab.1, the authors should provide more comparisons with the newer methods, e.g., vision transformers. I’m glad to see that the proposed CCNN can perform better with less or comparable computation budget when compared to recent vision transformers.\n6. I’m concerned about the transferring ability on 2D inputs, i.e., 2D images. For example, training the proposed CCNN and a normal CNN which have similar hierarchical architecture and parameters on ImageNet or other datasets and evaluate the performance on lower-resolution datasets. Providing fair comparisons and experimental evaluations will be more convincing in my opinion.\n\n[1] Remero et.al. CKConv: Continuous Kernel Convolution for Sequential Data. ICLR 2022.\n[2] Remero et.al. FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizes. ICLR 2022.\n\n### Typos\nMissing reference in Sec. 3.3: “with 9 parameters (Sec. ??)”.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "aCuoelPSOjK", "reviewer": "Reviewer_KStu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper aims to make CNN suitable for data of arbitrary resolution, dimensionality and length without any structural changes. A Continuous CNN is proposed by introducing continuous convolutional kernels which is a data independent parameterization for convolutional weight. The proposed CCNN can work on on sequential (1D), visual (2D) and point-cloud (3D) tasks with the same architecture.", "summary_of_the_review": "I like the idea to unify CNN architectures on various tasks.", "strength_and_weaknesses": "Strength:\n- The motivation to construct a unified convolution neural network is interesting.\n- The Continuous CNN is a reasonable solution for general architecture.\n- Experimental results on a range of sequence, image and point-cloud datasets show the effectiveness of the proposed method.\n\nWeaknesses:\n- What about the computational complexity? I'm afraid it will be large since there is no downsampling.\n- The number of parameters will be large since the method introduces a kernel neural network.\n- To make neural network independently from the input length, or resolution, there are several other alternatives like deformable convolution [1] and graph convolution [2]. Why not using these methods? Please include discussion.\n- Just a suggestion and not necessary: could the proposed CCNN trained once on all the datasets in Table 1 and work well on all of them? Large-scale ImageNet performance?\n- Typos: \"??\" in footnote in page 4.\n- The writting of the paper should be improved: what are $*$ and $\\tilde{x}$ in Eq.(1).\n\n[1] Dai J, Qi H, Xiong Y, et al. Deformable convolutional networks[C]//Proceedings of the IEEE international conference on computer vision. 2017: 764-773.\n\n[2] Han K, Wang Y, Guo J, et al. Vision GNN: An Image is Worth Graph of Nodes[J]. arXiv preprint arXiv:2206.00272, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Bk3Fiaxuw1b", "reviewer": "Reviewer_Av5z", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors proposed a unified CNN architecture that is tested on multiple datasets. The main idea is to learn a hyper-network to predict the discrete kernel given coordinates as input, which is instantiated based on an existing kernel-size differentiable convolution (FlexConv). Extensive experimental results are provided on multiple benchmark datastes covering inputs of different dimentionalities and modalities.", "summary_of_the_review": "Despite of limited novelty on the technical side and some missing exploration to further support the motivation of unifying the architecture among modalities, the reviewer still feels the current results and discussions could be possibly helpful for the community.", "strength_and_weaknesses": "Pros:\n1. The proposed idea is interesting.\n2. The empirical results are supporting.\n\nCons:\n1. The technical novelty is rather limited as the building blocks are mainly from exisiting work, e.g., FlexConv, although the reviewer does recognize the contribution of all the empirical results and discussions.\n2. One important aspect is missing: transformer is not only surprising as one architecture could be applied to different type of inputs. The more practical and important fact is that different inputs could indeed share most of the weights of the network [r1,r2]. \na. Therefore, it will be really more exciting if the authors also explores the setting where different types of inputs/tasks share the same set of network weights. \nb. Another interesting exploration is to interpret the network weights trained on different data: does model trained on different data in the same modality share certain weighs? do models trained on different types of data even also share certain patterns?\n\n[r1] All in one: Exploring unified video-language pre-training\n[r2] PolyViT: Co-training Vision Transformers on Images, Videos and Audio\n\n\n\nMinor:\n\"Sec. ??\" appears multiple times. For example, in Sec 3.3, in the second paragraph, \"Sec. ??\".", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster