FlexConv: Continuous Kernel Convolutions With Differentiable Kernel Sizes
Authors: David W. Romero
Robert-Jan Bruintjes
Jakub Mikolaj Tomczak
Erik J Bekkers
Mark Hoogendoorn
Jan van Gemert
Venue: ICLR 2022 Poster (2022)
Keywords: Convolutional neural networks
learnable kernel size
continuous convolutional kernels
alias-free convolutional networks
implicit neural representations
resolution-agnostic representations
time series
sequential data
computer vision

Abstract: When designing Convolutional Neural Networks (CNNs), one must select the size of the convolutional kernels before training. Recent works show CNNs benefit from different kernel sizes at different layers, but exploring all possible combinations is unfeasible in practice. A more efficient approach is to learn the kernel size during training. However, existing works that learn the kernel size have a limited bandwidth. These approaches scale kernels by dilation, and thus the detail they can describe is limited. In this work, we propose FlexConv, a novel convolutional operation with which high bandwidth convolutional kernels of learnable kernel size can be learned at a fixed parameter cost. FlexNets model long-term dependencies without the use of pooling, achieve state-of-the-art performance on several sequential datasets, outperform recent works with learned kernel sizes, and are competitive with much deeper ResNets on image benchmark datasets. Additionally, FlexNets can be deployed at higher resolutions than those seen during training. To avoid aliasing, we propose a novel kernel parameterization with which the frequency of the kernels can be analytically controlled. Our novel kernel parameterization shows higher descriptive power and faster convergence speed than existing parameterizations. This leads to important improvements in classification accuracy.

Published as a conference paper at ICLR 2022
FLEXCONV: CONTINUOUS KERNEL CONVOLUTIONS WITH DIFFERENTIABLE KERNEL SIZES
David W. Romero Robert-Jan Bruintjes"*
Erik J. Bekkers", Jakub M. Tomczak', Mark Hoogendoorn' Jan C. van Gemert
Vrije Universiteit Amsterdam Delft University of Technology University of Amsterdam The Netherlands
d. w. rome www.rrouuaaaaeeeene www.ouuuuuuuuuuuuu...........m..
r. r.briitt fesetueefft.nm
ABSTRACT
When designing Convolutional Neural Networks (CNNs), one must select the size of the convolutional kernels before training. Recent works show CNNs benefit from different kernel sizes at different layers, but exploring all possible combinations is unfeasible in practice. A more efficient approach is to learn the kernel size during training. However, existing works that learn the kernel size have a limited bandwidth. These approaches scale kernels by dilation, and thus the detail they can describe is limited. In this work, we propose FlexConv, a novel convolutional operation with which high bandwidth convolutional kernels of learnable kernel size can be learned at a fixed parameter cost. FlexNets model long-term dependen- cies without the use of pooling, achieve state-of-the-art performance on several sequential datasets, outperform recent works with learned kernel sizes, and are competitive with much deeper ResNets on image benchmark datasets. Additionally, FlexNets can be deployed at higher resolutions than those seen during training. To avoid aliasing, we propose a novel kernel parameterization with which the frequency of the kernels can be analytically controlled. Our novel kernel parameter- ization shows higher descriptive power and faster convergence speed than existing parameterizations. This leads to important improvements in classification accuracy.
INTRODUCTION
The kernel size of a convolutional layer defines the region from which features are computed, and crucial choice in their design. Commonly, small kernels (up 7px) used almost exclusively and are combined with pooling to model long term dependencies (Simonyan & Zisserman, 2014; Szegedy al., 2015; He et al., 2016; Tan & Le, 2019). Recent works indicate, however, that CNNs benefit from using convolutional kernels (i) of varying size at different layers (Pintea et al., 2021; Tomen al., 2021), and (ii) at the same resolution of the data (Peng et al., 2017; Cordonnier et al., 2019; Romero et al., 2021). Unfortunately, most CNNs represent convolutional kernels as tensors of discrete weights and their size must fixed prior to training. This makes exploring different kernel sizes at different layers difficult and time-consuming due to (i) the large search space, and (ii) the large number of weights required construct large kernels.
A more efficient way to tune differrnnt kernel sizes at different layers is to learn them during training. Existing methods define discrete weighted set of basis functions, e.g., shifted Delta-Diracs (Fig. 2b, Dai et al. (2017)) or Gaussian functions (Fig. 2c, Jacobsen et al. (2016); Shelhamer et al. (2019); Pintea et al. (2021)). During training they learn dilation factors over the basis functions to increase the kernel size, which crucially limits the bandwidth of the resulting kemels.
In this work, we present the Flexible Size Continuous Kernel Convolution (FlexConv), a convolutional layer able to learn high bandwidth convolutional kernels of varying size during training (Fig. I). Instead of using discrete weights, we provide a continuous parameterization of convolutional kernels via small neural network (Romero et al., 2021). This parameterization allows us to model continuous functions of arbitrary size with a fixed number of parameters. By multiplying the response of the neural network with Gaussian mask, the size of the kernel can be learned during training (Fig. 2a). This allows us to produce detailed kernels of small sizes (Fig. 3), and tune kernel sizes efficiently.
Equal contribution.

---

Published as a conference paper at ICLR 2022
Cont. Function MLP
Crop
Kernel Indices x
Kernel
Dif. Mask
Figure 1: The Flexible Size Continuous Kernel Convolution (FlexConv). FlexConv defines convo- lutional kernels as the multiplication of a continuous convolutional kernel MLP with a Gaussian mask of local support ggaus: (x, y) Wgauss (I. y; m -MLP (I, By learning the parameters of the mask, the size of the convolutional kernel can be optimized during training. See also Fig. 7.
FlexConvs can be deployed at higher resolutions than those observed during training, simply by using more densely sampled grid of kernel indices. However, the high bandwidth of the kernel can lead FlexConv to learn kernels that show aliasing at higher resolutions, if the kernel bandwidth exceeds the Nyquist frequency. To solve this problem, we propose to parameterize convolutional kernels as Mulriplicative Anisotropic Gabor Networks (MAGNets). MAGNets are new class Multiplicative Filter Networks (Fathony et al., 2021) that allows to analyze and control the frequency spectrum of the generated kernels. We use this analysis to regularize FlexConv against aliasing. With this regularization, FlexConvs be directly deployed higher resolutions with minimal accuracy loss. Furthermore, MAGNets provide higher descriptive power and faster convergence speed than existing continuous kernel parameterizations (Schütt et al., 2017; Finzi al., 2020; Romero et al., 2021). This leads to important improvements in classification accuracy (Sec. 4).
Our experiments show that CNNs with FlexConvs, coined FlexNets, achieve state-of-the-an art across several sequential datasets, match performance of recent works with learnable kernel sizes with less compute, and are competitive with much deeper ResNets (He al., 2016) when applied on image benchmark datasets. Thanks to the ability of FlexConvs to generalize across resolutions, FlexNets can efficiently trained fow-resolution to save compute, e.g., 16 16 CIFAR images, and be deployed on the original data resolution with marginal accuracy loss, e.g., 32 x 32 CIFAR images.
In summary, our contributions are:
We introduce the Flexible Size Continuous Kernel Convolution (FlexConv), convolution operation able learn high bandwidth convolutional kernels of varying size end-to-end.
Our proposed Multiplicative Anisotropic Gabor Networks (MAGNets) allow for analytic control the properties the generated kernels. This property allows us to construct analytic alias-free convolutional kernels that generalize higher resolutions, and train FlexNets at low resolution and deploy them at higher resolutions. Moreover, MAGNets show higher descriptive power and faster convergence speed than existing kernel parameterizations.
CNN architectures with FlexConvs (FlexNets) obtain state-of-the-ar across several sequential datasets, and match recent works with learnable kernel size on CIFAR-10 with less compute.
2 RELATED WORK
Adaptive kernel sizes. Loog & Lauze (2017) regularize the scale of convolutional kernels for filter learning. For image classification, adaptive kernel sizes have been proposed via learnable pixel-wiss offsets (Dai et al., 2017), learnable padding operations (Han et al., 201 8), learnable dilated Gaussian functions (Shelhamer et al., 2019; Xiong et al., 2020; Tabernik et al., 2020; Nguyen, 2020) and scalable Gaussian derivative filters (Pintea al., 2021; Tomen et al., 2021; Lindeberg, 2021). These approaches either dilate discrete kernels (Fig. 2b), or use discrete weights on dilated basis functions (Fig. 2c). Using dilation erucially limits the bandwidth of the resulting kernels. In contrast, FlexConvs are able to construct high bandwidth convolutional kernels of varying size with a fixed parameter count. Larger kernels obtained simply by passing more positions the kernel network (Fig. 1).
Continuous kernel convolutions. Discrete convolutional kernel parameterizations assign an inde- pendent weight to each specific position in the kemel. Continuous convolutional kemels, the
2

---

Published as a conference paper at ICLR 2022
Be Kernal Son
Se Kernel Size
3+ Kernel Son
Cost Caas Beal
(a) FlexConv kernels (ours)
(b) Dilation deformation
(c) (Learmalle) parametric dilation (Pintea al., 2021)
(Dai et al., 2017)
Existing approaches increase the size of convolutional kemelss via (learnable) parametric dilations, e.g., by deformation (b) or by Gaussian blur (c). However, dilation limits the bandwidth of the dilated kernel and with it, the amount of detail it can describe. Contrarily, FlexNets extend their kernels by passing larger vector of positions to the neural network parameterizing them, As result, FlexConvs are able to learn high bandwidth convolutional kernels of varying size end-to-end (a).
Figure 2:
(a) Ground Truth
(b) Reconstructions at varying degrees of localization
Figure 3: The importance of dynamic sizes in continuous kernel convolutions. Consider neural network predicting pixel values at each position. If the entire image is considered, the network must use part of its capacity to learn to predict zeros outside the flower region, which in turn degrades the quality of the approximation in the region of interest (b). Importantly, the better the localization of the flower, the higher the approximation fidelity becomes. FlexNets learn the size of their convolutional kernels each layer during training, and thus (i) use the capacity of the kernel efficiently, (ii) converge faster to good approximations, and (iii) are faster execution -via dynamic cropping-.
other hand, view convolutional kernels as continuous functions parameterized via small neural network MLP": IRD + Nour with D the data dimensionality. This defines a convolutional kernel for which arbitrary input positions can be queried. Continuous kemelss have primarily been used to handle tiregularly-samplee data locally, e.g., molecular data (Simonovsky & Komodakis, 2017; Schüit al., 2017) and point-clouds (Thomas al., 2;18; Wang et al., 2018; Shi et al., 2019).
Recently, Romero et al. (2021) introduced the Continuous Kernel Convolution (CKConv) as tool to model long-term dependencies. CKConv uses a continuous kernel parameterization to construct convolutional kernels big as the input signal with constant parameter cost. Contrarily, FlexConvs jointly learn the convolutional kernel as well as its size. This leads to important ad advantages in terms of expressivity (Fig. 3), convergence speed and compute costs of the operation.
Implicit neural representations. Parameterizing convolutional kernel via a neural network can be seen learning implicit neural representation of the underlying convolutional kernel (Romero et al., 2021). Implicit neural representations construct continuous data representations by encoding data in the weights of neural network (Park et al., 2019; Sitzmann al., 2020; Fathony al., 2021).
We replace the SIREN (Sitzmann et al., 2020) kernel parameterization used in Romero et al. (2021) by our Multiplicative Anisotropic Gabor Networks: new class of Multiplicative Filter Networks (Fathony et al., 2021). MFNs allow for analytic control of the resulting representations, and allow us construct analytic alias-free convolutional kernels. The higher expressivity and convergence speed of MAGNets lead to accuracy improvements in CNNs using them as kernel parameterization.
3 METHOD
In this section, we introduce approach. First, introduce FlexConv and the Gaussian mask. Next, we introduce our Multiplicative Anisotropic Gabor Networks (MAGNets) and provide a description of our regularization technique used to control the spectral components of the generated kemel.
3.1 FLEXIBLE SIZE CONTINUOUS KERNEL CONVOLUTION (FLEXCONV)
To learn the kernel size during training, FlexConvs define their convolutional kernels 2 as the product of the output of a neural network MLP with a Gaussian mask local support. The neural network MLP parameterizes the kernel, and the Gaussian mask parameterizes its size (Fig. 1).
3

---

Published as a conference paper at ICLR 2022
Anisotropic Gaussian mask. Let (I;; ox )= exp (- ( (x- =x)2² be a Gaussian function parameterized by mean-variance tuple (ux, xx). The anisotropic Gaussian mask is defined as:
Wganus (z, yi ( of G(x; pu, G pY, py)
(1)
By learning (zx and (uv, of)) independently, anisotropic non-centered windows can be learned.
3.2 MULTIPLICATIVE ANISOTROPIC GABOR NETWORKS (MAGNETS)
In this section, we formalize our proposed parameterization for the kernel MLP". We start by introducing Multiplicative Filter Networks (Fathony et al., 2021), 1), and present our MAGNets next.
Multiplicative Filter Networks (MFNs). Recently, Fathony et al. (2021) proposed to construct implicit neural representations as the linear combination of exponentially many basis functions g:
(1) = g([x,)]:00)
g IR? + R Nui
(2)
h (w (1-1) b() [((=,y]:00)
w( R NaiaNNNoi b(() RNNid
(3)
(bx (T-1) + (T)
W(L) e NNNNN (T) RR-
(4)
where (0(), w(), (0) depict the learnable parameters of the bases and the affine transformations,,
and N, Nua depict the number of output and hidden channels, respectively. Depending on the selection of g. MFNs obtain approximations comparable to those of SIRENS (Sitzmann al., 2020)
with faster convergence rate. The most successful instantiation of MNFs are the Multiplicative Gabor Network (MGN): MFNs constructed with isotropic Gabor functions as basis g (in Eq. 2):
700)
g(1x, u]: v0: = =exp
2 (x- + (v (V- Sin (W(). [xx y] + b (()).
(5)
IR Nuid (0) Naid w(O E8 N b E NN (()) 00==(((0)
(6)
Note that, by setting N=N Nin an MFN can parameterize convolutional kernel with N input and Nout output channels. Fathony et al. (2021) show that MFNs are equivalent to a linear combination of exponentially many basis functions g. This allows us to analytically derive properties of MFN representations, and plays a crucial role in the derivation of alias-free MAGNets (Sec. 3.3).
Multiplicative Anisotropic Gabor Networks (MAGNets). Our MAGNet formulation is based on the observation that isotropic Gabor functions, i.e., with equal y for the horizontal and vertical directions, are undesirable basis for the construction of MFNs. Whenever frequency is required along certain direction, an isotropic Gabor function automatically introduces that frequency in both directions. As result, other bases must counteract this frequency in the direction where the frequency is not required, and thus the capacity of the MFN is not used optimally (Daugman, 1988).
Following the original formulation of the 2D Gabor functions (Daugman, 1988), we alleviate this limitation by using anisotropic Gabor functions instead:
g(<,,g):00) exp (- -=((>> (x 122 ) + (V- (()))
Sin( ( [z, y] +b
(7)
(1) 0(0) 00= E IR TY E IR Nni .ux Nna MY E RRNui w e Ia Naax2 (1)) E NoNa (8)
The resulting Multiplicative Anisotropic Gabor Network (MAGNet) obtains better control upon frequency components introduced to the approximation, and demonstrates important improvements in terms of descriptive power and convergence speed (Sec. 4).
MAGNet initialization. Fathony et al. (2021) proposes to initialize MGNs by drawing the size of the Gaussian envelopes, i.e., the 7()) term, from Gamma( a I. I.-1. ,3) distribution at every layer e [1,..,L- -- 1]. We observe however that this initialization does not provide much variability on the initial extension of the Gaussian envelopes and in fact, most of them cover a large portion of the space at initialization. To stimulate diversity, we initialize the {>/x) Ty (1) terms by Gamma ( ,3) distribution at the I-th layer. We observe that our proposed initialization consistently leads to better accuracy than the initialization of Fathony et al. (2021) across all tasks considered. (Sec. 4).
3.3 ANALYTIC ALIAS- FREE MAGNETS
FlexConvs can be deployed at higher resolutions than those observed during training, simply by sampling the underlying continuous representation of the kernel more densely, and accounting for the
4

---

Published as a conference paper at ICLR 2022
change in sampling rate. Consider D-dimensional input signal f,0) with resolution r(0), FlexConv learns kernel (b,01) that can be inferred at a higher resolution r(2) (Romero et al., 2021):
D
(Frce) (1(co) sd
(Freo B,.C
(9)
r((2)
Note however, that Eq. 9 holds approximately. This is due to aliasing artifacts which can appear if the frequencies in the learned kernel surpass the Nyquist criterion of the target resolution. Consequently, an anti-aliased parameterization is vital to construct kernels that generalize well to high resolutions.
Towards alias-free implicit neural representations. We observe that SIRENs as well as uncon- strained MFNs and MAGNets exhibit aliasing when deployed on resolutions higher than the training resolution, which hurts performance of the model. An example kernel with aliasing is shown in Fig. 8.
To combat aliasing, would like to control the representation learned by MAGNets. MAGNets -nd MFNs in general- construct implicit neural representations that can be seen as linear combination of basis functions. This property allows us to analytically derive and study the properties of the resulting neural representation. Here, we use this property to derive the maximum frequency of MAGGet--enerated kernels, so as to regularize MAGNets against aliasing during training. We analytically derive the maximum frequency of a MAGNet, and penalize it whenever it exceeds the Nyquist frequency of the training resolution. We note that analytic derivations are difficult for other implicit neural representations, e.g., SIRENs, due to stacked layer-wise nonlinearities.
Maximum frequency of MAGNets. The maximum frequency component of a MAGNet given by:
I. MAGGNEt = Σ max [=1
( ((() g.ū.. max
Jeut min ( (x.6 TY }
(21)
2T
2x
where I, corresponds to the number of layers. ( (1) (1) in Eq. 8, and =2 stdev to the cut-off frequency of the Gaussian envelopes in the Gabor filters.
7Xx ,Y to the MAGNet parameters as defined
A formal treatment as well as the derivations can be found in Appx. A.I.
Effeet of the FlexConv mask. The Gaussian mask used to localize the response of the MAGNet also has an effect on the frequency spectrum. Hence, the maximum frequency of FlexConv kernel is:
FiexConv
MMANN
Tipeess' with J max oy )
Ceut
(22)
Here, ox, oy correspond to the mask parameters (Bq. 1). Intuitively, multiplication with the mask blurs in the frequency domain, it equivalent to convolution with the Fourier transform of the mask.
Aliasing regularization of FlexConv kernels. With the analytic derivation of FieeCCony we penalize the generated kernels to have frequencies smaller or equal to their Nyquist frequency fiNyq (k) via:
CEF = =| (fiexcoo fNyq f (+)] with fNyq = k-1
(25)
Here, k depicts the size of the FlexConv kernel before applying the Gaussian mask, and is equal to the size of the input signal. In practice, we implement Eq. 25 by regularizing the individual MAGNet layers, as is detailed in Appx. A.2. To verify our method, Fig. 8 (Appx. A. shows that the frequency components of FlexNet kernels are properly regularized for aliasing.
4 EXPERIMENTS
We evaluate Flex across classification tasks on sequential and image benchmark datasets, and validate the ability of MAGNets to approximate complex functions. A complete description of the datasets used is given in Appx. B. Appx. D.2 reports the parameters used in all our experiments.
4.1 WHAT KIND OF FUNCTIONS CAN MAGNETS APPROXIMATE?
Bandwidth of methods with learnable sizes. First, we compare the bandwidth of MAGNet against N-Jet (Pintea et al., 2021) by optimizing each to fit simple targets: (i) Gabor filters of known frequency, (ii) random noise and (iii) an a 11 11 AlexNet kernel from the first layer (Krizhevsky et al., 201 2). Fig. 4 shows that, even with 9 orders of Gaussian derivatives, N-Jets cannot fit high frequency signals in large kernels. Crucially, N-Jet models require many Gaussian derivative orders to model high frequency signals in large kernels: hyperparameter which proportionally increases their inference
Our code is publicly available at https: / / wtw.tttboo./bbrbuuu. wttooruinn/eeeoor../
5

---

Published as a conference paper at ICLR 2022
Target
Nictonder
SIREN
MAGNNe
MANGNE
0.020
Oxbor (1256)
MAGNet-
0.015
MAGNet
N-jet onder.
MSE
N-jet order
Randoon
0.010
ordee
0.005
0.000
2x
8it
128x
frequency (Hz)
Figure 4: Left: Final MSE after fitting each model to Gabor filters of different frequencies. N-Jets can- not fit high frequencies. Right: Kernels learned by each model. SIREN and MAGNet can fit all targets. MAGNet-S: small MAGNet of size akin to N-Jets, still does well on the Gabor and AlexNet targets.
Table Test accuracy and ablation studies on sMNIST, PMNIST, SCIFARIO and npCIFARIO.
MODEI.
SIzE
sMNIST
PMNIST
SCIFARIO
NECIFARIO
DiRNN (Chang et al., 2017)
44K
98.0
96.1
IndRNN
2018)
TCN (Bai ct al., 2018a)
70K
99.0
97.2
=LSTM (Trinh 2018)
0.5M
98.4
95.2
72.2
Self-Ait. (Trinh et 2018)
0.5M
98.9
97.9
62.2
TrellisNet (Bai et al., 20188)
8M
99.20
98.13
73.42
2020b)
99
URGRU Zoncout (Go ct 2020b)
99.27
96.51
74.40
HiPPO
2020s)
0.5M
98.30
Lipechitz RNN (Erichson et al., 2020)
158K
99.4
97.3
64.2
$9.0
coRNN (Rusch Mishru, 2020)
134K
99.4
97.3
59.0
UnICORNN
2021)
135K
pLMU (Chilkuri & Eliusmith, 2021)
165K
98.49
CKCNN-2
98K
99.31
98.00
62.25
60.5
C.C
1M
99.32
98.54
63.74
62.2
CKTCN
105K
CKTCNoase-2
106K
99.52
98,
69.26
67.37
CKTCNNANNE-2
105k
99.55
98.57
74.58
67.52
FlexTCN-2
108K
99.60
98.61
78.99
67.11
FexTCN-4
FlexTCN-6
375K
99.62
98.63
80.82
69.87
FexTCNaRrNe66
HiexTCNRaii-66
370K
99.49
97.97
74.79
67.35
FleTTNNGa.-66
373K
99.50
37
78. 36
67. 67.56
FlexTCNasseer
375K
99.62
98.63
80.82
69.87
time and parameter count. MAGNets, on the other hand, accurately model large high frequency signals. This allows FlexNets learn large kernels with high frequency components.
Expressivity of MLP parameterizations. Next, we compare the descriptive power and convergence speed of MAGNets, Gabor MFNs, Fourier MFNs and SIRENs for image approximation. To this end, we fit the images in the Kodak dataset (Kodak, 1991) with each of these methods. Our results (Tab. 5) show that MAGNets outperform other methods, and converge faster to good approximations.
4.2 CLASSIFICATION TASKS
Network specifications. Here, we specify our networks for all our classification experiments. We parameterize all our convolutional kernels as the superposition of 3-layer MAGNet and learnable anisotropic Gaussian mask. We construct two network instances for sequential and image datasets respectively: FlexTCNs and FlexNets. Both are constructed by taking the structure of a baseline network -TCN (Bai et al., 201 8a) or CIFARResNet (He et al., 2016)-, removing all internal pooling layers, and replacing convolutional kernels by FlexConvs. The FlexNet architecture is shown in Fig. 10 and varies only in the number of channels and blocks, e.g., FlexNet-16 16 has 7 blocks. Akin to Romero et al. (2021) we utilize the Fourier theorem to speed up convolutions with large kernels.
Mask initialization. We initialize the FlexConv masks to be small. Preliminary experiments show this leads to better performance, faster execution, and faster training convergence. For sequences, the mask center is initialized the last kernel position to prioritize the last information seen.
Time series and sequential data. First we evaluate FlexTCNs on sequential classification datasets, for which long-term dependencies play important role. We validate approach on intrinsic

---

Published as a conference paper at ICLR 2022
Table Results on CIFAR-10. Results from *orig- inal works and i single run.
Table 2: Test accuracy on CT, SC and SC_raw
raw
MODEL
S1ZE
CT
SC
SC_RAW
CIFAR-I0 - ACC.
TIME (SEC/EPOCH)
GRU-ODE
89k
96.2
44,8
-10.0
MODEL
GRU-A
89k
97.8
20.0
-100
GRU-D
89k
95.9
23.9
-10.0
CIFARResNeI-44
0.66M
92
22
ODE-RNN
89k
97.1
93.2
-10.0
DCN
0.47M
89.7 = 0.3*
NCDE
89k
98.8
88.5
-10.0
N-JR-CIFARRReNNG22
0.52M
92.3 0.3*
N-Jet-ALLCNN
1.07M
92.5 0.1*
CKCNN
100K
99,53
95.27
71.66
CKTCN
95.65
74,90
FlexNet-16 w/ conv. (k 3)
89.5
CKTCN
96.66
78.10
FlexNet-16 w/ conv. (k 33)
20.0M
78.0 0.3
242
CKTCN
105K
99.53
97.01
FlexNet--6 N-Jet
0.70M
91.7
FlexTCN-2
1055K
99.53
97.10
88.03
CKCNN-16
0.63M
72.1 0.2
68
FIeTTCN
239K
99.53
97.73
90.45
CKCNNewau-16
0.67M
86.8
FeeTCN-6
373K
99.53
97.67
91.73
FlexNotswex--1 16
0.63M
89.0 0.3
89
C
370K
95.83
FlexNetraae-166 16
0.67M
91.9 0.2
161
Oreaaan 6
342K
97.62
91.02
FlexNctoass- 16 + anis. Geuss.
0.67M
92.0 0.1
147
373K
97.35
91.50
FlexNeteaw- 16 Gabor
0.67M
92.0
150
FiexTCNassss-66
373K
97.67
91.73
F6
0.67M
92.2 0.1
127
discrete data: sequential MNIST, permuted MNIST' (Le et al., 2015), sequential CIFARIO (Chang et al., 2017), noise-padded CIFARIO (Chang et al., 2019), as well as time-series data: CharacterTrjjectorie (CT) (Bagnall et al., 2018), SpeechCommandh (Warden, 2018) with raw waveform (SC_raw) and MFCC input representations (SC).
Our results are summarized in Tables 1 and 2. FlexTCNs with two residual blocks obtain state-of- the-art results on all tasks considered. In addition, depth further improves performance. FlexTCN-6 improves the current state-of-the-art on SCIFAR10 and npCIFAR10 by more than 6%. On the difficult SC_raw dataset- -with sequences of length 16000-, FleXTCN-6 outperform the previous state-of-the-ard by 20.07% a remarkable improvement.
Furthermore, we conduct ablation studies by changing the parameterization of MLP", and switching off the learnable kernel size ("CKTCNs") and considering global kernel sizes instead. CKTCNs and FlexTCNs with MAGNet kernels outperform corresponding models with all other kernel parameteri- zations: SIRENs (Sitzmann et al., 2020), MGNs and MFNs (Fathony et al., 2021). Morcover, we see consistent improvement with respect to CKCNNs (Romero et al., 2021) by using learnable kernel sizes. This shows that both MAGNets and learnable kernel sizes contribute to the performance of FlexTCNS.. Note that in ID, MAGNets are equivalent MGNs. However, MAGNets consistently perform better than MGNs. This improvement in accuracy is a result of our MAGNet initialization.
Image classification. Next, we evaluate FlexNets for image classification on CIFAR-10 (Krizhevsky et al., 2009). Additional experiments on Imagenet-32, MNIST and STL-10 can be found in Appx. C.
Table 3 shows our results on CIFAR-10. FlexNets are competitive with pooling- based methods such as CIFARResNet (He et al., 2016) and outperform learnable kernel size method DCNs (Tomen al., 2021). In addition, we compare using N-Jet layers of order three (as in Pintea al. (2021)) in FlexNets against using MAGNet kernels. We observe that N-Jet layers lead to worse performance, and are significantly slower than FlexConv layers with MAGNet kernels. The low accuracy of N-Jet layers is likely to be linked to the fact that FlexNets do not use pooling. Consequently, N-Jets are forced to learn large kernels with high-frequencies, which we show N-Jets struggle learning in Sec. 4.1.
To illustrate the effect of learning kernel sizes, we also compare FlexNets against FlexNets with large and small discrete convolutional kernels (Tab. 3). Using small kernel sizes is parameter efficient, but is not competitive with FlexNets. Large discrete kernels on the other hand require copious amount of parameters and lead to significantly worse performance. These results indicate that the best solution is somewhere in the middle and varying kernel sizes learn the optimal kernel size for the task at hand.
Similar to the sequential case, we conduct ablation studies on image data with learnable, non- learnalle kernel sizes and different kernel parameterizations, Table 3 shows that FlexNets outperform CKCNNs with corresponding kernel parameterizations. In addition, a clear difference in performance is apparent for MAGNets with respect to other parameterizations. These results corroborate that both MAGNets and FlexConvs contribute to the performance of FlexNets. Moreover, Tab. 3 illustrates the effect of the two contributions of MAGNet over MGN: anisotropic Gabor filters, and our improved initialization. Our results in image data are in unison with our previous results for sequential data (Tabs. 1, 2) and illustrate the value of the proposed improvements in MAGNets.
7

---

Published as a conference paper at ICLR 2022
Accuracy change at upsampling
Accuracy change after fine-tuning
MAGNet HileyCone
16
-1.7%
-1.2%
-3.4%
-1.9
-1.%
+3.2%
+3.0%
+5.9%
+1.0%
+3.8%
20
-0.4%
-0.5%
-144
-0.4%
-0.4%
+1.2%
+1.4%
+3.0%
+1.7%
+1.8%
16x16
7.4%
18.6%
17.3%
14.7%
15.6%
19.3%
x0
-5.1%
-3.8%
-3.9%
-4.8%
-4.5%
+6.3%
+5.1%
+5.7%
+5.9%
+5.6%
28
30
32x32
31x31
36x36
28x:28
30x30
32x32
36x31
36 36
Figure 5: Alias-free FlexNet-16 16 CIFAR-10. We report change in accuracy between source and
target resolutions, directly after upsampling (left) and after fine-tuning (right) (means over five runs).
4.3 ALIAS-FREE FLEXNETS
Regularizing the FlexConv mask. Though including fu in the frequency analysis of MAGNets crucial for the accuracy of the derivation, including the FlexConv mask in aliasing regularization is undesirable, as it steers the model to learn large kernels in order to minimize the loss (see Eq. 25). However, excluding the mask from regularization could compromise the ability of FlexNet to generalize to higher resolutions. Here, we experiment with this trade-off.
Figure shows accuracy change between ten source and target resolution combinations on CIFAR-10, both for including and excluding the FlexConv mask in the aliasing regulariza- tion. We train at the source resolution for 100 epochs, before testing the model at the target resolution with the upsampling described in Sec. 3.3. Next, we adjust fNyg (k) to the tar- get resolution, and finctune each model for 100 epochs at the target resolution.
Table 4: Alias-free FlexNets on CIFAR-10.
CIFAR-10 ACC.
MODEL
S1ZE
16
Aiga 32 px
CieARResNett44
0.66M
85.8 = 0.2
-31.6 1,3
FlexNet-16 16 w/ conv. (k= 3)
0.17M
85.3 0.2
-21.2 1.0
FlexNet-16 conv. (k 33)
20.0M
67.7 a 0.6
-57.1
FlexNet- 16 w/ N-Jets
0.70M
86.4 0.2
-5.5 1.3
CKCNN- 16gHEN
0.63M
45,9 a: 1.0
-15,8 1.2
FlexNet-16aaes
0.63M
70.4 a 0.8
-50.0 a 16.9
FlexNet-16 w/o reg.
0.67M
86.4 = 0.4
-34.4 = 14.3
FlexNet-16 w/ reg. fanaNe F6 w/ reg. JrieaCom
0.67M
86.5 0.1
-3.8 a 2.0
We find that regularizing just fMAGN yields a trade-off. It increases the accuracy difference between low and high resolution inference, but also increases the fine-tune accuracy at the target resolution. We therefore choose to, by default, regularize MMAGNE only.
0.67M
85.1 0.3
-3.3
Results of our alias-free FlexNet training on CIFAR-10 are in Table 4. We observe that the perfor- mance of a FlexNet trained without aliasing regularization largely breaks down when the dataset is upscaled. However, with our aliasing regularization most of the performance is retained.
Comparatively, FlexNet retains more of the source resolution performance than FlexNets with N-Jet layers, while baselines degrade drastically at the target resolution. Fig. 8 shows the effect of aliasing regularization on the frequency components of FlexConv.
Training at lower resolutions saves compute. We can train firee-free FlexNets at lower resolutions. To verify that this saves compute, we time the first 32 batches of training FlexNet-7 on CIFAR-10 10. We compare against training on 16 x 16 images (downsampled before training). On 16x16 images, each batch takes 179ms (+ 7ms). On 32x32 images, each batch takes 222ms (= 9ms). Therefore, we save 24% training time when training FlexNets alias-free at half the native CIFAR-10 resolution.
DISCUSSION
Learned kernel sizes match conventional priors. Commonly, CNNs use architectures of small kemels and pooling layers. This allows convolutions to build progressively growing receptive field. With learnable kernel sizes, FlexNet could learn a different prior over receptive fields, e.g., large kernels first, and small kernels next. However, FlexNets learn to increase kernel sizes progressively (Fig. 6), and match the network design that has been popular since AlexNet (Krizhevsky et al., 2012).
Mask initialization as a prior for feature importance. The initial values of the FlexConv mask can be used to prioritize information at particular input regions. For instance, initializing the center be used prioritize information from the
of mask on the first element of sequential FlexConvs far past. This prior is advantageous for tasks such as npCIFAR1O. 10.. We observe that using this prior on npCIFAR11 10 leads much faster convergence and better results (68.33% acc. w/ FlexTCN-22.
8

---

Published as a conference paper at ICLR 2022
Depth Figure 6: Learned FlexConv masks for FlexNets with 3, 5 and 7 residual blocks. FlexNets learn very small kernels shallow layers, which become larger function of depth.
MeGNet regularization as prior induction. MAGNets allow for analytic control of the properties of the resulting representations. We this property to generate alias-free kernels. However, other desiderata could be induced, e.g., smoothness, for the construction of implicit neural representations.
Benefits of cropping and the influence of PyTorch. Dynamic cropping adjust the computational cost of the convolutions on the fly. For a signal of size and a cropped kernel size k, this incurs in savings from O(M²)) " ) to O(MkkD) relative to using global kernel sizes (O(M)) to OM?2²² in 2D). test this theoretical speed up controlled environment for the Speech Commands and CIFAR- 10 datasets. Cropping reduces the per-epoch time by factor of 11.8x and 5.5x for Speech Commands and CIFAR-10, respectively. Interestingly, however, both run times become similar if the flag torch. backends cudnn. benchmark is activated, with global kernel sizes being sometimes faster. This is because this flag tells PyTorch to optimize the convolution algorithms used under the hood, and some of these CUDA algorithms seem to be faster than our masking strategy on Python.
LIMITATIONS
computation and memory cost of convolutions with large kernels. Per- forming convolutions with large convolutional kernels is compute-intensive operation. FlexConvs are initialized with small kernel sizes and their inference cost is relatively small at the start of training. Tab. 3).
Dynamic kernel sizes:
However, despite the cropping operations used improve computational efficiency (Figs. the inference time may increase up to double as the learned masks increase in size. At the cost of more memory, convolutions can be sped up by performing them in the frequency domain. However, we observe that this does not bring gains for the image data considered because FFT convolutions are faster only for very large convolutional kernels (in the order hundreds of pixels).
Remaining accuracy drop in alias-free FlexNets. Some drop in accuracy is still observed when using alias-free free FlexNets at a higher test resolutions (Tab, 4). Although more evidence is needed, this may be caused by aliasing effects introduced by ReLU (Vasconcelos et al., 2021), or changes in the activation statistics of the feature maps passed to global average pooling (Touvron et al., 2019).
7 CONCLUSION
We propose FlexConv, convolutional operation able to learn high bandwidth convolutional kemels of varying size during training at a fixed parameter cost. We demonstrate that FlexConvs are able model long-term dependencies without the need of pooling, and shallow pooling-free FlexNets achieve state-of-the-art performance on several sequential datasets, match performance of recent works with learned kernel sizes with less compute, and are competitive with much deeper ResNets on image benchmark datasets. In addition, we show that our alias-free free convolutional kernels allow FlexNets to be deployed higher resolutions than seen during training with minimal precision loss.
Future work. MAGNets give control over the bandwidth of the kernel. We anticipate that this control has more uses, such as fighting sub-sampling aliasing (Zhang, 2019; Kayhan & Gemert, 2020; Karras et al., 2021). With the ability to upscale Flex to different input image sizes comes the possibility of transfer learning representations between previously incompatible datasets, such as CIFAR-I0 and Imagenet. In similar vein, the automatic adaptation of FlexConv to the kernel sizes required for the task at hand may make it possible to generalize the FlexNet architecture across different tasks and datasets. Neural architecture search (Zoph& & Le, 2016) could see benefits from narrowing the search space to exclude kernel size and pooling layers. In addition, we envisage additional improvements from structural developments of FlexConvs such as attentive FlexNets.

---

Published as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
We hope to inspire others to use and reproduce our work. We publish the source code of this work, for which the link is provided in Sec. 4.2. Sec. 4 and Appx. D.I detail FlexNet, its hyperparameters and optimization procedure. The full derivation of the aliasing regularization objective is included in Appx. A.I. We report means over multiple runs for many experiments, to ensure the reported results are fair and reproducible, and do not rely tuning of the random seed. All datasets used in our experiments are publicly available. If any questions remain, we welcome one and all to contact the corresponding author.
ACKNOWLEDGMENTS
We thank Nergis Tömen for her valuable insights regarding signal processing principles for Flex Conx,onv, and Silvia-Laura Pintea for explanations and access to code of her work Pintea et al. (2021). We thank Yerlan Idelbayev for the use of the CIFARResNet code.
This work is co-supported by the Qualcomm Innovation Fellowship granted to David W. Romero. David W. Romero sincerely thanks Qualcomm for his support. David W. Romero financed as part of the Efficient Deep Learning (EDL) programme (grant number P16-25), partly funded by the Dutch Research Council (NWO). Robert-Jan Bruintjes is financed by the Dutch Research Council (NWO) (project VI. Vidi. 192.100) All authors sincerely thank everyone involved in funding this work.
This work was partially carried out on the Dutch national infrastructure with the support of SURF Cooperative. We used Weights & Biases (Biewald, 2020) for experiment tracking and visualizations.
REFERENCES
Anthony Bagnall, Hoang Anh Dau, Jason Lines, Michael Flynn, James Large, Aaron Bostrom, Paul Southam, and Eamonn Keogh. The multivariate time series classification archive, 2018. arXiv preprint XXiv:/8//.0007:: 2018.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint vrXiv:1803.0127) 2018a.
Shaojie Bai, Zico Kolter, and Vladlen Koltun. Trellis networks for sequence modeling. arXiv preprint 1810.v08100066 2018b.
Lukas Biewald. Experiment tracking with weights and biases, 2020. URL https: wandb. com/. Software available from wandb.com.
Bo Chang, Minmin Chen, Eldad Haber, and Ed H Chi. Antisymmetricran: A dynamical system view recurrent neural networks, arXiv preprint ArXix
2019.
Shiyu Chang, Yang Zhang, Wei Han, Mo Yu, Xiaoxiao Guo, Wei Tan, Xiaodong Cui, Michael Witbrock, Mark A Hosegawa-Johnson, and Thomas S Huang. Dilated recurrent neural networks. In Advances in neural information processing systems, pp. 77-87, 2017.
Narsimha Chilkuri and Chris Eliasmith. Parallelizing legendre memory unit training, arXiv preprint arXiv:2102.114177 2021.
Patryk Chrabaszez, Ilya Loshchilov, and Frank Hutter. downsampled variant of imagenet as an alternative to the CIFAR datasets. CoRR, abs/1707 hos//777.00819 19, 2017. URL http: / /aw.iv.vvvooo/////o abs/ foss/1007. 08819.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-211222 223. JMLR Workshop and Conference Proceedings, 2011.
Cean-Baptiste Cordonnier. Andreas Loukas, and Martin Jaggi. On the relationship between self- attention and convolutional layers. arXiv preprint arXiv: 1ee.1/91113300 1.03584, 2019.
10

---

Published as a conference paper at ICLR 2022
Jifeng Dai, Haozhi Qi, Yuwen Xiong. Li, Guodong Zhang, Han Hu, and Yichen Wei. Deformable convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 764-773, 2017.
J.G. Daugman. Complete discrete 2-d gabor transforms by neural networks for image analysis and compression. IEEE Transactions on Acoustics, Speech, and Signal Processing, 36(7): 169-11199 1988. doi:
Benjamin Erichson, Omri Azencot, Alejandro Queiruga, Liam Hodgkinson, and Michael W Mahoney. Lipschitz recurrent neural networks. arXiv preprint arXiv:2006. /20000//2200 /2070, 2020.
Rizal Fathony, Anit Kumar Sahu, Devin Willmott, and J Zico Kolter. Multiplicattive filter networks. In International Conference on Learning Representations, 2021. URL httpss:///ppeeeeeiiii net/ / Eorum? 1d=OmtmcPkkhT.
Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. arXiv preprint arXiv:2002.11288: 2020.
Albert Gu, Tri Dao, Stefano Ermon, Atri Rudra, and Christopher Ré. Hippo: Recurrent memory with optimal polynomial projections. arXiv preprint 0rXiv:2008.07660 2020a.
Albert Gu, Caglar Gulcehre, Thomas Paine, Matt Hoffman, and Razvan Pascanu. Improving the gating mechanism of recurrent neural networks. In International Conference on Machine Learning, pp. 2800-3809. PMLR, 2020b.
Anders Hald. De moivre's normal approximation the binomial, 1733, and its generalization. A History of Parametric Statistical Inference from Bernoulli to Fisher, 1713-1935, pp. 17-24, 2007.
Shizhong Han, Zibo Meng, Zhiyuan Li, James O' ReReily Jie Cai, Xiaofeng Wang, and Yan Tong, Optimizing filter size convolutional neural networks for facial action unit recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448-456. PMLR, 2015.
Jom--enrii Jacobsen, Jan Van Gemert, Zhongyu Lou, and Amoll WM Smeulders. Structured receptive fields in enns. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 2610-2619, 2016.
HM Kabir, Moloud Abdar, Seyed Mohammad Jafar Jalali, Abbas Khosravi, Amir F Atiya, Saeid Nahavandi, and Dipti Srinivasan. Spinalnet: Deep neural network with gradual input. arXiv preprint arXiv:2007.033:2 2020.
Tero Karras, Miik Aittala, Samuli Laine, Erik Härkönen, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Alias-free generative adversarial networks. arXiv preprini arXiv:2 12423, 2021.
Osman Semih Kayhan and Jan C. van Gemert, On translation invariance in cnns: Convolutional layers can exploit absolute spatial location. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Patrick Kidger, James Morrill, James Foster, and Terry Lyons. Neural controlled differential equations for irregular time series. arXiv preprint arXiv:2005.0880 08926, 2020.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 1rXiv:1412.6980, 2014.
Kodak. Kodak dataset, 1991. URL URU.http::/rkk..s//rrraae

---

Published as a conference paper at ICLR 2022
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. Technical report, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. ImageNet Classification with Deep Convolutional Neural Networks. In Pereira, C. J. C. Burges, Bottou, and K. Q. Wein- berger (eds.), Advances in Neural Information Processing Systems, volume 25. Curran Asso- ciates, Inc., 2012. URL https: httpp//ooceeens..ccpooo hewpiinnnnnnnnnngnnneneeeeeeenne neurips http./aaar//..2/fii....... c399862d3b9d663 6c84668843330 6684 5bb4 pdf.
Quoc V Le, Navdeep Jaitly, and Geoffrey E Hinton. A simple way initialize recurrent networks of rectified linear units. arXiv preprint arXi::1504.00944 2015.
Yann LeCun and Corinna Cortes. MNIST handwritten digit database. 2010. URL http: /aap:://yannn /yann.
lecun. www.cooeeeeeoooo. wwtp//tw/////e//et/tt/oo//eeee/t wwwd//nnssstteeeeeeeeeeeemmmmmme /wtps//ns///.mmommee/
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-23)) 2324, 1998.
Shuai Li, Wanqing Li, Chris Cook, Ce Zhu, and Yanbo Gao. Independently recurrent neural network (indrnn): Building a longer and deeper rnn. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5457-5466, 2018.
Min Lin, Qiang Chen, and Shuicheng Yan. Network in network. arXiv preprint arXiv: 1xXiv:1312.4400 2013.
Tony Lindeberg. Scale-covariant and scale-invariant gaussian derivative networks. In Scale Space and Variational Methods in Computer Vision :, volume 12679 of Springer Lecture Notes in Computer Science, pp. 3-14. Springer Nature, 2021. ISBN 978- 978-3-030-75548-5 doi: 10.1007/ 14759. Not duplicate with
978-3-030 75580--5554--22 I. URL https://arxii, org/ abs 2011.
DiVA 1505585QC 20210317.
Marco Loog and Francois Lauze. Supervised scale-regularized linear convolutionary filters. In Gabriel Brostow Tae-Kyun Kim, Stefanos Zafeiriou and Krystian Mikolajezyk (eds.), Proceedings of the British Machine Vision Conference (BMVC), pp. 162.1-162.11. BMVA Press, September 2017. ISBN 1-901725-60-X. doi: 10.5244/C.11.16. URL https: /dxx doi org/10. 5244 /2.44/C.30 31 162.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint 1XXv::6688003883 2016.
Chunjie Luo. Jianfeng Zhan, Lei Wang, and Wanling Gao. Extended batch normalization. arXiv preprint arXiv:2003.05566 2020.
Vittorio Mazzia, Francesco Salvetti, and Marcello Chiaberge. Efficient-capsnet Capsule network with self-attention routing. arXiv preprint arXiv:2101. 12491, 2021.
Due Nguyen. Robust deep learning for computer vision to counteract data scarcity and label noise. PhD thesis, 01 2020.
Jeong Joon Park, Peter Florence, Julian Straub, Richard Newcombe, and Steven Lovegrove. Deepsdf: Learning continuous signed distance functions for shape representation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 165-174, 2019.
Chao Peng, Xiangyu Zhang, Gang Yu, Guiming Luo, and Jian Sun. Large kernel matters-improv semantic segmentation by global convolutional network. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 4353-4361. 2017.
Silvia L Pintea, Nergis Tomen, Stanley F Goes, Marco Loog, and Jan van Gemert. Resolution learning in deep convolutional networks using scale-space theory. arXiv preprint arXiv:2106.03411 2021.
David W Romero, Anna Kuzina, Erik Bekkers, Jakub M Tomezak, and Mark Hoogendoorn. Ckcony: Continuous kernel convolution for sequential data. arXiv preprint 102.v:2102.02611 I, 2021.
12

---

Published as a conference paper at ICLR 2022
T Konstantin Rusch and Siddhartha Mishra. Coupled oscillatory recurrent neural network (comn)): An accurate and (gradient) stable architecture for learning long time dependencies. arXiv preprint TrXiv:2010.00051, 2020.
Konstantin Rusch and Siddhartha Mishra. Unicornn: A recurrent model for learning very long time dependencies. arXiv preprint arXiv:2103.05487, 2021.
Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg. and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115 (3):211-253) 2015. doi: 10.1007/s1 1263-015-081644
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmicla, Alexandre Tkatchenko, and Klaus- Müller. Schnet: A continuous-fille convolutional neural network for modeling quantum interactions. In Advances in neural information processing systems, pp. 991-1001, 2017.
Evan Shelhamer, Dequan Wang, and Trevor Darrell. Blurring the line hetween structure and learning to optimize and adapt receptive fields. ArXiv, abs/1904.1 1487, 2019.
Shaoshuai Shi, Zhe Wang, Jianping Shi, Xiaogang Wang, and Hongsheng Li. From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network. arXiv preprint 190%:0:907.03677 2019.
Martin Simonovsky and Nikos Komodakis. Dynamic edge-conditioned filters in convolutional neural networks on graphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3693-3702, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Vincent Sitzmann, Julien Martel, Alexander Bergman, David Lindell, and Gordon Wetzstein. Im- plicit neural representations with periodic activation functions. Advances in Neural Information Processing Systems, 33, 2020.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. Journal of Machine Learning Research, 15(56):1929 1988-199999998 2014. URL http: http://jmlr.. org/ parg/papers. / v srivastavale 4a 1at.aallaalll............
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
Domen Tabernik, Matej Kristan, and Ales Leonardis. caatially-adaptii filter units for compact and efficient deep neural networks. International Journal of Computer Vision, 128, 09 2020. doi: 10.1007/s1 1263-019-0122--11
Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pp. 6105-61 14, PMLR, 2019.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang. Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation -equivvainnnnuiviiiaiiiiaa neural networks for 3d point clouds. arXiv preprini arXiv: 0288802002211 19, 2018.
Nergis Tomen, Silvia-Laura Pintea, and Jan Van Gemert. Deep continuous networks. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference Machine Learning, volume 139 of Proceedings of Machine Learning Research, pp. 10324-10335. PMLR, 18-24 Jul 2021. URL https::pp/............ proccedd ngs ml mlr. www.:reerr.........mmmmmm.mm.m.. / 39/13//tmtmmm1m11 html.
Hugo Touvron, A. Vedaldi, M. Douze, and H. Jégou. Fixing the train-test resolution discrepancy. In NeurlPS, 2019.
13

---

Published as a conference paper at ICLR 2022
Trieu H Trinh, Andrew M Dai, Minh-Thang Luong, and Quoe V Le. Learning longer-term dependen- cies in rnss with auxiliary losses. arXiv preprint arXiv:/803.00144 2018.
Cristina Vasconcelos, Hugo Larochelle, Vincent Dumoulin, Rob Romijnders, Nicolas Le Roux, and Ross Goroshin, Impact of aliasing on generalization in deep convolutional networks, arXiv preprint orXiv:2108.03489, 2021.
Sheniong Wang, Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky. and Raquel Urtasun. Deep parametric continuous convolutional neural networks. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 2589-2597, 2018.
Pete Warden. Speech commands: A dataset for limited-vocabulary speech recognition. arXiv preprint arXiv:1884.03320, 2018.
Zhitong Xiong, Yuan Yuan, Nianhui Guo, and Qi Wang. Variational context-deformable convnets for indoor scene parsing. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. arXiv preprint arXiv: 1605.07746, 2016.
Richard Zhang. Making convolutional networks shift- invariant again. In International conference on machine learning, pp. 7324-7334. PMLR, 2019.
Barret Zoph and Quoc V Le. Neural architecture search with reinforcement learning, arXiv preprint 1rX1v:161..0157,, 2016.
14

---

Published as a conference paper at ICLR 2022
(a) M output
(b) [( ((<z,y):00)
(c) (x,,))
(a) (b)
(d) After cropping
Figure Example kernels, generated step by step. FlexConv samples kernel from MLP (a), which attenuated by anistropic Gaussian envelope with learned parameters ((1) (b), creating (c) which is cropped to contain only values of > 0.1 (d).
A ALIAS-FREE FLEXCONV REGULARIZATION
In this section we provide the complete derivation and analysis for our FlexConv regularization against aliasing. First, we derive the analytic maximum frequency component of FlexConv kernel. Next, we compute the Nyquist frequency of FlexConv kernel, and subsequently show how to combine the previous results into regularization term to train alias-free FlexConvs.
A.1 ANALYZING THE FREQUENCY SPECTRUM OF FLEXCONV
In order to make FlexConv alias-free free (Sec. 3,3), we need to compute the maximum frequency component of the kernels generated by MAGNet, that this section we analytically derive this maximum frequency component from the parameters of the MAGNet.
regularize it during training. In
Recall that MAGNets generate kernel (zx y) through of a succession of anisotropic Gabor filters and linear layers (Sec. 3.2, Eqs. 2-7):
( ([Ix,,];001))
g: R.2 ~ R.ad
h (w (1-1) + (0)) . ((1x,)):00))
w()) e R NNa ( RR Nua
(F.,)) w('h (L-1) (())
W(1)) R (Nou N b(L) e IR (Nou
((0,,)):00)) = exp ( (a- (x- (2) (u- Sin [T, u] bld)
0 = E Nusa 10 RR.Nui pxx ! TR. Nuia py !1) Naa w!) IR NMaxx b
IR Nuia
To analyse the maximum frequency component JMAGNe we analyse the frequency components of the Gabor filters used in MAGNet, and retain their maximum. We then plug the found frequency component into the analysis of Fathony et (2021) show how the frequency responses of Gabor filters and linear layers interact in MFNs. Finally, we add the effect of the FlexConv Gaussian mask to our analysis to obtain the maximum frequency component ot the final FlexConv kernel fHiexConn
Sine term in Gabor filters. In Gabor filter, the sine term is multiplied with Gaussian envelope. The frequency (in radians) of sine function the form Sin(w) (ww" [x. v) + b) is given by w. We divide by 2x to convert the frequency units to Hertz, for compatibility with the rest of the analysis. For 2D inputs, the maximum frequency component of the sine function correspond to the largest frequency in the two input dimensions:
Wi
J = max
(10)
2mT
The sine terms in MAGNets have multiple output channels: Sin Wg [x. b(O). Effectively, we compute the sine term independently for each channel:
W g.4.
fSm. max
(1I)
2T
15

---

Published as a conference paper at ICLR 2022
Block FlexConw 17x17
log10IFFT)
65x65
log10
(a) No regularization, block
Block FlexConv 17x17
logio/rFT:
Kernel sampled at 65x85
log10(FFT)
(b) Regularizing MaNGNa block 4 of 7.
Block FlexConv 17x17
log10/FFT:
sampled 65x65
log10
(c) Regularizing friesCone block 4
Figure 8: Example kernels from FlexNet-16 models trained (i) without regularization, (ii with aliasing regularization of MAGNNr (iii) with aliasing regularization of Jiiexoom In the columns, from left right: (i) original kernel 33 33, (ii) FFT of the original kernel, (iii) kernel inferred at 65 x 65, to find aliasing effects, (iii)) FFT of the 65 65 kernel, with the solid line showing the Nyquist frequency of the 33 33 kernel, and the red dotted line showing the maximum frequency component as computed by our analysis. For FFlexCony the maximum frequency matches almost exactly with the Nyquist frequency, showing that our aliasing regularization works. For fMAGNe the maximum frequency is slightly higher than the Nyquist frequency, as the FlexConv mask is not included in the frequency term derivation. This reflected in the slightly worse resolution generalization results reported in Sec. 4.3. Furthermore, some aliasing effects are still apparent for the aliasing regularized models, as discussed in Sec. 6.
16

---

Published as a conference paper at ICLR 2022
Gaussian term in Gabor filters. In Gabor filter, Gaussian envelope modulates sine term. Let us assume for now that the Gaussian envelope is isotropic, rather than anisotropic as in MAGNets, and has single-channel output. By applying the convolution theorem, the sine term is equivalently convolved with the Fourier transform of the Gaussian envelope in the frequency domain. Since the Fourier transform of Gaussian envelope is another Gaussian envelope, the application of Gaussian envelope amounts to blurring with Gaussian kernel in the frequency domain. The size of the envelope in the Fourier domain dp can be derived from the standard deviation of the Guassian envelope in the spatial domain or as follows:
OTOF
OF 2TTT
(12)
2x
Gaussian blurs induce impulse signals to have a long tail. Consequently, we must define a cutoff point for this tail terms of standard deviations derive the maximum added frequency induced by the blur. We describe the cutoff point as oout E N. Typical choices for cut are known as the empirical, or the "68-95-99.7" rule (Hald, 2007). We choose a standard of two standard deviations, ie., 22, which covers 95% of the mass of the Gaussian envelope.
For an isotropic Gabor filter with y=oT² the maximum frequency of its Gaussian envelope feav is:
fen 2T (oT))
Ocut
Ocut Y
(13)
2m
Anisotropic envelopes. Our analysis so far assumes an isotropic Gaussian envelope in the Gabor filter. However, need to account for the anisotropic Gaussian envelopes in MAGNets. Anisotropic filters have not one but two 7 parameters: (7x, }. The smallest of these will contribute most to fenv. as it will blur the most, so it is sufficient compute fönv only using the smallest of the two terms:
Seav (7x, ( Seav (min(7x, 7Y)).
(14)
The other assumption we made before was to work with single-channel outputs. MAGNets however use multi-channel outputs with independent Gaussian terms. The maximum frequency of multi- channel Gaussian envelopes is given by:
fenv, (7x, 7Y) = ft (min (7x,,, ry,3)) Cutt min Tv.))
(15)
2x
where the subscript i indexes the channels of the multi-channel Gaussian envelopes.
Maximum frequency component of anisotropic Gabor filters. Finally, the maximum frequency component of the i-th channel of an anisotropic Gabor filter g is given by:
fGabor, SSim, (Wa) 2)) + Setv, TY)
W g.i.j Ccut min (x.. yv
mmxx 3
(16)
2x
2x
Figure illustrates the frequency spectrum of an example Gabor filter.
Maximum frequency component of MAGNet. Fathony et al. (2021) characterize the expansion of each term the isotropic Gabor layers MENs the final MFN output. In Eq. 25, Fathony et al. (2021) 1) demonstrate that the MFN representation contains set of sine frequencies w given by:
(L) (T 51-1W i
(L-1) +...+ stw w.
(2) (1)
(17)
Here, the indexes 12,42,., ".., range over all possible indices of each hidden unit of each layer of an MFN, and 82, "1, +1) range over all 2L-1 possible binary signs. In other words, Fathony et al. (2021) demonstrate that the representation of an MFN at a particular layer contains an exponential combination of all possible positive and negative combinations of the frequencies of the sine terms in each hidden unit at each layer in the MFN up to the current layer.
The original analysis uses these terms to argue that MFNs model exponentially many terms through a linear amount of layers. For our purpose of computing the frequency response of the MAGNet generated kernel, can plug our derivation of the frequencies of the Gabor filter fGabor into W to compute the frequency spectrum of the generated kernel:
SMAGNa {
(LL) Gabor.i.
-8.--11((11)
(1-1) Gabor +... 82. Gabor. Gaboor.
(2)
(1)
(18)
17

---

Published as a conference paper at ICLR 2022
[(z) = M(r 0.0.
h(z) s5n(30.3r)
g(a) (fr 0.1 (500..33r)
F(((a))
F(h(x))
F(g(a))
Figure 9: Decomposition of Gabor filter and its frequency spectrum. Top row: decomposition of Gabor filter (right) into its Gaussian term (left) and its sine term (center). Bottom row: frequency responses for each respective filter. The Fourier transform of Gaussian envelope is Gaussian envelope (blue circles show oF for h = {1.2)). The Fourier transform of a sine pattern is collection of symmetrical impulse signals (red box shows the Nyquist frequency). The Gaussian envelope blurs the frequency response of the sine term (purple boxes show the frequency response for h (1,2,3)).
As stated before, we are only interested in the maximum frequency in the frequency spectrum. We can therefore simplify Eq. 18 in two ways. First, we simplify over MAGNet layers by taking the maximum value of the spectrum, which is the sum over all layers using only the positive binary signs sL (Eq. 19). Next, we simplify over channel indices by retaining only the channel index that results in the highest frequency (Eq. 20). The maximum frequency of a MAGNet is shown in Eq. 21:
fMAGNE: (+1))t(() (L)
Gabor.i. (+1)s (L-1) Gabor,iL
(+1) (2)
Gabur, Gaborr,
(1)
(19)
[eabbor.n & + + Gabbori Gabor,
st (I.
(1)
MMAGNN
max (L) (SCaborsee) in
max ( (L-1)
(Gabori .... max
Gabor
(2)
max
(
Gabor...
(20)
SL
i2
81
Σ max s Gahor..
1
it
((I
Cun min (x.., 7Y
(
Σ
max
max
g.te.j
(21)
2x
2T
1
Effect of the Gaussian mask in the frequency components of FlexConv. FlexConvs attenuate the MAGNet output with Gaussian mask. The Gaussian mask (Eq. 1) works analogously to the Gaussian envelope term in the Gabor filter: blurs the frequency components of the generated kernel reuse derivation for the Gaussian envelope of the
with standard deviation dp. Therefore,
18

---

Published as a conference paper at ICLR 2022
Gabor filter (Eq. 15).
The maximum frequency component of a FlexConv kernel is given by:
FllesCome
|MAGNNe + Senv
Ocut min(ox' fox ,
Jeut
.MAGNN
MAGNet + max (xx JT
2π
W
Oent min min(77x,i (7x,4 7Y
Σ max E=1
max
out
(22)
21
2n
2m
max 0y
oxx
Visualization of regularized kernels. Fig. 8 shows example kernels from FlexNets trained with aliasing regularization. The frequency domain plots confirm the accuracy of our frequency component regularization.
A.2 REGULARIZING THE FREQUENCY RESPONSE OF FLEXCONV
Nyquist frequency of FlexConv kernel. Given the sampling rate fa of the kernel, we can compute its Nyquist frequency fNyg as:
fNy4 f
(23)
To compute the sampling rate, note that the kernel coordinates input to our MAGNet stretch over [-1,1]] domain. For a kernel of length k. we therefore sample one point in every fs
A-1 units.
Knowing the sampling rate in terms of the kernel size allows us to express the Nyquist frequency in terms of the (pre-masked) kernel size:
1k k-1
fNyg (k)
(24)
2
4
Note that the kernel size in a FlexConv is initialized to be equal to the resolution of the data, if odd. For even resolutions, it corresponds to the resolution of the data plus one.
Constructing the regularization term, We train FlexConv with a regularization term on the fre- quency response of the generated kernel to ensure that aliasing effects do not distort the performance of the model when it inferred higher resolution. This section details the implementation of the regularization function.
From the parameters of each FlexConv module, we compute fixexCom according to Eq. 22. For amount of standard deviations use in determining feav (Eq. 15) we use h h= 2. From the kernel size
of the FlexConv module we compute f according to Eq. 24. We then apply an L2 regularizer over the amount that FFiexConv exceeds fNyq(k):
Lur = (J JNyy (k)) INyy (k)11
(25)
We weight CHF by == 0.1 when adding it to our loss function.
Improved implementation. Eq. 25 contains a sum over the layers of the MAGNet. In practice, we prefer to regularize each layer E L separately, so that the gradients of the regularization of different layers are not dependent on each other. We therefore implement the anti-aliasing regularization by regularizing each MAGNet layer independently, and spreading the fön term from the gaussian mask uniformly over all MAGNet layers:
feuv
[Nyq(A:)
(N)g(k)
LHR.
|mxx
IMAGNEL
(26)
L
fenw
(Ny
JNNy
|max
max
Gabor,
(27)
L
L
L
In the code, we refer to this method as the together method, versus the summed method of Eq. 25. In preliminary experiments, observed improved performance of anti anti-aliasing training when using the together method. All of our experiments anti-aliasing experiments therefore use the together setting.
19

---

Published as a conference paper at ICLR 2022
B DATASET DESCRIPTION
B.I IMAGE FITTING DATASETS
Kodak dataset. The Kodak dataset (Kodak, 1991) consists of 24 natural images of size 768 x 512. This dataset popular benchmark used for compression and image fitting methods.
B.2 SEQUENTIAL DATASETS
Sequential and Permuted MNIST. The sequential MNIST dataset (sMNIST) (Le et al., ttakes the 28x28 images from the original MNIST dataset (LeCun et al., 1998), and presents them a sequence of 784 pixels. The goal of this task is to perform digit classification given the representation of the last sequence element of a sequential model. Consequently, good predictions require the model to preserve long-term dependencies up to 784 steps in the past.
The permuted MNIST dataset (pMNIST) additionally changes the order of all the sMNIST sequences by random permutation. Consequently, models can no longer rely on local features to construct good feature representations. As a result, the classification problem becomes more difficult, and the importance of long-term dependencies more pronounced.
Sequential and Noise-Padded CIFAR10. The sequential CIFAR10 dataset (sCIFAR10) (Chang al., 2017) takes the 32x32 images from the original CIFARIO dataset (Krizhevsky et al., 2009) and presents them as a sequence of 1,024 pixels. The goal of this task to perform image classification given the representation of the last sequence element of sequential model. This task is more difficult than sMNIST, as a larger memory horizon is required to solve the task and more complex structures and intra-class variations present in the data (Bai et al., 2018b).
The noise-padded CIFAR10 dataset (npCIFARI 10) (Chang et al., 2019) flattens the images from the original CIFAR10 dataset (Krizhevsky et al., 2009) along their rows to create a sequence of length 32, and 96 channels (32 rows x channels). Next, these sequences are concatenated with 968 entries of noise to form the final sequences of length 1000. As for SCIFARIO, the goal of the task is to perform image classification given the representation of the last sequence element of sequential model.
Character CharacterTrajectoriiss The CharacterTrajectories dataset part of the UEA time series classifi- cation archive (Bagnall et al., 2018). It consists of 2858 time series of length 182 and 3 channels representing the x, / positions, and the tip force pen while writing Latin alphabet characters in single stroke. The goal is to classify out of 20 classes the written character using the time series data.
Speech Commands. The Speech Commands dataset (Warden, 2018) consists of 105,809 one- second audio recordings of 35 spoken words sampled at 16kHz. Following Kidger et al. (2020), we extract 34975 recordings from ten spoken words to construct a balanced classification problem. We refer to this dataset as SpmechCommands raw, or SC_rat for short. Furhtermore, we utilize the preprocessing steps of Kidger et al. (2020) and extract mel-frequency cepstrum coefficients from the raw data. The resulting dataset, abreviated SC, consists of time series of length 101, and 20 channels.
B.3 IMAGE BENCHMARK DATASETS
MNIST. The MNIST hadwritten digits datset (LeCun & Cortes, 2010) consists of 70,000 gray- scale handwritten digits of size 28x28, divided into training and test sets of 60,000 and 10,000 images, respectively. The goal of the task is to classify these digits as one of the ten possible digits 10,1
CIFAR-10 The CIFAR-10 dataset (Krizhevsky et al., 2009) consists of 60,000 natural images from 10 classes of size 32x32, divided into training and test sets of 50,000 and 10,000 images, respectively.
STL-10. The STL- 10 dataset (Coates et al., 201 1) is a subset of the ImageNet dataset (Krizhevsky al., 2012) consisting of 5,000 natural images from 10 classes of size 96x96, divided into trainint and test sets of 4,500 and 500 images, respectively.
ImageNet-k. The Imagenet-k (Chrabaszeze et al., 2017) dataset is derived from the ImageNet dataset Russakovsky et al. (2015) by downsampling all samples to a resolution k e [64, 32, 16, 8]. The dataset contains 1000 classes with 1,281,167 training samples and 50,000 validation samples.
20

---

Published as a conference paper at ICLR 2022
Table Average PSNR for fitting of images in the Kodak dataset. Both our improved initialization scheme, as well as the inclusion of anisotropic Gabor functions lead to better reconstructions.
IMPROVED INIT
MODEL
PARAMS
PSNR
SIREN
7.14K
25.665
MINNeuurtrr
23,276
25.361 X
MFNGave
7.1KK
25.606
25.791
MAGNet
7.36K
25,893
Table 6: Full results on CIFAR-10. We report results over three runs per setting. CIFARResNet-44 w/ CKConv a CIFARResNet-44 where all convolutional layers are replaced with CKConvs with CiFARResNet-44 w/ FlexConv is CIFARResNet-44 where all convolutional layers are replaced with FlexConv with learned kernel size, except for the shortcut connections of the strided convolutional layers, which are pointwise convolutions. *Results taken from the respective original works instead of reproduced. RResults are from single run.
CIFAR
MODEI
SIZE
ACC:
DCN-s (Tomen 2021)
0.47M
89.7 0,3*
N-Ja-CIFARRRaNNe222 (Pintea 2021)
0.52M
92.3
N-Jer Aite-ALLINN (Pintea al., 2021)
1.07M
92.5 0.1*
CIFARResNet:44 (He et al., 2016)
0.66M
92,9*1
CIFARResNet et 2016) reproduction)
0.66M
90,9 = 0.2
CIFARResNer-44 w/ CKConv (k CIFARResNa-44 FlexConv
2.58M
86.1 = 0.9
2.58M
FlexNei-7 w/ cony. (R =3)
0.17M
89.5 = 0.3
HeexNet-7 coav. (k 33)
FlexNet- 7 w/ Jei (Pintea et al., 2021)
0.70M
91.7 0.1
CKCNNemENN3 CKCNN
0.26M
72.4*
0.27M
83.8"
3KCNN
0,28M
85.6*
CKCNNaxiar3
0.28M
CKCNN-7
0.63M
71.7*
CKCNNRaake
0.63M
84.6"
CKCNN 7
0.67M
87.7*
CKCNNMAORa
FlesNetsmex-7
0.63M
88.9*
0.66M
91.6*
FlexNesuase77
0.67M
92.0*
FlexNet-3
0.27M
90.4 a 0.2
FlexNet-5
0.44M
91.0 0.5
FlexNet-7
0.67M
92.2 =
C ADDITIONAL EXPERIMENTS
C.1 IMAGE CLASSIFICATION
CIFAR-10. Tab. shows all results for CIFAR-10 experiments, including ablations.
ImageNet-322 Results for the ImageNet-32 experiment are shown in Table 7. FlexNets are slightly worse than CeF2RResNet-32 (He et al., 2016) with slightly less parameters. However, the results
Table 7: Results ImageNet-32. *Results taken from the respective original works instead of reproduced. *Results are from a single run.
MODEL
SIzB
IMAGENET-32
ToP-1
Top-5
CIFARResNes 32 0.53M
26.41 + 0.13
49.37 + 0.15
WRN-28-1
0.44M
32.03**
57.51*4
FlexNet-5
0.44M
24.9 0.4
47.7 0.6
21

---

Published as a conference paper at ICLR 2022
Table 8: Results for alias-free FlexNets on CIFAR-I0 and ImageNet-k. A denotes difference in accuracy.
IMAGENET-K Top-1
MODEL
S1ZE
k= 16
Åt-18 & 32
CIFARResNet-32
0.52M
16.1 a 0.0
-11.6 0.4
FlexNet-5 w/ N-Jets
0.46M
15.7 a: 0.1
-1.9 a 0.4
FlexNet-5
0.44M
14.9 a 0.1
-1.9 a 1.7
Table 9: Results on MNIST. We train each model with three different seeds and report mean and standard deviation. *Results are taken from the respective original works instead of reproduced. *Results are from single run.
MNIST ACC.
MODEL
SIZE
Efficient- Napseee (Mazzia al., 2021)
99,8*;
Network in Network (Lin al.. 2013)
NIA
99.6*
VGG-5 tresults from Kabir et al. (2020))
3.65M
99.7*
FlexNet-16 16
0.67M
99.7 = 0.0
reported by Chrabaszez et al. (2017) for Wide ResNets (Zagoruyko & Komodakis, 2016) outperform FlexNets by significant margin.
Alias-free ImageNet-32. We report results for alias-free FlexNets on ImageNet-k (Chrabaszez et al., 2017) in Table 8, to verify the results of alias-free training at a larger scale. We find that FlexConv and N-Jet both mostly retain classification accuracy between source and target resolution, while CIFARResNet-32 degrades drastically.
MNIST and STL-10. We additionally report results on MNIST (Tab. 9) and STL-10 (Tab. 10. We choose these dataset for the difference in image sizes of the training data. On MNIST, though performance MNIST is quite saturated, STL-10 we are significantly worse than the baseline CIFARResNet from (Luo et al., 2020), though with significantly less parameters. We not able prepare more relevant baseline for this experiment.
competitive with state of the methods. On
D EXPERIMENTAL DETAILS
D.1 FLEXNET
We propose image classification architecture named FlexNer (Fig. 10), consisting of stack of FlexConv blocks followed by a global average pooling layer and a linear layer. FlexNets are named "FlexNet-L" where L indicates the amount of layers in the architecture.
FlexBlock. Each FlexBlock consists of two FlexConvs with BatchNorm (loffe & Szegedy, 2015) and dropout (Srivastava al., 2014) (d 0.2) as well residual connection. The width of block i is determined by scaling a base amount by progressively increasing factors: [c, c 1.5, c 1.5, x 2.0, ex (1). The default configuration of FlexNet uses 22. In FlexNet-N-Jet models, we scale to match the amount of parameters of the Feex in the comparison.
Table 10: Results on STL-10. We train each model with three different seeds and report mean and standard deviation. *Results are taken from Luo et al. (2020). *Results are from single run.
STL-10
MODEL
ACC.
CIFARReeNet-18
11.2M
81.0*
11exNet-16
0.67M
22

---

Published as a conference paper at ICLR 2022
input.length
Input
CKConv
LayerNorm
GetRelPositions
ReLU
RelPositions
LL
CKBlock
DropOut
Kernelleet
Linear
CKConv
ConvKernel
LayerNorm
ReLU
GaussMask
Output
DropOut
MaskedKernel
FFTConv
ReLU
Figure 10: FlexNet architecture. FlexNet-T. consists of L FlexBlocks, where each FlexBlock is residual block of FlexConvs.
FlexConv initialization. We initialize the FlexConv mask variances small, at x o = 0.125. For initializing MAGNet, we initialize the Gaussian envelopes as discussed in Sec. 3.2. We initialize the linear layer weights by the same Gamma distribution as used for the enveloped, modulated by scaling factor of 25.6. We found that this value of the scaling factor, rather than a higher one, helped in reducing the performance of alias-free models. initialize the bias of the linear layers by ((-
CIFAR-10. In FlexNet- 16 models for CIFAR-10 we use c 24 to approximate the parameter count of Cesss in the experiment.
D.2 OPTIMIZATION
We use Adam (Kingma & Ba, 2014) to optimize FlexNet. Unless otherwise specified, we use learning rate of 0.01 with cosine annealing scheme (Loshchilov & Hutter, 2016) with five warmup epochs. We use a different learning rate of 0.1x x the regular learning rate for the FlexConv Gaussian mask parameters. We not use weight decay, unless otherwise specified.
Kodak. We overfit on each image of the dataset for 20,000 iterations. To this end, we use learning rate 0.01 without any learning rate scheme. We observe that SIRENs diverge with this learning rate and thus, reduce the learning rate to 0.001 for these models.
C1. train for 350 epochs with a batch size 64. We use the data augmentation from He al. (2016) when training CIFAR-10: a four pixel padding, followed by random 32 pixel crop and a random horizontal flip.
ImageNet-32 We train for 350 epochs with batch size of 2048. We the same data augmentation as used for CIFAR-10, We do use weight decay of 1e-5 for ImageNet-32 training.
Sequential and Permuted MNIST. We train for 200 epochs with batch size of 64 and learning rate 0.01. We use a weight decay 1c-5.
Sequential and Noise-Padded CIFAR-10. For sequential CIFAR-10, 10, train for 200 epochs with a batch size 64, a learning rate 0.001 and a weight decay of 1e-5. For noise caise-padded CIFAR-10 10, train for 300 epochs with batch size of 32, learning rate of 0.01 and weight decay.
23

---

Published as a conference paper at ICLR 2022
Speech Commands and CharTrajectorres. We train for 300 epochs with batch size of 32 and learning rate of 0.001. For CharTrajectories, we use a weight decay 1e-5.
D.3 ROTATED GAUSSIAN MASKS
MAGNets anisotropic Gaussian terms in the Gabor filters, which yields improvements in descrip- tive power and convergence speed (Sec. 3.2). For the same reason, we explore making the anisotropic FlexConv Gaussian mask steerable. by including an additional vector of learnable angle parameters ( R that rotates the Gaussian masks. Although preliminary experiments show rotated masks lead to slight additional improvements, the computational overhead required to rotate the masks is large. Consequently, we do consider rotated Gaussian masks in our final experiments.
24


Reviews:
[{"review_id": "xwHno887QrU", "reviewer": "Reviewer_hUMj", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros  \nThe FlexConv is proposed to produce learn kernels with various kernel sizes.\nAn effective regularization is proposed to address the aliasing issue and generalization when dealing with higher resolution at the inference stage.\n\nCons\n1. Incremental novelty compared to Multiplicative Filter Networks (Fathony et al., 2021). Anisotropic gabor functions are used to replace the original isotropic gabor functions in MFNs.\nIn addition, there is no explanation on why anisotropic gabor functions are used here and why it is better.\n2. The idea of using gaussian mask to control the effective receptive field has also been proposed in Chapter 3.3 in [a]. The author even discussed different parametric kernel functions in that thesis.\n[a] Duc Tam Nguyen, Robust Deep Learning for Computer Vision to counteract Data Scarcity and Label Noise, Doctoral Thesis, 2020.\n3. There is no explanation on how the diversity in the initialization of $\\gamma^{(l)}_X$ and $\\gamma^{(l)}_Y$ relates to the accuracy.\n4. There is no explanation on how equation 9 is derived.\n5. Why not put relative change instead of absolute number in the left table of Figure 5, but use the absolute number in the right table?\n6. Lack comparison with other methods on image benchmark datasets. Why only compare with ResNet44 instead of more popular ones such as ResNet50, ResNet 18, ResNet 101?\n7. Lack comparison between a ResNet and the one whose all normal convolutional layers are replaced by FlexConv Layers.\n8. The result of figure 6 is a little different from intuition and classic convolutional neural networks such as AlexNet and ResNet. Any explanation on this?\n\n--------------------------------------------------------------  \n\nThe explanation of authors has clarified some of my concerns but there are still some issues remained. Therefore I only raise my ratings to '6: marginally above the acceptance threshold'.", "summary_of_the_paper": "This paper presents a novel convolutional operation named FlexConv, to produce high bandwidth convolutional kernels with learnable kernel size at a fixed parameter cost. It is able to generate kernels with large kernel size and model long-term dependencies among elements in a sequence or an image. State-of-the-art performance on both sequential datasets and image datasets demonstrates the effectiveness of the proposed method. In addition, a novel kernel parameterization method is proposed to control the frequency of the generated kernels and avoid aliasing, and hence can well generalize to higher resolution cases which are never seen during training.", "summary_of_the_review": "The novelty of the paper is incremental and further explanation and experiment are required to demonstrate its effectiveness.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "STKmHzgqaex", "reviewer": "Reviewer_ub8r", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths \n1. Nice ideas and well-argumented design\n2. A mechanism to control aliasing\n3. Results above those of existing works\n\n__Idea.__ The approach is based on the idea of allowing the learning of filters of different size within a convolutional network, and it is supported by literature and empirical studies. The use of continuous convolutions via implicit representations is becoming of interest for the community. This work has thus relevance and places itself well within the current developments in representation learning.\n\n__Anti-aliasing.__ The observation that other implicit representations (e.g. SIREN) suffer from aliasing when kernels learned for a given resolution are applied to higher resolution data stimulated the authors to investigate the possibility of learning an anti-aliased implicit representation network. The use of anisotropic Gabor functions as kernels for the implicit representation networks (which can be considered, in my opinion, the true contribution of this work) allows for controlling the maximum learnable frequency.\n\n__Results.__ The results obtained are higher than those reported by other existing works. This places this paper as state-of-the-art for learnable kernel networks.\n\n\n### Weaknesses\n1. Contributions are not made crystal clear. Novelty seems oversold.\n2. The work seems a marginal improvement on CKConv, with the difference that an anisotropic Gabor kernel is used to modulate the implicit network representations.\n3. The computational impact of the learnable kernels is not discussed enough. Application to 'normal-size' images is \n4. Some comparison statements are questionable.\n\n__Contributions.__ In my opinion, the authors make overstatements about their contributions. FlexConvs seems like a re-branding of CKConvs, while FlexNets are networks that deploy FlexConvs but are presented as a contribution themselves. The true contribution of this paper, and the main part where novelty is presented, can be considered the use of anisotropic Gabor functions as kernels of the implicit representation networks. The Gabor functions indeed allow to control the extension of the Gaussian masks that determine the size of the kernels, and also constrain the maximum learnable frequency for the kernels in order to avoid aliasing in the networks. \n\n__CKConv.__ This work seems to build a lot on top of CKConvs (also under review @ICRL22 - here results of CKConv are reported, not likewise for the CKConv paper), but fails to highlight what the real contribution is (see point above). \n\n__Computational impact.__ The computational impact of using MAGNets is only briefly addressed in this paper. An obvious observation is that bigger kernels require more computations because of the increased number of convolutional weights. This also relates to the increased size of the images in the case a network is applied to higher-resolution images (the case of a network learned on CIFAR 16x16 and then upscaled to CIFAR 32x32). In this work the maximum threated resolution is indeed 32x32, also in the case of ImageNet, which is rescaled. An actual discussion and experiments about the learning or application (by upscaling) of the proposed approach to REAL higher-resolution images (e.g. the original ImageNet) are missing. This limits, in my opinion, the transferrability of this work to real-case uses. \n\n__Questionable statements.__ Some statements about results are questionable or not precise. For instance, the authors claim that the proposed FlexNet achieves comparable results to much deeper ResNets. Actually, the ResNet and the FlexNet concerned by this statement have roughly the same number of parameters, which may influence more directly the learning capabilities of the networks (and thus the results). The ResNet has  blocks (although more convolutional layers), while the FlexNet has 7 FlexConv Blocks. Should it then be considered deeper than the ResNet? I think that the authors should not bring the depth as an argument at their advantage, but rather focus on comparing the networks in terms of the learnable parameters.\n\n\nOn a general note, the paper introduces nice ideas, but it should be better written to highlight the contributions and what is taken from the state-of-the-art.\n\n=========================================================================\nAfter reading the response to reviews and the revision of the paper, I had my concerns clarified. I identify the novelty and amount of contributions provided by the authors in this paper, and recognize their value for further development of CNNs. Although relation with training at low resolution and testing at higher resolution is 'only' drafted in this paper, one can deal with the fact that this paper is a good seed for further experiments in future works. Thus I am inclined to raise my score.\n\n", "summary_of_the_paper": "The paper proposes a method for learning convolutional filters with trainable size, that builds on top of multiplicative filter networks. The learnable convolutions are called FlexConv, and a network that deploys them is called FlexNet. In order to control the aliasing eventually introduced when learning the convolutional filters, anisotropic Gabor kernels are used within the multiplicative filter networks.", "summary_of_the_review": "The paper introduces some new nice idea, but it seems overstating some of the contributions. The experimental analysis is extensive but on very small data set of low-resolution, while actual problems encountered to scale this method up to more 'real-case' resolutions are not discussed appropriately.  ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Ij-AXiBWLNu", "reviewer": "Reviewer_T41g", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nPros.\n1.\tThis paper introduced a new parameterization method of CNN kernels by multiplying continuous kernels and with sigma-learnable Gaussian masks. Especially, multiplication with learnable Gaussian is first demonstrated.\n2.\tFlexconv operation is robust to fit high frequency signals in large kernels by MagNets.\n3.\tAccording to Table 4, the proposed network shows good performance on higher resolution images without additional training.\n4.\tThis work achieves state-of-the-art performance on several sequential datasets. \n\nCons.\n1.\t“Large kernels” and “small kernels” are too obscure.\n2.\tAlthough the meaning of “bandwidth” seems to be important, the descriptions are not enough to understand. It confuses the original meaning of bandwidth; originally, bandwidth in signal processing is a term for finite support in frequency domain.\n3.\tIn table 4, the two scales of 16 pixels and 32 pixels are inefficient to explain the performance of FlexConv. This is exaggerated without further scale experiments.\n4.\tThe authors claimed that the cropping operation in Flexconv improves computational efficiency. But there is no ablation study about the cropping operations, reviewer has no clues about the relationship between cropping size and computational efficiency.\n\nMinor.\n\nIn page 20, there are type such as ‘??’.\n", "summary_of_the_paper": "This paper proposed a new convolutional operation with varying-sized kernels. Flexconv constructs their convolutional kernels with the product between the CKConv and Gaussian masks so that the size of the kernels can be learned during training. Also, in the network of Flexconv, MAGNets allow the Flexconv to be used at higher resolutions without aliasing which are unseen resolution. The experiment on several sequential datasets showed that the classification performances from both TCN and ResNet based on FlexConv outperform that of state-of-the-art networks.", "summary_of_the_review": "The way of representing and analyzing the results is a bit overclaimed without serious supporting experiments. However, their method demonstrated state-of-the-art performances compared with recent approaches. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-MfOsaDFYhJ", "reviewer": "Reviewer_APZa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\n(1) Section 4.1 is not very clear, how the target image is fitted by different kernels? What is an AlexNet kernel, 3x3, 7x7? Please give more descriptions of the experiment.\n\n(2) In Table 3, it would be better to show the time consumption of ResNet-44 or its reproduction since FlexNet has additional operations that will reduce the inference speed, which is much more crucial than the model size.\n\n(3)  This paper and its related work focus on kernel sizes, dilations, etc., which may obtain greater improvement on high-resolution images. However, it is not clear whether FlexNet performs well on high-resolution image tasks, such as ImageNet or MS-coco in the cited work [1].\nThe authors are expected to demonstrate the superiority of FlexNet on typical and practical tasks (classification, generalization, detection), which would draw more attention from the CNN community.\n\n(4) Why \"alias-free\" is important? The authors claim that FlexNets can be deployed at higher resolutions than those seen during training. But the experiments only show the improvement on accuracy. It would be more clear if authors can show the superiority on other aspects, for example, vanilla nets are trained on high resolutions while FlexNet are trained on lower resolutions, they have comparable classification accuracy but the latter are more efficient in terms of time/computation cost.\n\n[1] Dai, Jifeng, et al. \"Deformable convolutional networks.\" Proceedings of the IEEE international conference on computer vision. 2017.\n\n==========================================================================\n\nI have read the authors' rebuttal, some of the concerns have been addressed, others still remain.\nThe authors show that the reproduction of the ResNet-44 is 22s per epoch during training, which is much faster than FlexNet series with similar # parameters and accuracy, which almost can be extended to the inference stage.\nIn the reply to point 4, the author also notes the computational efficiency of alias-free models. \nHowever, a direct comparison between FlexNets and classical models (ResNet, DenseNet, etc) with regard to the accuracy, training time, model size, inference time, etc, is still lacking.\nAside from ablation study in the settings of FlexNet models, the authors are encouraged to provide more evidence such that the community is willing to discard common practices and adopt your methods on some occasions.\nIn summary, the reviewer will keep the score.\n\n", "summary_of_the_paper": "The authors propose FlexConv, a novel convolutional operation with which high bandwidth convolutional kernels of learnable kernel\nsize can be learned at a fixed parameter cost. The FlexNet obtains state-of-the-art across several sequential datasets, and matchs recent works with learnable kernel size on CIFAR-10 with less compute.", "summary_of_the_review": "This is a very interesting study, the method and theoretical analysis seem sold, but the motivation and comparative experiments should be improved.\n\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This submission proposes a method for learning convolutional filters with trainable size, that builds on top of multiplicative filter networks.  Anti-aliasing is achieved by parametrization with anisotropic Gabor filters.  The reviewers were unanimous in their opinion that the paper is suitable for acceptance to ICLR.  The authors are encouraged to make use of the extensive reviewer discussion in improving the final version of the paper.
Decision: Accept (Poster)