FLEXCONV: CONTINUOUS KERNEL CONVOLUTIONS
Year: 2021

Abstract:
When designing Convolutional Neural Networks (CNNs), one must select the size of the convolutional kernels before training. Recent works show CNNs benefit from different kernel sizes at different layers, but exploring all possible combinations is unfeasible in practice. A more efficient approach is to learn the kernel size during training. However, existing works that learn the kernel size have a limited bandwidth. These approaches scale kernels by dilation, and thus the detail they can describe is limited. In this work, we propose FlexConv, a novel convolutional operation with which high bandwidth convolutional kernels of learnable kernel size can be learned at a fixed parameter cost. FlexNets model long-term dependencies without the use of pooling, achieve state-of-the-art performance on several sequential datasets, outperform recent works with learned kernel sizes, and are competitive with much deeper ResNets on image benchmark datasets. Additionally, FlexNets can be deployed at higher resolutions than those seen during training. To avoid aliasing, we propose a novel kernel parameterization with which the frequency of the kernels can be analytically controlled. Our novel kernel parameterization shows higher descriptive power and faster convergence speed than existing parameterizations. This leads to important improvements in classification accuracy.

heading: 1 INTRODUCTION
text: The kernel size of a convolutional layer defines the region from which features are computed, and is a crucial choice in their design. Commonly, small kernels (up to 7px) are used almost exclusively and are combined with pooling to model long term dependencies (Simonyan & Zisserman, 2014; Szegedy et al., 2015; He et al., 2016; Tan & Le, 2019). Recent works indicate, however, that CNNs benefit from using convolutional kernels (i) of varying size at different layers (Pintea et al., 2021; Tomen et al., 2021), and (ii) at the same resolution of the data (Peng et al., 2017; Cordonnier et al., 2019; Romero et al., 2021). Unfortunately, most CNNs represent convolutional kernels as tensors of discrete weights and their size must be fixed prior to training. This makes exploring different kernel sizes at different layers difficult and time-consuming due to (i) the large search space, and (ii) the large number of weights required to construct large kernels.
A more efficient way to tune different kernel sizes at different layers is to learn them during training. Existing methods define a discrete weighted set of basis functions, e.g., shifted Delta-Diracs (Fig. 2b, Dai et al. (2017)) or Gaussian functions (Fig. 2c, Jacobsen et al. (2016); Shelhamer et al. (2019); Pintea et al. (2021)). During training they learn dilation factors over the basis functions to increase the kernel size, which crucially limits the bandwidth of the resulting kernels.
In this work, we present the Flexible Size Continuous Kernel Convolution (FlexConv), a convolutional layer able to learn high bandwidth convolutional kernels of varying size during training (Fig. 1). Instead of using discrete weights, we provide a continuous parameterization of convolutional kernels via a small neural network (Romero et al., 2021). This parameterization allows us to model continuous functions of arbitrary size with a fixed number of parameters. By multiplying the response of the neural network with a Gaussian mask, the size of the kernel can be learned during training (Fig. 2a). This allows us to produce detailed kernels of small sizes (Fig. 3), and tune kernel sizes efficiently.
FlexConvs can be deployed at higher resolutions than those observed during training, simply by using a more densely sampled grid of kernel indices. However, the high bandwidth of the kernel can lead FlexConv to learn kernels that show aliasing at higher resolutions, if the kernel bandwidth exceeds
the Nyquist frequency. To solve this problem, we propose to parameterize convolutional kernels as Multiplicative Anisotropic Gabor Networks (MAGNets). MAGNets are a new class of Multiplicative Filter Networks (Fathony et al., 2021) that allows us to analyze and control the frequency spectrum of the generated kernels. We use this analysis to regularize FlexConv against aliasing. With this regularization, FlexConvs can be directly deployed at higher resolutions with minimal accuracy loss. Furthermore, MAGNets provide higher descriptive power and faster convergence speed than existing continuous kernel parameterizations (Schütt et al., 2017; Finzi et al., 2020; Romero et al., 2021). This leads to important improvements in classification accuracy (Sec. 4).
Our experiments show that CNNs with FlexConvs, coined FlexNets, achieve state-of-the-art across several sequential datasets, match performance of recent works with learnable kernel sizes with less compute, and are competitive with much deeper ResNets (He et al., 2016) when applied on image benchmark datasets. Thanks to the ability of FlexConvs to generalize across resolutions, FlexNets can be efficiently trained at low-resolution to save compute, e.g., 16 × 16 CIFAR images, and be deployed on the original data resolution with marginal accuracy loss, e.g., 32 × 32 CIFAR images. In summary, our contributions are:
• We introduce the Flexible Size Continuous Kernel Convolution (FlexConv), a convolution operation able to learn high bandwidth convolutional kernels of varying size end-to-end.
• Our proposed Multiplicative Anisotropic Gabor Networks (MAGNets) allow for analytic control of the properties of the generated kernels. This property allows us to construct analytic alias-free convolutional kernels that generalize to higher resolutions, and to train FlexNets at low resolution and deploy them at higher resolutions. Moreover, MAGNets show higher descriptive power and faster convergence speed than existing kernel parameterizations.
• CNN architectures with FlexConvs (FlexNets) obtain state-of-the-art across several sequential datasets, and match recent works with learnable kernel size on CIFAR-10 with less compute.

heading: 2 RELATED WORK
text: Adaptive kernel sizes. Adaptive kernel sizes have been proposed via learnable pixel-wise offsets (Dai et al., 2017), learnable padding operations (Han et al., 2018), learnable dilated Gaussian functions (Shelhamer et al., 2019; Xiong et al., 2020; Tabernik et al., 2020; Nguyen, 2020) and scalable Gaussian
derivative filters (Pintea et al., 2021; Tomen et al., 2021; Lindeberg, 2021). These approaches either dilate discrete kernels (Fig. 2b), or use discrete weights on dilated basis functions (Fig. 2c). The ability of these methods to learn the kernel size depends on dilation, which crucially limits the bandwidth of the resulting kernels. In contrast, FlexConvs are able to construct high bandwidth convolutional kernels of varying size, and with a fixed parameter count. Larger kernels are obtained simply by passing a larger vector of positions to the neural network parameterizing the kernel (Fig. 1).
Continuous kernel convolutions. Discrete convolutional kernel parameterizations assign an independent weight to each specific position in the kernel. Continuous convolutional kernels, on the other hand, view convolutional kernels as continuous functions parameterized via a small neural network MLPψ ∶ RD → RNout×Nin , with D the data dimensionality. This defines a convolutional kernel for which arbitrary input positions can be queried. Continuous kernels have primarily been used to handle irregularly-sampled data locally, e.g., molecular data (Simonovsky & Komodakis, 2017; Schütt et al., 2017) and point-clouds (Thomas et al., 2018; Wang et al., 2018; Shi et al., 2019).
Recently, Romero et al. (2021) introduced the Continuous Kernel Convolution (CKConv) as a tool to model long-term dependencies. CKConv uses a continuous kernel parameterization to construct convolutional kernels as big as the input signal with a constant parameter cost. Contrarily, FlexConvs jointly learn the convolutional kernel as well as its size. This leads to important advantages in terms of expressivity (Fig. 3), convergence speed and compute costs of the operation.
Implicit neural representations. Parameterizing a convolutional kernel via a neural network can be seen as learning an implicit neural representation of the underlying convolutional kernel (Romero et al., 2021). Implicit neural representations construct continuous data representations by encoding data in the weights of a neural network (Park et al., 2019; Sitzmann et al., 2020; Fathony et al., 2021).
We replace the SIREN (Sitzmann et al., 2020) kernel parameterization used in Romero et al. (2021) by our Multiplicative Anisotropic Gabor Networks: a new class of Multiplicative Filter Networks (Fathony et al., 2021). MFNs allow for analytic control of the resulting representations, and allow us to construct analytic alias-free convolutional kernels. The higher expressivity and convergence speed of MAGNets lead to accuracy improvements in CNNs using them as kernel parameterization.

heading: 3 METHOD
text: In this section, we introduce our approach. First, we introduce FlexConv and the Gaussian mask. Next, we introduce our Multiplicative Anisotropic Gabor Networks (MAGNets) and provide a description of our regularization technique used to control the spectral components of the generated kernel.

heading: 3.1 FLEXIBLE SIZE CONTINUOUS KERNEL CONVOLUTION (FLEXCONV)
text: To learn the kernel size during training, FlexConvs define their convolutional kernels ψ as the product of the output of a neural network MLPψ with a Gaussian mask of local support. The neural network MLPψ parameterizes the kernel, and the Gaussian mask parameterizes its size (Fig. 1).
Anisotropic Gaussian mask. Let G(x;µX, σ2X)∶= exp{− 12σ −2 X (x − µX)2} be a Gaussian function parameterized by a mean-variance tuple (µX, σ2X). The anisotropic Gaussian mask is defined as:
wgauss(x, y;{µX, σ2X, µY, σ2Y}) = G(x;µX, σ2X)G(y;µY, σ2Y). (1) By learning (µX, σ2X) and (µY, σ2Y) independently, anisotropic non-centered windows can be learned.

heading: 3.2 MULTIPLICATIVE ANISOTROPIC GABOR NETWORKS (MAGNETS)
text: In this section, we formalize our proposed parameterization for the kernel MLPψ. We start by introducing Multiplicative Filter Networks (Fathony et al., 2021), and present our MAGNets next.
Multiplicative Filter Networks (MFNs). Recently, Fathony et al. (2021) proposed to construct implicit neural representations as the linear combination of exponentially many basis functions g:
h(1) = g([x, y];θ(1)) g ∶ R2 → RNhid (2)
h(l) = (W(l)h(l−1) + b(l)) ⋅ g([x, y];θ(l)) W(l) ∈ RNhid×Nhid ,b(l) ∈ RNhid (3)
ψ(x, y) =W(L)h(L−1) + b(L) W(L) ∈ RN×Nhid ,b(L) ∈ RN (4)
where {θ(l),W(l), b(l)} depict the learnable parameters of the bases and the affine transformations, and N,Nhid depict the number of output and hidden channels, respectively. Depending on the selection of g, MFNs obtain approximations comparable to those of SIRENs (Sitzmann et al., 2020) with faster convergence rate. The most successful instantiation of MNFs are the Multiplicative Gabor Network (MGN): MFNs constructed with isotropic Gabor functions as basis g (in Eq. 2):
g([x, y];θ(l)) = exp( − γ (l)
2 [(x −µ(l))2 + (y −µ(l))2])Sin(W(l)g ⋅ [x, y] + b (l) g ), (5)
θ(l)={γ(l) ∈ RNhid ,µ(l) ∈ RNhid ,W(l)g ∈ RNhid×2,b (l) g ∈ RNhid}. (6)
Note that, by setting N=Nout×Nin, an MFN can parameterize a convolutional kernel with Nin input and Nout output channels. Fathony et al. (2021) show that MFNs are equivalent to a linear combination of exponentially many basis functions g. This allows us to analytically derive properties of MFN representations, and plays a crucial role in the derivation of alias-free MAGNets (Sec. 3.3).
Multiplicative Anisotropic Gabor Networks (MAGNets). Our MAGNet formulation is based on the observation that isotropic Gabor functions, i.e., with equal γ for the horizontal and vertical directions, are undesirable as basis for the construction of MFNs. Whenever a frequency is required along a certain direction, an isotropic Gabor function automatically introduces that frequency in both directions. As a result, other bases must counteract this frequency in the direction where the frequency is not required, and thus the capacity of the MFN is not used optimally (Daugman, 1988).
Following the original formulation of the 2D Gabor functions (Daugman, 1988), we alleviate this limitation by using anisotropic Gabor functions instead:
g([x, y];θ(l)) = exp( − 1 2 [(γ(l)X (x −µ (l) X ))
2
+ (γ(l)Y (y −µ (l) Y ))
2
])Sin(W(l)g [x, y] + b (l) g ) (7)
θ(l)={γ(l)X ∈ R Nhid ,γ (l) Y ∈ R Nhid ,µ (l) X ∈ R Nhid ,µ (l) Y ∈ R Nhid ,W(l)g ∈ RNhid×2,b (l) g ∈ RNhid}. (8)
The resulting Multiplicative Anisotropic Gabor Network (MAGNet) obtains better control upon frequency components introduced to the approximation, and demonstrates important improvements in terms of descriptive power and convergence speed (Sec. 4).
MAGNet initialization. Fathony et al. (2021) proposes to initialize MGNs by drawing the size of the Gaussian envelopes, i.e., the γ(l) term, from a Gamma(α ⋅ L−1, β) distribution at every layer l ∈ [1, ..,L − 1]. We observe however that this initialization does not provide much variability on the initial extension of the Gaussian envelopes and in fact, most of them cover a large portion of the space at initialization. To stimulate diversity, we initialize the {γ(l)X ,γ (l) Y } terms by a Gamma(αl
−1, β) distribution at the l-th layer. We observe that our proposed initialization consistently leads to better accuracy than the initialization of Fathony et al. (2021) across all tasks considered. (Sec. 4).

heading: 3.3 ANALYTIC ALIAS-FREE MAGNETS
text: FlexConvs can be deployed at higher resolutions than those observed during training, simply by sampling the underlying continuous representation of the kernel more densely, and accounting for the
change in sampling rate. Consider a D-dimensional input signal fr(1) with resolution r (1). FlexConv learns a kernel ψr(1) that can be inferred at a higher resolution r (2) (Romero et al., 2021):
(fr(2) ∗ψr(2)) ≈ ( r(1) r(2) ) D (fr(1) ∗ψr(1)). (9)
Note however, that Eq. 9 holds approximately. This is due to aliasing artifacts which can appear if the frequencies in the learned kernel surpass the Nyquist criterion of the target resolution. Consequently, an anti-aliased parameterization is vital to construct kernels that generalize well to high resolutions.
Towards alias-free implicit neural representations. We observe that SIRENs as well as unconstrained MFNs and MAGNets exhibit aliasing when deployed on resolutions higher than the training resolution, which hurts performance of the model. An example kernel with aliasing is shown in Fig. 8.
To combat aliasing, we would like to control the representation learned by MAGNets. MAGNets –and MFNs in general– construct implicit neural representations that can be seen as a linear combination of basis functions. This property allows us to analytically derive and study the properties of the resulting neural representation. Here, we use this property to derive the maximum frequency of MAGNet-generated kernels, so as to regularize MAGNets against aliasing during training. We analytically derive the maximum frequency of a MAGNet, and penalize it whenever it exceeds the Nyquist frequency of the training resolution. We note that analytic derivations are difficult for other implicit neural representations, e.g., SIRENs, due to stacked layer-wise nonlinearities.
Maximum frequency of MAGNets. The maximum frequency component of a MAGNet is given by:
f+MAGNet = L
∑ l=1 max il ⎛ ⎝ ⎛ ⎝ max j
W (l) g,il,j
2π ⎞ ⎠ + σcut min{γ(l)X,il ,γ (l) Y,il } 2π ⎞ ⎠ , (21)
where L corresponds to the number of layers, W(l)g ,γ (l) X ,γ (l) Y to the MAGNet parameters as defined in Eq. 8, and σcut=2 ⋅ stdev to the cut-off frequency of the Gaussian envelopes in the Gabor filters. A formal treatment as well as the derivations can be found in Appx. A.1.
Effect of the FlexConv mask. The Gaussian mask used to localize the response of the MAGNet also has an effect on the frequency spectrum. Hence, the maximum frequency of a FlexConv kernel is:
f+FlexConv = f+MAGNet + f+wgauss , with f + wgauss = σcut max{σX, σY}2π . (22)
Here, σX, σY correspond to the mask parameters (Eq. 1). Intuitively, multiplication with the mask blurs in the frequency domain, as it is equivalent to convolution with the Fourier transform of the mask.
Aliasing regularization of FlexConv kernels. With the analytic derivation of f+FlexConv we penalize the generated kernels to have frequencies smaller or equal to their Nyquist frequency fNyq(k) via:
LHF = ∣∣max{f+FlexConv, fNyq(k)} − fNyq(k)∣∣2, with fNyq(k) = k−14 . (25) Here, k depicts the size of the FlexConv kernel before applying the Gaussian mask, and is equal to the size of the input signal. In practice, we implement Eq. 25 by regularizing the individual MAGNet layers, as is detailed in Appx. A.2. To verify our method, Fig. 8 (Appx. A.1) shows that the frequency components of FlexNet kernels are properly regularized for aliasing.

heading: 4 EXPERIMENTS
text: We evaluate FlexConv across classification tasks on sequential and image benchmark datasets, and validate the ability of MAGNets to approximate complex functions. A complete description of the datasets used is given in Appx. B. Appx. D.2 reports the parameters used in all our experiments.1

heading: 4.1 WHAT KIND OF FUNCTIONS CAN MAGNETS APPROXIMATE?
text: Bandwidth of methods with learnable sizes. First, we compare the bandwidth of MAGNet against N-Jet (Pintea et al., 2021) by optimizing each to fit simple targets: (i) Gabor filters of known frequency, (ii) random noise and (iii) an a 11 × 11 AlexNet kernel from the first layer (Krizhevsky et al., 2012). Fig. 4 shows that, even with 9 orders of Gaussian derivatives, N-Jets cannot fit high frequency signals in large kernels. Crucially, N-Jet models require many Gaussian derivative orders to model high frequency signals in large kernels: a hyperparameter which proportionally increases their inference
1Our code is publicly available at anonymous-link-due-to-double-blind-review.
time and parameter count. MAGNets, on the other hand, accurately model large high frequency signals. This allows FlexNets to learn large kernels with high frequency components.
Expressivity of MLP parameterizations. Next, we compare the descriptive power and convergence speed of MAGNets, Gabor MFNs, Fourier MFNs and SIRENs for image approximation. To this end, we fit the images in the Kodak dataset (Kodak, 1991) with each of these methods. Our results (Tab. 5) show that MAGNets outperform all other methods, and converge faster to good approximations.

heading: 4.2 CLASSIFICATION TASKS
text: Network specifications. Here, we specify our networks for all our classification experiments. We parameterize all our convolutional kernels as the superposition of a 3-layer MAGNet and a learnable anisotropic Gaussian mask. We construct two network instances for sequential and image datasets respectively: FlexTCNs and FlexNets. Both are constructed by taking the structure of a baseline network –TCN (Bai et al., 2018a) or CIFARResNet (He et al., 2016)–, removing all internal pooling layers, and replacing convolutional kernels by FlexConvs. The FlexNet architecture is shown in Fig. 10 and varies only in the number of channels and blocks, e.g., FlexNet-16 has 7 blocks. Akin to Romero et al. (2021) we utilize the Fourier theorem to speed up convolutions with large kernels.
Mask initialization. We initialize the FlexConv masks to be small. Preliminary experiments show this leads to better performance, faster execution, and faster training convergence. For sequences, the mask center is initialized at the last kernel position to prioritize the last information seen.
Time series and sequential data. First we evaluate FlexTCNs on sequential classification datasets, for which long-term dependencies play an important role. We validate our approach on intrinsic
discrete data: sequential MNIST, permuted MNIST (Le et al., 2015), sequential CIFAR10 (Chang et al., 2017), noise-padded CIFAR10 (Chang et al., 2019), as well as time-series data: CharacterTrajectories (CT) (Bagnall et al., 2018), SpeechCommands (Warden, 2018) with raw waveform (SC_raw) and MFCC input representations (SC).
Our results are summarized in Tables 1 and 2. FlexTCNs with two residual blocks obtain state-ofthe-art results on all tasks considered. In addition, depth further improves performance. FlexTCN-6 improves the current state-of-the-art on sCIFAR10 and npCIFAR10 by more than 6%. On the difficult SC_raw dataset –with sequences of length 16000–, FlexTCN-6 outperform the previous state-of-the-art by 20.07%: a remarkable improvement.
Furthermore, we conduct ablation studies by changing the parameterization of MLPψ , and switching off the learnable kernel size ("CKTCNs") and considering global kernel sizes instead. CKTCNs and FlexTCNs with MAGNet kernels outperform corresponding models with all other kernel parameterizations: SIRENs (Sitzmann et al., 2020), MGNs and MFNs (Fathony et al., 2021). Moreover, we see a consistent improvement with respect to CKCNNs (Romero et al., 2021) by using learnable kernel sizes. This shows that both MAGNets and learnable kernel sizes contribute to the performance of FlexTCNs. Note that in 1D, MAGNets are equivalent to MGNs. However, MAGNets consistently perform better than MGNs. This improvement in accuracy is a result of our MAGNet initialization.
Image classification. Next, we evaluate FlexNets for image classification on CIFAR-10 (Krizhevsky et al., 2009). Additional experiments on Imagenet-32, MNIST and STL-10 can be found in Appx. C.
Table 3 shows our results on CIFAR-10. FlexNets are competitive with pooling-based methods such as CIFARResNet (He et al., 2016) and outperform learnable kernel size method DCNs (Tomen et al., 2021). In addition, we compare using N-Jet layers of order three (as in Pintea et al. (2021)) in FlexNets against using MAGNet kernels. We observe that N-Jet layers lead to worse performance, and are significantly slower than FlexConv layers with MAGNet kernels. The low accuracy of N-Jet layers is likely to be linked to the fact that FlexNets do not use pooling. Consequently, N-Jets are forced to learn large kernels with high-frequencies, which we show N-Jets struggle learning in Sec. 4.1.
To illustrate the effect of learning kernel sizes, we also compare FlexNets against FlexNets with large and small discrete convolutional kernels (Tab. 3). Using small kernel sizes is parameter efficient, but is not competitive with FlexNets. Large discrete kernels on the other hand require a copious amount of parameters and lead to significantly worse performance. These results indicate that the best solution is somewhere in the middle and varying kernel sizes can learn the optimal kernel size for the task at hand.
Similar to the sequential case, we conduct ablation studies on image data with learnable, nonlearnable kernel sizes and different kernel parameterizations. Table 3 shows that FlexNets outperform CKCNNs with corresponding kernel parameterizations. In addition, a clear difference in performance is apparent for MAGNets with respect to other parameterizations. These results corroborate that both MAGNets and FlexConvs contribute to the performance of FlexNets. Moreover, Tab. 3 illustrates the effect of the two contributions of MAGNet over MGN: anisotropic Gabor filters, and our improved initialization. Our results in image data are in unison with our previous results for sequential data (Tabs. 1, 2) and illustrate the value of the proposed improvements in MAGNets.

heading: 4.3 ALIAS-FREE FLEXNETS
text: Regularizing the FlexConv mask. Though including f+wgauss in the frequency analysis of MAGNets is crucial for the accuracy of the derivation, including the FlexConv mask in aliasing regularization is undesirable, as it steers the model to learn large kernels in order to minimize the loss (see Eq. 25). However, excluding the mask from regularization could compromise the ability of FlexNet to generalize to higher resolutions. Here, we experiment with this trade-off.
Table 4: Alias-free FlexNets on CIFAR-10.
MODEL SIZE CIFAR-10 ACC.16 px ∆16px 32 px
CIFARResNet-44 0.66M 85.8 ± 0.2 -31.6 ± 1.3 FlexNet-16 w/ conv. (k = 3) 0.17M 85.3 ± 0.2 -21.2 ± 1.0 FlexNet-16 w/ conv. (k = 33) 20.0M 67.7 ± 0.6 -57.1 ± 1.6
FlexNet-16 w/ N-Jets 0.70M 86.4 ± 0.2 -5.5 ± 1.3 CKCNN-16SIREN 0.63M 45.9 ± 1.0 -15.8 ± 1.2 FlexNet-16SIREN 0.63M 70.4 ± 0.8 -50.0 ± 16.9
FlexNet-16 w/o reg. 0.67M 86.4 ± 0.4 -34.4 ± 14.3 FlexNet-16 w/ reg. f+MAGNet 0.67M 86.5 ± 0.1 -3.8 ± 2.0 FlexNet-16 w/ reg. f+FlexConv 0.67M 85.1 ± 0.3 -3.3 ± 0.3
Figure 5 shows accuracy change between ten source and target resolution combinations on CIFAR-10, both for including and excluding the FlexConv mask in the aliasing regularization. We train at the source resolution for 100 epochs, before testing the model at the target resolution with the upsampling described in Sec. 3.3. Next, we adjust fNyq(k) to the target resolution, and finetune each model for 100 epochs at the target resolution.
We find that regularizing just f+MAGNet yields a trade-off. It increases the accuracy difference between low and high resolution inference, but also increases the fine-tune accuracy at the target resolution.We therefore choose to, by default, regularize f+MAGNet only.
Results of our alias-free FlexNet training on CIFAR-10 are in Table 4. We observe that the performance of a FlexNet trained without aliasing regularization largely breaks down when the dataset is upscaled. However, with our aliasing regularization most of the performance is retained.
Comparatively, FlexNet retains more of the source resolution performance than FlexNets with N-Jet layers, while baselines degrade drastically at the target resolution. Fig. 8 shows the effect of aliasing regularization on the frequency components of FlexConv.
Training at lower resolutions saves compute. We can train alias-free FlexNets at lower resolutions. To verify that this saves compute, we time the first 32 batches of training a FlexNet-7 on CIFAR-10. We compare against training on 16 × 16 images (downsampled before training). On 16x16 images, each batch takes 179ms (± 7ms). On 32x32 images, each batch takes 222ms (± 9ms). Therefore, we save 24% training time when training FlexNets alias-free at half the native CIFAR-10 resolution.

heading: 5 DISCUSSION
text: Learned kernel sizes match conventional priors. Commonly, CNNs use architectures of small kernels and pooling layers. This allows convolutions to build a progressively growing receptive field. With learnable kernel sizes, FlexNet could learn a different prior over receptive fields, e.g., large kernels first, and small kernels next. However, FlexNets learn to increase kernel sizes progressively (Fig. 6), and match the network design that has been popular since AlexNet (Krizhevsky et al., 2012).
Mask initialization as a prior for feature importance. The initial values of the FlexConv mask can be used to prioritize information at particular input regions. For instance, initializing the center of mask on the first element of sequential FlexConvs can be used to prioritize information from the far past. This prior is advantageous for tasks such as npCIFAR10. We observe that using this prior on npCIFAR10 leads to much faster convergence and better results (68.33% acc. w/ FlexTCN-2).
MAGNet regularization as prior induction. MAGNets allow for analytic control of the properties of the resulting representations. We use this property to generate alias-free kernels. However, other desiderata could be induced, e.g., smoothness, for the construction of implicit neural representations.
Benefits of cropping and the influence of PyTorch. Dynamic cropping adjust the computational cost of the convolutions on the fly. For a signal of size M and a cropped kernel size k, this incurs in savings from O(M2 D
) to O(MDkD) relative to using global kernel sizes (O(M4) to O(M2k2) in 2D). We test this theoretical speed up in a controlled environment for the Speech Commands and CIFAR10 datasets. Cropping reduces the per-epoch run time by a factor of 11.8x and 5.5x for Speech Commands and CIFAR-10, respectively. Interestingly, however, both run times become similar if the flag torch.backends.cudnn.benchmark is activated, with global kernel sizes being sometimes faster. This is because this flag tells PyTorch to optimize the convolution algorithms used under the hood, and some of these CUDA algorithms seem to be faster than our masking strategy on Python.

heading: 6 LIMITATIONS
text: Dynamic kernel sizes: computation and memory cost of convolutions with large kernels. Performing convolutions with large convolutional kernels is a compute-intensive operation. FlexConvs are initialized with small kernel sizes and their inference cost is relatively small at the start of training. However, despite the cropping operations used to improve computational efficiency (Figs. 1, 3, Tab. 3), the inference time may increase to up to double as the learned masks increase in size. At the cost of more memory, convolutions can be sped up by performing them in the frequency domain. However, we observe that this does not bring gains for the image data considered because FFT convolutions are faster only for very large convolutional kernels (in the order of hundreds of pixels).
Remaining accuracy drop in alias-free FlexNets. Some drop in accuracy is still observed when using alias-free FlexNets at a higher test resolutions (Tab. 4). Although more evidence is needed, this may be caused by aliasing effects introduced by ReLU (Vasconcelos et al., 2021), or changes in the activation statistics of the feature maps passed to global average pooling (Touvron et al., 2019).

heading: 7 CONCLUSION
text: We propose FlexConv, a convolutional operation able to learn high bandwidth convolutional kernels of varying size during training at a fixed parameter cost. We demonstrate that FlexConvs are able to model long-term dependencies without the need of pooling, and shallow pooling-free FlexNets achieve state-of-the-art performance on several sequential datasets, match performance of recent works with learned kernel sizes with less compute, and are competitive with much deeper ResNets on image benchmark datasets. In addition, we show that our alias-free convolutional kernels allow FlexNets to be deployed at higher resolutions than seen during training with minimal precision loss.
Future work. MAGNets give control over the bandwidth of the kernel. We anticipate that this control has more uses, such as fighting sub-sampling aliasing (Zhang, 2019; Kayhan & Gemert, 2020; Karras et al., 2021). With the ability to upscale FlexNets to different input image sizes comes the possibility of transfer learning representations between previously incompatible datasets, such as CIFAR-10 and Imagenet. In a similar vein, the automatic adaptation of FlexConv to the kernel sizes required for the task at hand may make it possible to generalize the FlexNet architecture across different tasks and datasets. Neural architecture search (Zoph & Le, 2016) could see benefits from narrowing the search space to exclude kernel size and pooling layers. In addition, we envisage additional improvements from structural developments of FlexConvs such as attentive FlexNets.
REPRODUCIBILITY STATEMENT
We hope to inspire others to use and reproduce our work. We publish the source code of this work, for which the link is provided in Sec. 4.2. Sec. 4 and Appx. D.1 detail FlexNet, its hyperparameters and optimization procedure. The full derivation of the aliasing regularization objective is included in Appx. A.1. We report means over multiple runs for many experiments, to ensure the reported results are fair and reproducible, and do not rely on tuning of the random seed. All datasets used in our experiments are publicly available. If any questions remain, we welcome one and all to contact the corresponding author.

heading: A ALIAS-FREE FLEXCONV REGULARIZATION
text: In this section we provide the complete derivation and analysis for our FlexConv regularization against aliasing. First, we derive the analytic maximum frequency component of a FlexConv kernel. Next, we compute the Nyquist frequency of a FlexConv kernel, and subsequently show how to combine the previous results into a regularization term to train alias-free FlexConvs.
A.1 ANALYZING THE FREQUENCY SPECTRUM OF FLEXCONV
In order to make FlexConv alias-free (Sec. 3.3), we need to compute the maximum frequency component of the kernels generated by a MAGNet, so that we can regularize it during training. In this section we analytically derive this maximum frequency component from the parameters of the MAGNet.
Recall that MAGNets generate a kernel ψ(x, y) through of a succession of anisotropic Gabor filters and linear layers (Sec. 3.2, Eqs. 2–7):
h(1) = g([x, y];θ(1)) g ∶ R2 → RNhid
h(l) = (W(l)h(l−1) + b(l)) ⋅ g([x, y];θ(l)) W(l) ∈ RNhid×Nhid ,b(l) ∈ RNhid
ψ(x, y) =W(L)h(L−1) + b(L) W(L) ∈ R(Nout×Nin)×Nhid ,b(L) ∈ R(Nout×Nin)
g([x, y];θ(l)) = exp( − 1 2 [(γ(l)X (x −µ (l) X ))
2
+ (γ(l)Y (y −µ (l) Y ))
2
])Sin(W(l)g [x, y] + b (l) g )
θ(l)={γ(l)X ∈ R Nhid ,γ (l) Y ∈ R Nhid ,µ (l) X ∈ R Nhid ,µ (l) Y ∈ R Nhid ,W(l)g ∈ RNhid×2,b (l) g ∈ RNhid}
To analyse the maximum frequency component f+MAGNet, we analyse the frequency components of the Gabor filters used in MAGNet, and retain their maximum. We then plug the found frequency component into the analysis of Fathony et al. (2021) to show how the frequency responses of Gabor filters and linear layers interact in MFNs. Finally, we add the effect of the FlexConv Gaussian mask to our analysis to obtain the maximum frequency component ot the final FlexConv kernel f+FlexConv.
Sine term in Gabor filters. In a Gabor filter, the sine term is multiplied with a Gaussian envelope. The frequency (in radians) of a sine function of the form Sin(wT [x, y]+ b) is given byw. We divide by 2π to convert the frequency units to Hertz, for compatibility with the rest of the analysis. For 2D inputs, the maximum frequency component of the sine function correspond to the largest frequency in the two input dimensions:
f+Sin = max j
wj 2π . (10)
The sine terms in MAGNets have multiple output channels: Sin(Wg ⋅ [x, y] + b(l)g ). Effectively, we compute the sine term independently for each channel:
f+Sin,i = max j
Wg,i,j
2π . (11)
Gaussian term in Gabor filters. In a Gabor filter, a Gaussian envelope modulates a sine term. Let us assume for now that the Gaussian envelope is isotropic, rather than anisotropic as in MAGNets, and has single-channel output. By applying the convolution theorem, the sine term is equivalently convolved with the Fourier transform of the Gaussian envelope in the frequency domain. Since the Fourier transform of a Gaussian envelope is another Gaussian envelope, the application of a Gaussian envelope amounts to blurring with a Gaussian kernel in the frequency domain. The size of the envelope in the Fourier domain σF can be derived from the standard deviation of the Guassian envelope in the spatial domain σT as follows:
σTσF = 1
2π ⇒ σF =
1
2πσT . (12)
Gaussian blurs induce impulse signals to have a long tail. Consequently, we must define a cutoff point for this tail in terms of standard deviations to derive the maximum added frequency induced by the blur. We describe the cutoff point as σcut ∈ N. Typical choices for σcut are known as the empirical, or the "68-95-99.7" rule (Hald, 2007). We choose a standard of two standard deviations, i.e., σcut=2, which covers 95% of the mass of the Gaussian envelope. For an isotropic Gabor filter with γ=σ−1T , the maximum frequency of its Gaussian envelope f+env is:
f+env = σcut 2π(σT)−1 = σcutγ 2π . (13)
Anisotropic envelopes. Our analysis so far assumes an isotropic Gaussian envelope in the Gabor filter. However, we need to account for the anisotropic Gaussian envelopes in MAGNets. Anisotropic filters have not one but two γ parameters: {γX, γY}. The smallest of these will contribute most to f+env, as it will blur the most, so it is sufficient to compute f + env only using the smallest of the two γ terms: f+env(γX , γY ) = f+env(min{γX , γY }). (14)
The other assumption we made before was to work with single-channel outputs. MAGNets however use multi-channel outputs with independent Gaussian terms. The maximum frequency of multichannel Gaussian envelopes is given by:
f+env,i(γX,γY) = f+env (min{γX,i,γY,i}) = σcut min{γX,i,γY,i}
2π , (15)
where the subscript i indexes the channels of the multi-channel Gaussian envelopes.
Maximum frequency component of anisotropic Gabor filters. Finally, the maximum frequency component of the i-th channel of an anisotropic Gabor filter g is given by:
f+Gabor,i = f+Sin,i(Wg) + f+env,i(γX,γY)
= (max j
Wg,i,j
2π ) + σcut min{γX,i,γY,i} 2π . (16)
Figure 9 illustrates the frequency spectrum of an example Gabor filter.
Maximum frequency component of a MAGNet. Fathony et al. (2021) characterize the expansion of each term of the isotropic Gabor layers in MFNs in the final MFN output. In Eq. 25, Fathony et al. (2021) demonstrate that the MFN representation contains a set of sine frequencies ω given by:
ω = {sLω(L)iL + sL−1ω (L−1) iL−1 +⋯ + slω(2)i2 + ω (1) i1 } . (17)
Here, the indexes i1, i2,⋯, iL−1 range over all possible indices of each hidden unit of each layer of an MFN, and s2,⋯, sL ∈ {−1,+1} range over all 2L−1 possible binary signs. In other words, Fathony et al. (2021) demonstrate that the representation of an MFN at a particular layer contains an exponential combination of all possible positive and negative combinations of the frequencies of the sine terms in each hidden unit at each layer in the MFN up to the current layer.
The original analysis uses these terms to argue that MFNs model exponentially many terms through a linear amount of layers. For our purpose of computing the frequency response of the MAGNet generated kernel, we can plug our derivation of the frequencies of the Gabor filter fGabor into ω to compute the frequency spectrum of the generated kernel:
f+MAGNet = {sLf (L) Gabor,iL + sL−1f (L−1)Gabor,iL−1 +⋯ + s2f (2) Gabor,i2 + f (1)Gabor,i1} (18)
As stated before, we are only interested in the maximum frequency in the frequency spectrum. We can therefore simplify Eq. 18 in two ways. First, we simplify over MAGNet layers by taking the maximum value of the spectrum, which is the sum over all layers using only the positive binary signs in sL (Eq. 19). Next, we simplify over channel indices by retaining only the channel index that results in the highest frequency (Eq. 20). The maximum frequency of a MAGNet is shown in Eq. 21:
f+MAGNet = {(+1)f + (L) Gabor,iL + (+1)f+ (L−1)Gabor,iL−1 +⋯ + (+1)f + (2) Gabor,i2 + f+ (1)Gabor,i1} (19)
= {f+ (L)Gabor,iL + f + (L−1) Gabor,iL−1 +⋯ + f+ (2)Gabor,i2 + f + (1) Gabor,i1 }
f+MAGNet = max iL (f+ (L)Gabor,iL) +maxiL−1 (f+ (L−1)Gabor,iL−1)⋯+maxi2 (f+ (2)Gabor,i2) +maxi1 (f+ (1)Gabor,i1) (20)
= L
∑ l=1 max il
(f+ (l)Gabor,il)
= L
∑ l=1 max il ⎛ ⎝ ⎛ ⎝ max j
W (l) g,il,j
2π ⎞ ⎠ + σcut min{γ(l)X,il ,γ (l) Y,il } 2π ⎞ ⎠ . (21)
Effect of the Gaussian mask in the frequency components of a FlexConv. FlexConvs attenuate the MAGNet output with a Gaussian mask. The Gaussian mask (Eq. 1) works analogously to the Gaussian envelope term in the Gabor filter: it blurs the frequency components of the generated kernel with standard deviation σF. Therefore, we can reuse our derivation for the Gaussian envelope of the
Gabor filter (Eq. 15). The maximum frequency component of a FlexConv kernel is given by: f+FlexConv = f+MAGNet + f+env
= f+MAGNet + σcut min{σ−1X , σ−1Y }
2π = f+MAGNet + σcut max{σX, σY}2π
= L
∑ l=1 max il ⎛ ⎝ ⎛ ⎝ max j
W (l) g,il,j
2π ⎞ ⎠ + σcut min{γ(l)X,il ,γ (l) Y,il } 2π ⎞ ⎠ + σcut max{σX, σY}2π . (22)
Visualization of regularized kernels. Fig. 8 shows example kernels from FlexNets trained with aliasing regularization. The frequency domain plots confirm the accuracy of our frequency component regularization.
A.2 REGULARIZING THE FREQUENCY RESPONSE OF FLEXCONV
Nyquist frequency of a FlexConv kernel. Given the sampling rate fs of the kernel, we can compute its Nyquist frequency fNyq as:
fNyq = 1
2 fs (23)
To compute the sampling rate, we note that the kernel coordinates input to our MAGNet stretch over a [−1,1]D domain. For a kernel of length k, we therefore sample one point in every fs = k−12 units. Knowing the sampling rate in terms of the kernel size allows us to express the Nyquist frequency in terms of the (pre-masked) kernel size:
fNyq(k) = 1
2 k − 1 2 = k − 1 4 . (24)
Note that the kernel size in a FlexConv is initialized to be equal to the resolution of the data, if it is odd. For even resolutions, it corresponds to the resolution of the data plus one.
Constructing the regularization term. We train FlexConv with a regularization term on the frequency response of the generated kernel to ensure that aliasing effects do not distort the performance of the model when it is inferred at a higher resolution. This section details the implementation of the regularization function.
From the parameters of each FlexConv module, we compute f+FlexConv according to Eq. 22. For the amount of standard deviations to use in determining f+env (Eq. 15) we use h = 2. From the kernel size k of the FlexConv module we compute fNyq(k) according to Eq. 24. We then apply an L2 regularizer over the amount that f+FlexConv exceeds fNyq(k):
LHF = ∣∣max{f+FlexConv, fNyq(k)} − fNyq(k)∣∣2. (25)
We weight LHF by λ = 0.1 when adding it to our loss function. Improved implementation. Eq. 25 contains a sum over the L layers of the MAGNet. In practice, we prefer to regularize each layer l ∈ L separately, so that the gradients of the regularization of different layers are not dependent on each other. We therefore implement the anti-aliasing regularization by regularizing each MAGNet layer independently, and spreading the f+env term from the gaussian mask uniformly over all MAGNet layers:
LHF,l = ∣∣max{f+MAGNet,l + f+env L , fNyq(k) L } − fNyq(k) L ∣∣2 (26)
= ∣∣max{max il
(f+ (l)Gabor,il) + f+env L , fNyq(k) L } − fNyq(k) L ∣∣2. (27)
In the code, we refer to this method as the together method, versus the summed method of Eq. 25. In preliminary experiments, we observed improved performance of anti-aliasing training when using the together method. All of our experiments anti-aliasing experiments therefore use the together setting.

heading: B DATASET DESCRIPTION
text: B.1 IMAGE FITTING DATASETS
Kodak dataset. The Kodak dataset (Kodak, 1991) consists of 24 natural images of size 768 × 512. This dataset is a popular benchmark used for compression and image fitting methods.
B.2 SEQUENTIAL DATASETS
Sequential and Permuted MNIST. The sequential MNIST dataset (sMNIST) (Le et al., 2015)takes the 28×28 images from the original MNIST dataset (LeCun et al., 1998), and presents them as a sequence of 784 pixels. The goal of this task is to perform digit classification given the representation of the last sequence element of a sequential model. Consequently, good predictions require the model to preserve long-term dependencies up to 784 steps in the past.
The permuted MNIST dataset (pMNIST) additionally changes the order of all the sMNIST sequences by a random permutation. Consequently, models can no longer rely on local features to construct good feature representations. As a result, the classification problem becomes more difficult, and the importance of long-term dependencies more pronounced.
Sequential and Noise-Padded CIFAR10. The sequential CIFAR10 dataset (sCIFAR10) (Chang et al., 2017) takes the 32×32 images from the original CIFAR10 dataset (Krizhevsky et al., 2009) and presents them as a sequence of 1,024 pixels. The goal of this task is to perform image classification given the representation of the last sequence element of a sequential model. This task is more difficult than sMNIST, as a larger memory horizon is required to solve the task and more complex structures and intra-class variations are present in the data (Bai et al., 2018b).
The noise-padded CIFAR10 dataset (npCIFAR10) (Chang et al., 2019) flattens the images from the original CIFAR10 dataset (Krizhevsky et al., 2009) along their rows to create a sequence of length 32, and 96 channels (32 rows × 3 channels). Next, these sequences are concatenated with 968 entries of noise to form the final sequences of length 1000. As for sCIFAR10, the goal of the task is to perform image classification given the representation of the last sequence element of a sequential model.
CharacterTrajectories. The CharacterTrajectories dataset is part of the UEA time series classification archive (Bagnall et al., 2018). It consists of 2858 time series of length 182 and 3 channels representing the x, y positions, and the tip force of a pen while writing Latin alphabet characters in a single stroke. The goal is to classify out of 20 classes the written character using the time series data.
Speech Commands. The Speech Commands dataset (Warden, 2018) consists of 105,809 onesecond audio recordings of 35 spoken words sampled at 16kHz. Following Kidger et al. (2020), we extract 34975 recordings from ten spoken words to construct a balanced classification problem. We refer to this dataset as SpeechCommands_raw, or SC_raw for short. Furhtermore, we utilize the preprocessing steps of Kidger et al. (2020) and extract mel-frequency cepstrum coefficients from the raw data. The resulting dataset, abreviated SC, consists of time series of length 101, and 20 channels.
B.3 IMAGE BENCHMARK DATASETS
MNIST. The MNIST hadwritten digits datset (LeCun & Cortes, 2010) consists of 70,000 grayscale handwritten digits of size 28×28, divided into a training and test sets of 60,000 and 10,000 images, respectively. The goal of the task is to classify these digits as one of the ten possible digits (0,1, ..8,9). CIFAR-10 The CIFAR-10 dataset (Krizhevsky et al., 2009) consists of 60,000 natural images from 10 classes of size 32×32, divided into training and test sets of 50,000 and 10,000 images, respectively. STL-10. The STL-10 dataset (Coates et al., 2011) is a subset of the ImageNet dataset (Krizhevsky et al., 2012) consisting of 5,000 natural images from 10 classes of size 96×96, divided into trainint and test sets of 4,500 and 500 images, respectively.
ImageNet-k. The Imagenet-k (Chrabaszcz et al., 2017) dataset is derived from the ImageNet dataset Russakovsky et al. (2015) by downsampling all samples to a resolution k ∈ [64,32,16,8]. The dataset contains 1000 classes with 1,281,167 training samples and 50,000 validation samples.

heading: C ADDITIONAL EXPERIMENTS
text: C.1 IMAGE CLASSIFICATION
CIFAR-10. Tab. 6 shows all results for our CIFAR-10 experiments, including more ablations.
ImageNet-32. Results for the ImageNet-32 experiment are shown in Table 7. FlexNets are slightly worse than CIFARResNet-32 (He et al., 2016) with slightly less parameters. However, the results
reported by Chrabaszcz et al. (2017) for Wide ResNets (Zagoruyko & Komodakis, 2016) outperform FlexNets by a significant margin.
Alias-free ImageNet-32. We report results for alias-free FlexNets on ImageNet-k (Chrabaszcz et al., 2017) in Table 8, to verify the results of alias-free training at a larger scale. We find that FlexConv and N-Jet both mostly retain classification accuracy between source and target resolution, while CIFARResNet-32 degrades drastically.
MNIST and STL-10. We additionally report results on MNIST (Tab. 9) and STL-10 (Tab. 10. We choose these dataset for the difference in image sizes of the training data. On MNIST, though performance on MNIST is quite saturated, we are competitive with state of the art methods. On STL-10 we are significantly worse than the baseline CIFARResNet from (Luo et al., 2020), though with significantly less parameters. We were not able to prepare a more relevant baseline for this experiment.

heading: D EXPERIMENTAL DETAILS
text: D.1 FLEXNET
We propose an image classification architecture named FlexNet (Fig. 10), consisting of a stack of FlexConv blocks followed by a global average pooling layer and a linear layer. FlexNets are named "FlexNet-L" where L indicates the amount of layers in the architecture.
FlexBlock. Each FlexBlock consists of two FlexConvs with BatchNorm (Ioffe & Szegedy, 2015) and dropout (Srivastava et al., 2014) (d = 0.2) as well as a residual connection. The width of a block i is determined by scaling a base amount c by progressively increasing factors: ci = [c, c × 1.5, c × 1.5, c × 2.0, c × 2.0](i). The default configuration of FlexNet uses c = 22. In FlexNet-N-Jet models, we scale c to match the amount of parameters of the FlexNet in the comparison.
FlexConv initialization. We initialize the FlexConv mask variances small, at σ2X, σ 2 Y = 0.125. For initializing MAGNet, we initialize the Gaussian envelopes as discussed in Sec. 3.2. We initialize the linear layer weights by the same Gamma distribution as used for the enveloped, modulated by a scaling factor of 25.6. We found that this value of the scaling factor, rather than a higher one, helped in reducing the performance of alias-free models. We initialize the bias of the linear layers by U(−π,π). CIFAR-10. In FlexNet-16 models for CIFAR-10 we use c = 24 to approximate the parameter count of CIFARResNets in the experiment.
D.2 OPTIMIZATION
We use Adam (Kingma & Ba, 2014) to optimize FlexNet. Unless otherwise specified, we use a learning rate of 0.01 with a cosine annealing scheme (Loshchilov & Hutter, 2016) with five warmup epochs. We use a different learning rate of 0.1× the regular learning rate for the FlexConv Gaussian mask parameters. We do not use weight decay, unless otherwise specified.
Kodak. We overfit on each image of the dataset for 20,000 iterations. To this end, we use a learning rate of 0.01 without any learning rate scheme. We observe that SIRENs diverge with this learning rate and thus, reduce the learning rate to 0.001 for these models.
CIFAR-10. We train for 350 epochs with a batch size of 64. We use the data augmentation from He et al. (2016) when training CIFAR-10: a four pixel padding, followed by a random 32 pixel crop and a random horizontal flip.
ImageNet-32. We train for 350 epochs with a batch size of 2048. We use the same data augmentation as used for CIFAR-10. We do use a weight decay of 1e−5 for ImageNet-32 training. Sequential and Permuted MNIST. We train for 200 epochs with a batch size of 64 and a learning rate of 0.01. We use a weight decay of 1e−5. Sequential and Noise-Padded CIFAR-10. For sequential CIFAR-10, we train for 200 epochs with a batch size of 64, a learning rate of 0.001 and a weight decay of 1e−5. For noise-padded CIFAR-10, we train for 300 epochs with a batch size of 32, a learning rate of 0.01 and no weight decay.
Speech Commands and CharTrajectories. We train for 300 epochs with a batch size of 32 and a learning rate of 0.001. For CharTrajectories, we use a weight decay of 1e−5.
D.3 ROTATED GAUSSIAN MASKS
MAGNets use anisotropic Gaussian terms in the Gabor filters, which yields improvements in descriptive power and convergence speed (Sec. 3.2). For the same reason, we explore making the anisotropic FlexConv Gaussian mask steerable, by including an additional vector of learnable angle parameters φ(l) ∈ RNhid that rotates the Gaussian masks. Although preliminary experiments show rotated masks lead to slight additional improvements, the computational overhead required to rotate the masks is large. Consequently, we do not consider rotated Gaussian masks in our final experiments.

Review summaries:
This paper presents a novel convolutional operation named FlexConv, to produce high bandwidth convolutional kernels with learnable kernel size at a fixed parameter cost. It is able to generate kernels with large kernel size and model long-term dependencies among elements in a sequence or an image. State-of-the-art performance on both sequential datasets and image datasets demonstrates the effectiveness of the proposed method. In addition, a novel kernel parameterization method is proposed to control the frequency of the generated kernels and avoid aliasing, and hence can well generalize to higher resolution cases which are never seen during training.

The paper proposes a method for learning convolutional filters with trainable size, that builds on top of multiplicative filter networks. The learnable convolutions are called FlexConv, and a network that deploys them is called FlexNet. In order to control the aliasing eventually introduced when learning the convolutional filters, anisotropic Gabor kernels are used within the multiplicative filter networks.

This paper proposed a new convolutional operation with varying-sized kernels. Flexconv constructs their convolutional kernels with the product between the CKConv and Gaussian masks so that the size of the kernels can be learned during training. Also, in the network of Flexconv, MAGNets allow the Flexconv to be used at higher resolutions without aliasing which are unseen resolution. The experiment on several sequential datasets showed that the classification performances from both TCN and ResNet based on FlexConv outperform that of state-of-the-art networks.