FlexConv: Continuous Kernel Convolutions With Differentiable Kernel Sizes

1 INTRODUCTION . The kernel size of a convolutional layer defines the region from which features are computed , and is a crucial choice in their design . Commonly , small kernels ( up to 7px ) are used almost exclusively and are combined with pooling to model long term dependencies ( Simonyan & Zisserman , 2014 ; Szegedy et al. , 2015 ; He et al. , 2016 ; Tan & Le , 2019 ) . Recent works indicate , however , that CNNs benefit from using convolutional kernels ( i ) of varying size at different layers ( Pintea et al. , 2021 ; Tomen et al. , 2021 ) , and ( ii ) at the same resolution of the data ( Peng et al. , 2017 ; Cordonnier et al. , 2019 ; Romero et al. , 2021 ) . Unfortunately , most CNNs represent convolutional kernels as tensors of discrete weights and their size must be fixed prior to training . This makes exploring different kernel sizes at different layers difficult and time-consuming due to ( i ) the large search space , and ( ii ) the large number of weights required to construct large kernels . A more efficient way to tune different kernel sizes at different layers is to learn them during training . Existing methods define a discrete weighted set of basis functions , e.g. , shifted Delta-Diracs ( Fig . 2b , Dai et al . ( 2017 ) ) or Gaussian functions ( Fig . 2c , Jacobsen et al . ( 2016 ) ; Shelhamer et al . ( 2019 ) ; Pintea et al . ( 2021 ) ) . During training they learn dilation factors over the basis functions to increase the kernel size , which crucially limits the bandwidth of the resulting kernels . In this work , we present the Flexible Size Continuous Kernel Convolution ( FlexConv ) , a convolutional layer able to learn high bandwidth convolutional kernels of varying size during training ( Fig . 1 ) . Instead of using discrete weights , we provide a continuous parameterization of convolutional kernels via a small neural network ( Romero et al. , 2021 ) . This parameterization allows us to model continuous functions of arbitrary size with a fixed number of parameters . By multiplying the response of the neural network with a Gaussian mask , the size of the kernel can be learned during training ( Fig . 2a ) . This allows us to produce detailed kernels of small sizes ( Fig . 3 ) , and tune kernel sizes efficiently . FlexConvs can be deployed at higher resolutions than those observed during training , simply by using a more densely sampled grid of kernel indices . However , the high bandwidth of the kernel can lead FlexConv to learn kernels that show aliasing at higher resolutions , if the kernel bandwidth exceeds the Nyquist frequency . To solve this problem , we propose to parameterize convolutional kernels as Multiplicative Anisotropic Gabor Networks ( MAGNets ) . MAGNets are a new class of Multiplicative Filter Networks ( Fathony et al. , 2021 ) that allows us to analyze and control the frequency spectrum of the generated kernels . We use this analysis to regularize FlexConv against aliasing . With this regularization , FlexConvs can be directly deployed at higher resolutions with minimal accuracy loss . Furthermore , MAGNets provide higher descriptive power and faster convergence speed than existing continuous kernel parameterizations ( Schütt et al. , 2017 ; Finzi et al. , 2020 ; Romero et al. , 2021 ) . This leads to important improvements in classification accuracy ( Sec . 4 ) . Our experiments show that CNNs with FlexConvs , coined FlexNets , achieve state-of-the-art across several sequential datasets , match performance of recent works with learnable kernel sizes with less compute , and are competitive with much deeper ResNets ( He et al. , 2016 ) when applied on image benchmark datasets . Thanks to the ability of FlexConvs to generalize across resolutions , FlexNets can be efficiently trained at low-resolution to save compute , e.g. , 16 × 16 CIFAR images , and be deployed on the original data resolution with marginal accuracy loss , e.g. , 32 × 32 CIFAR images . In summary , our contributions are : • We introduce the Flexible Size Continuous Kernel Convolution ( FlexConv ) , a convolution operation able to learn high bandwidth convolutional kernels of varying size end-to-end . • Our proposed Multiplicative Anisotropic Gabor Networks ( MAGNets ) allow for analytic control of the properties of the generated kernels . This property allows us to construct analytic alias-free convolutional kernels that generalize to higher resolutions , and to train FlexNets at low resolution and deploy them at higher resolutions . Moreover , MAGNets show higher descriptive power and faster convergence speed than existing kernel parameterizations . • CNN architectures with FlexConvs ( FlexNets ) obtain state-of-the-art across several sequential datasets , and match recent works with learnable kernel size on CIFAR-10 with less compute . 2 RELATED WORK . Adaptive kernel sizes . Adaptive kernel sizes have been proposed via learnable pixel-wise offsets ( Dai et al. , 2017 ) , learnable padding operations ( Han et al. , 2018 ) , learnable dilated Gaussian functions ( Shelhamer et al. , 2019 ; Xiong et al. , 2020 ; Tabernik et al. , 2020 ; Nguyen , 2020 ) and scalable Gaussian derivative filters ( Pintea et al. , 2021 ; Tomen et al. , 2021 ; Lindeberg , 2021 ) . These approaches either dilate discrete kernels ( Fig . 2b ) , or use discrete weights on dilated basis functions ( Fig . 2c ) . The ability of these methods to learn the kernel size depends on dilation , which crucially limits the bandwidth of the resulting kernels . In contrast , FlexConvs are able to construct high bandwidth convolutional kernels of varying size , and with a fixed parameter count . Larger kernels are obtained simply by passing a larger vector of positions to the neural network parameterizing the kernel ( Fig . 1 ) . Continuous kernel convolutions . Discrete convolutional kernel parameterizations assign an independent weight to each specific position in the kernel . Continuous convolutional kernels , on the other hand , view convolutional kernels as continuous functions parameterized via a small neural network MLPψ ∶ RD → RNout×Nin , with D the data dimensionality . This defines a convolutional kernel for which arbitrary input positions can be queried . Continuous kernels have primarily been used to handle irregularly-sampled data locally , e.g. , molecular data ( Simonovsky & Komodakis , 2017 ; Schütt et al. , 2017 ) and point-clouds ( Thomas et al. , 2018 ; Wang et al. , 2018 ; Shi et al. , 2019 ) . Recently , Romero et al . ( 2021 ) introduced the Continuous Kernel Convolution ( CKConv ) as a tool to model long-term dependencies . CKConv uses a continuous kernel parameterization to construct convolutional kernels as big as the input signal with a constant parameter cost . Contrarily , FlexConvs jointly learn the convolutional kernel as well as its size . This leads to important advantages in terms of expressivity ( Fig . 3 ) , convergence speed and compute costs of the operation . Implicit neural representations . Parameterizing a convolutional kernel via a neural network can be seen as learning an implicit neural representation of the underlying convolutional kernel ( Romero et al. , 2021 ) . Implicit neural representations construct continuous data representations by encoding data in the weights of a neural network ( Park et al. , 2019 ; Sitzmann et al. , 2020 ; Fathony et al. , 2021 ) . We replace the SIREN ( Sitzmann et al. , 2020 ) kernel parameterization used in Romero et al . ( 2021 ) by our Multiplicative Anisotropic Gabor Networks : a new class of Multiplicative Filter Networks ( Fathony et al. , 2021 ) . MFNs allow for analytic control of the resulting representations , and allow us to construct analytic alias-free convolutional kernels . The higher expressivity and convergence speed of MAGNets lead to accuracy improvements in CNNs using them as kernel parameterization . 3 METHOD . In this section , we introduce our approach . First , we introduce FlexConv and the Gaussian mask . Next , we introduce our Multiplicative Anisotropic Gabor Networks ( MAGNets ) and provide a description of our regularization technique used to control the spectral components of the generated kernel . 3.1 FLEXIBLE SIZE CONTINUOUS KERNEL CONVOLUTION ( FLEXCONV ) . To learn the kernel size during training , FlexConvs define their convolutional kernels ψ as the product of the output of a neural network MLPψ with a Gaussian mask of local support . The neural network MLPψ parameterizes the kernel , and the Gaussian mask parameterizes its size ( Fig . 1 ) . Anisotropic Gaussian mask . Let G ( x ; µX , σ2X ) ∶= exp { − 12σ −2 X ( x − µX ) 2 } be a Gaussian function parameterized by a mean-variance tuple ( µX , σ2X ) . The anisotropic Gaussian mask is defined as : wgauss ( x , y ; { µX , σ2X , µY , σ2Y } ) = G ( x ; µX , σ2X ) G ( y ; µY , σ2Y ) . ( 1 ) By learning ( µX , σ2X ) and ( µY , σ2Y ) independently , anisotropic non-centered windows can be learned . 3.2 MULTIPLICATIVE ANISOTROPIC GABOR NETWORKS ( MAGNETS ) . In this section , we formalize our proposed parameterization for the kernel MLPψ . We start by introducing Multiplicative Filter Networks ( Fathony et al. , 2021 ) , and present our MAGNets next . Multiplicative Filter Networks ( MFNs ) . Recently , Fathony et al . ( 2021 ) proposed to construct implicit neural representations as the linear combination of exponentially many basis functions g : h ( 1 ) = g ( [ x , y ] ; θ ( 1 ) ) g ∶ R2 → RNhid ( 2 ) h ( l ) = ( W ( l ) h ( l−1 ) + b ( l ) ) ⋅ g ( [ x , y ] ; θ ( l ) ) W ( l ) ∈ RNhid×Nhid , b ( l ) ∈ RNhid ( 3 ) ψ ( x , y ) =W ( L ) h ( L−1 ) + b ( L ) W ( L ) ∈ RN×Nhid , b ( L ) ∈ RN ( 4 ) where { θ ( l ) , W ( l ) , b ( l ) } depict the learnable parameters of the bases and the affine transformations , and N , Nhid depict the number of output and hidden channels , respectively . Depending on the selection of g , MFNs obtain approximations comparable to those of SIRENs ( Sitzmann et al. , 2020 ) with faster convergence rate . The most successful instantiation of MNFs are the Multiplicative Gabor Network ( MGN ) : MFNs constructed with isotropic Gabor functions as basis g ( in Eq . 2 ) : g ( [ x , y ] ; θ ( l ) ) = exp ( − γ ( l ) 2 [ ( x −µ ( l ) ) 2 + ( y −µ ( l ) ) 2 ] ) Sin ( W ( l ) g ⋅ [ x , y ] + b ( l ) g ) , ( 5 ) θ ( l ) = { γ ( l ) ∈ RNhid , µ ( l ) ∈ RNhid , W ( l ) g ∈ RNhid×2 , b ( l ) g ∈ RNhid } . ( 6 ) Note that , by setting N=Nout×Nin , an MFN can parameterize a convolutional kernel with Nin input and Nout output channels . Fathony et al . ( 2021 ) show that MFNs are equivalent to a linear combination of exponentially many basis functions g. This allows us to analytically derive properties of MFN representations , and plays a crucial role in the derivation of alias-free MAGNets ( Sec . 3.3 ) . Multiplicative Anisotropic Gabor Networks ( MAGNets ) . Our MAGNet formulation is based on the observation that isotropic Gabor functions , i.e. , with equal γ for the horizontal and vertical directions , are undesirable as basis for the construction of MFNs . Whenever a frequency is required along a certain direction , an isotropic Gabor function automatically introduces that frequency in both directions . As a result , other bases must counteract this frequency in the direction where the frequency is not required , and thus the capacity of the MFN is not used optimally ( Daugman , 1988 ) . Following the original formulation of the 2D Gabor functions ( Daugman , 1988 ) , we alleviate this limitation by using anisotropic Gabor functions instead : g ( [ x , y ] ; θ ( l ) ) = exp ( − 1 2 [ ( γ ( l ) X ( x −µ ( l ) X ) ) 2 + ( γ ( l ) Y ( y −µ ( l ) Y ) ) 2 ] ) Sin ( W ( l ) g [ x , y ] + b ( l ) g ) ( 7 ) θ ( l ) = { γ ( l ) X ∈ R Nhid , γ ( l ) Y ∈ R Nhid , µ ( l ) X ∈ R Nhid , µ ( l ) Y ∈ R Nhid , W ( l ) g ∈ RNhid×2 , b ( l ) g ∈ RNhid } . ( 8 ) The resulting Multiplicative Anisotropic Gabor Network ( MAGNet ) obtains better control upon frequency components introduced to the approximation , and demonstrates important improvements in terms of descriptive power and convergence speed ( Sec . 4 ) . MAGNet initialization . Fathony et al . ( 2021 ) proposes to initialize MGNs by drawing the size of the Gaussian envelopes , i.e. , the γ ( l ) term , from a Gamma ( α ⋅ L−1 , β ) distribution at every layer l ∈ [ 1 , .. , L − 1 ] . We observe however that this initialization does not provide much variability on the initial extension of the Gaussian envelopes and in fact , most of them cover a large portion of the space at initialization . To stimulate diversity , we initialize the { γ ( l ) X , γ ( l ) Y } terms by a Gamma ( αl −1 , β ) distribution at the l-th layer . We observe that our proposed initialization consistently leads to better accuracy than the initialization of Fathony et al . ( 2021 ) across all tasks considered . ( Sec . 4 ) .

Summary: This paper presents a novel convolutional operation named FlexConv, to produce high bandwidth convolutional kernels with learnable kernel size at a fixed parameter cost. It is able to generate kernels with large kernel size and model long-term dependencies among elements in a sequence or an image. State-of-the-art performance on both sequential datasets and image datasets demonstrates the effectiveness of the proposed method. In addition, a novel kernel parameterization method is proposed to control the frequency of the generated kernels and avoid aliasing, and hence can well generalize to higher resolution cases which are never seen during training.