FlexConv: Continuous Kernel Convolutions With Differentiable Kernel Sizes

1 INTRODUCTION . The kernel size of a convolutional layer defines the region from which features are computed , and is a crucial choice in their design . Commonly , small kernels ( up to 7px ) are used almost exclusively and are combined with pooling to model long term dependencies ( Simonyan & Zisserman , 2014 ; Szegedy et al. , 2015 ; He et al. , 2016 ; Tan & Le , 2019 ) . Recent works indicate , however , that CNNs benefit from using convolutional kernels ( i ) of varying size at different layers ( Pintea et al. , 2021 ; Tomen et al. , 2021 ) , and ( ii ) at the same resolution of the data ( Peng et al. , 2017 ; Cordonnier et al. , 2019 ; Romero et al. , 2021 ) . Unfortunately , most CNNs represent convolutional kernels as tensors of discrete weights and their size must be fixed prior to training . This makes exploring different kernel sizes at different layers difficult and time-consuming due to ( i ) the large search space , and ( ii ) the large number of weights required to construct large kernels . A more efficient way to tune different kernel sizes at different layers is to learn them during training . Existing methods define a discrete weighted set of basis functions , e.g. , shifted Delta-Diracs ( Fig . 2b , Dai et al . ( 2017 ) ) or Gaussian functions ( Fig . 2c , Jacobsen et al . ( 2016 ) ; Shelhamer et al . ( 2019 ) ; Pintea et al . ( 2021 ) ) . During training they learn dilation factors over the basis functions to increase the kernel size , which crucially limits the bandwidth of the resulting kernels . In this work , we present the Flexible Size Continuous Kernel Convolution ( FlexConv ) , a convolutional layer able to learn high bandwidth convolutional kernels of varying size during training ( Fig . 1 ) . Instead of using discrete weights , we provide a continuous parameterization of convolutional kernels via a small neural network ( Romero et al. , 2021 ) . This parameterization allows us to model continuous functions of arbitrary size with a fixed number of parameters . By multiplying the response of the neural network with a Gaussian mask , the size of the kernel can be learned during training ( Fig . 2a ) . This allows us to produce detailed kernels of small sizes ( Fig . 3 ) , and tune kernel sizes efficiently . FlexConvs can be deployed at higher resolutions than those observed during training , simply by using a more densely sampled grid of kernel indices . However , the high bandwidth of the kernel can lead FlexConv to learn kernels that show aliasing at higher resolutions , if the kernel bandwidth exceeds the Nyquist frequency . To solve this problem , we propose to parameterize convolutional kernels as Multiplicative Anisotropic Gabor Networks ( MAGNets ) . MAGNets are a new class of Multiplicative Filter Networks ( Fathony et al. , 2021 ) that allows us to analyze and control the frequency spectrum of the generated kernels . We use this analysis to regularize FlexConv against aliasing . With this regularization , FlexConvs can be directly deployed at higher resolutions with minimal accuracy loss . Furthermore , MAGNets provide higher descriptive power and faster convergence speed than existing continuous kernel parameterizations ( Schütt et al. , 2017 ; Finzi et al. , 2020 ; Romero et al. , 2021 ) . This leads to important improvements in classification accuracy ( Sec . 4 ) . Our experiments show that CNNs with FlexConvs , coined FlexNets , achieve state-of-the-art across several sequential datasets , match performance of recent works with learnable kernel sizes with less compute , and are competitive with much deeper ResNets ( He et al. , 2016 ) when applied on image benchmark datasets . Thanks to the ability of FlexConvs to generalize across resolutions , FlexNets can be efficiently trained at low-resolution to save compute , e.g. , 16 × 16 CIFAR images , and be deployed on the original data resolution with marginal accuracy loss , e.g. , 32 × 32 CIFAR images . In summary , our contributions are : • We introduce the Flexible Size Continuous Kernel Convolution ( FlexConv ) , a convolution operation able to learn high bandwidth convolutional kernels of varying size end-to-end . • Our proposed Multiplicative Anisotropic Gabor Networks ( MAGNets ) allow for analytic control of the properties of the generated kernels . This property allows us to construct analytic alias-free convolutional kernels that generalize to higher resolutions , and to train FlexNets at low resolution and deploy them at higher resolutions . Moreover , MAGNets show higher descriptive power and faster convergence speed than existing kernel parameterizations . • CNN architectures with FlexConvs ( FlexNets ) obtain state-of-the-art across several sequential datasets , and match recent works with learnable kernel size on CIFAR-10 with less compute . 2 RELATED WORK . Adaptive kernel sizes . Adaptive kernel sizes have been proposed via learnable pixel-wise offsets ( Dai et al. , 2017 ) , learnable padding operations ( Han et al. , 2018 ) , learnable dilated Gaussian functions ( Shelhamer et al. , 2019 ; Xiong et al. , 2020 ; Tabernik et al. , 2020 ; Nguyen , 2020 ) and scalable Gaussian derivative filters ( Pintea et al. , 2021 ; Tomen et al. , 2021 ; Lindeberg , 2021 ) . These approaches either dilate discrete kernels ( Fig . 2b ) , or use discrete weights on dilated basis functions ( Fig . 2c ) . The ability of these methods to learn the kernel size depends on dilation , which crucially limits the bandwidth of the resulting kernels . In contrast , FlexConvs are able to construct high bandwidth convolutional kernels of varying size , and with a fixed parameter count . Larger kernels are obtained simply by passing a larger vector of positions to the neural network parameterizing the kernel ( Fig . 1 ) . Continuous kernel convolutions . Discrete convolutional kernel parameterizations assign an independent weight to each specific position in the kernel . Continuous convolutional kernels , on the other hand , view convolutional kernels as continuous functions parameterized via a small neural network MLPψ ∶ RD → RNout×Nin , with D the data dimensionality . This defines a convolutional kernel for which arbitrary input positions can be queried . Continuous kernels have primarily been used to handle irregularly-sampled data locally , e.g. , molecular data ( Simonovsky & Komodakis , 2017 ; Schütt et al. , 2017 ) and point-clouds ( Thomas et al. , 2018 ; Wang et al. , 2018 ; Shi et al. , 2019 ) . Recently , Romero et al . ( 2021 ) introduced the Continuous Kernel Convolution ( CKConv ) as a tool to model long-term dependencies . CKConv uses a continuous kernel parameterization to construct convolutional kernels as big as the input signal with a constant parameter cost . Contrarily , FlexConvs jointly learn the convolutional kernel as well as its size . This leads to important advantages in terms of expressivity ( Fig . 3 ) , convergence speed and compute costs of the operation . Implicit neural representations . Parameterizing a convolutional kernel via a neural network can be seen as learning an implicit neural representation of the underlying convolutional kernel ( Romero et al. , 2021 ) . Implicit neural representations construct continuous data representations by encoding data in the weights of a neural network ( Park et al. , 2019 ; Sitzmann et al. , 2020 ; Fathony et al. , 2021 ) . We replace the SIREN ( Sitzmann et al. , 2020 ) kernel parameterization used in Romero et al . ( 2021 ) by our Multiplicative Anisotropic Gabor Networks : a new class of Multiplicative Filter Networks ( Fathony et al. , 2021 ) . MFNs allow for analytic control of the resulting representations , and allow us to construct analytic alias-free convolutional kernels . The higher expressivity and convergence speed of MAGNets lead to accuracy improvements in CNNs using them as kernel parameterization . 3 METHOD . In this section , we introduce our approach . First , we introduce FlexConv and the Gaussian mask . Next , we introduce our Multiplicative Anisotropic Gabor Networks ( MAGNets ) and provide a description of our regularization technique used to control the spectral components of the generated kernel . 3.1 FLEXIBLE SIZE CONTINUOUS KERNEL CONVOLUTION ( FLEXCONV ) . To learn the kernel size during training , FlexConvs define their convolutional kernels ψ as the product of the output of a neural network MLPψ with a Gaussian mask of local support . The neural network MLPψ parameterizes the kernel , and the Gaussian mask parameterizes its size ( Fig . 1 ) . Anisotropic Gaussian mask . Let G ( x ; µX , σ2X ) ∶= exp { − 12σ −2 X ( x − µX ) 2 } be a Gaussian function parameterized by a mean-variance tuple ( µX , σ2X ) . The anisotropic Gaussian mask is defined as : wgauss ( x , y ; { µX , σ2X , µY , σ2Y } ) = G ( x ; µX , σ2X ) G ( y ; µY , σ2Y ) . ( 1 ) By learning ( µX , σ2X ) and ( µY , σ2Y ) independently , anisotropic non-centered windows can be learned . 3.2 MULTIPLICATIVE ANISOTROPIC GABOR NETWORKS ( MAGNETS ) . In this section , we formalize our proposed parameterization for the kernel MLPψ . We start by introducing Multiplicative Filter Networks ( Fathony et al. , 2021 ) , and present our MAGNets next . Multiplicative Filter Networks ( MFNs ) . Recently , Fathony et al . ( 2021 ) proposed to construct implicit neural representations as the linear combination of exponentially many basis functions g : h ( 1 ) = g ( [ x , y ] ; θ ( 1 ) ) g ∶ R2 → RNhid ( 2 ) h ( l ) = ( W ( l ) h ( l−1 ) + b ( l ) ) ⋅ g ( [ x , y ] ; θ ( l ) ) W ( l ) ∈ RNhid×Nhid , b ( l ) ∈ RNhid ( 3 ) ψ ( x , y ) =W ( L ) h ( L−1 ) + b ( L ) W ( L ) ∈ RN×Nhid , b ( L ) ∈ RN ( 4 ) where { θ ( l ) , W ( l ) , b ( l ) } depict the learnable parameters of the bases and the affine transformations , and N , Nhid depict the number of output and hidden channels , respectively . Depending on the selection of g , MFNs obtain approximations comparable to those of SIRENs ( Sitzmann et al. , 2020 ) with faster convergence rate . The most successful instantiation of MNFs are the Multiplicative Gabor Network ( MGN ) : MFNs constructed with isotropic Gabor functions as basis g ( in Eq . 2 ) : g ( [ x , y ] ; θ ( l ) ) = exp ( − γ ( l ) 2 [ ( x −µ ( l ) ) 2 + ( y −µ ( l ) ) 2 ] ) Sin ( W ( l ) g ⋅ [ x , y ] + b ( l ) g ) , ( 5 ) θ ( l ) = { γ ( l ) ∈ RNhid , µ ( l ) ∈ RNhid , W ( l ) g ∈ RNhid×2 , b ( l ) g ∈ RNhid } . ( 6 ) Note that , by setting N=Nout×Nin , an MFN can parameterize a convolutional kernel with Nin input and Nout output channels . Fathony et al . ( 2021 ) show that MFNs are equivalent to a linear combination of exponentially many basis functions g. This allows us to analytically derive properties of MFN representations , and plays a crucial role in the derivation of alias-free MAGNets ( Sec . 3.3 ) . Multiplicative Anisotropic Gabor Networks ( MAGNets ) . Our MAGNet formulation is based on the observation that isotropic Gabor functions , i.e. , with equal γ for the horizontal and vertical directions , are undesirable as basis for the construction of MFNs . Whenever a frequency is required along a certain direction , an isotropic Gabor function automatically introduces that frequency in both directions . As a result , other bases must counteract this frequency in the direction where the frequency is not required , and thus the capacity of the MFN is not used optimally ( Daugman , 1988 ) . Following the original formulation of the 2D Gabor functions ( Daugman , 1988 ) , we alleviate this limitation by using anisotropic Gabor functions instead : g ( [ x , y ] ; θ ( l ) ) = exp ( − 1 2 [ ( γ ( l ) X ( x −µ ( l ) X ) ) 2 + ( γ ( l ) Y ( y −µ ( l ) Y ) ) 2 ] ) Sin ( W ( l ) g [ x , y ] + b ( l ) g ) ( 7 ) θ ( l ) = { γ ( l ) X ∈ R Nhid , γ ( l ) Y ∈ R Nhid , µ ( l ) X ∈ R Nhid , µ ( l ) Y ∈ R Nhid , W ( l ) g ∈ RNhid×2 , b ( l ) g ∈ RNhid } . ( 8 ) The resulting Multiplicative Anisotropic Gabor Network ( MAGNet ) obtains better control upon frequency components introduced to the approximation , and demonstrates important improvements in terms of descriptive power and convergence speed ( Sec . 4 ) . MAGNet initialization . Fathony et al . ( 2021 ) proposes to initialize MGNs by drawing the size of the Gaussian envelopes , i.e. , the γ ( l ) term , from a Gamma ( α ⋅ L−1 , β ) distribution at every layer l ∈ [ 1 , .. , L − 1 ] . We observe however that this initialization does not provide much variability on the initial extension of the Gaussian envelopes and in fact , most of them cover a large portion of the space at initialization . To stimulate diversity , we initialize the { γ ( l ) X , γ ( l ) Y } terms by a Gamma ( αl −1 , β ) distribution at the l-th layer . We observe that our proposed initialization consistently leads to better accuracy than the initialization of Fathony et al . ( 2021 ) across all tasks considered . ( Sec . 4 ) .

Summary: The paper proposes a method for learning convolutional filters with trainable size, that builds on top of multiplicative filter networks. The learnable convolutions are called FlexConv, and a network that deploys them is called FlexNet. In order to control the aliasing eventually introduced when learning the convolutional filters, anisotropic Gabor kernels are used within the multiplicative filter networks.