Learning Strides in Convolutional Neural Networks

1 INTRODUCTION . Convolutional neural networks ( CNNs ) ( Fukushima , 1980 ; LeCun et al. , 1989 ) have been the most widely used neural architecture across a wide range of tasks , including image classification ( Krizhevsky et al. , 2012 ; He et al. , 2016a ; Huang et al. , 2017 ; Bello et al. , 2021 ) , audio pattern recognition ( Kong et al. , 2020 ) , text classification ( Conneau et al. , 2017 ) , machine translation ( Gehring et al. , 2017 ) and speech recognition ( Amodei et al. , 2016 ; Sercu et al. , 2016 ; Zeghidour et al. , 2018 ) . Convolution layers , which are the building block of CNNs , project input features to a higher-level representation while preserving their resolution . When composed with non-linearities and normalization layers , this allows for learning rich mappings at a constant resolution , e.g . autogressive image synthesis ( van den Oord et al. , 2016 ) . However , many tasks infer high-level low-resolution information ( identity of a speaker ( Muckenhirn et al. , 2018 ) , presence of a face ( Chopra et al. , 2005 ) ) by integrating over low-level , high-resolution measurements ( waveform , pixels ) . This integration requires extracting the right features , discarding irrelevant information over several downsampling steps . To that end , pooling layers and strided convolutions aggressively reduce the resolution of their inputs , providing several benefits . First , they act as a bottleneck that forces features to focus on information relevant to the task at hand . Second , pooling layers such as low-pass filters ( Zhang , 2019 ) improve shift-invariance . Third , a reduced resolution implies a reduced number of floating-point operations and a higher receptive field in the subsequent layers . Pooling layers can usually be decomposed into two basic steps : ( 1 ) computing local statistics densely over the whole input ( 2 ) sub-sampling these statistics by an integer striding factor . Past work has mostly focused on improving ( 1 ) , by proposing better alternatives to max and average pooling that avoid aliasing ( Zhang , 2019 ; Fonseca et al. , 2021 ) , preserve the important local details ( Saeedan et al. , 2018 ) , or adapt to the training data distribution ( Gulcehre et al. , 2014 ; Lee et al. , 2016 ) . Ob- serving that integer strides reduce resolution too quickly ( e.g . a ( 2 , 2 ) striding reduces the output size by 75 % ) , Graham ( 2014 ) proposed fractional max-pooling , that allows for fractional ( i.e . rational ) strides , allowing for integration of more downsampling layers into a network . Similarly , Rippel et al . ( 2015 ) introduce spectral pooling which , by cropping its inputs in the Fourier domain , performs downsampling with fractional strides while emphasizing lower frequencies . While fractional strides give more flexibility in designing downsampling layers , they increase the size of an already gigantic search space . Indeed , as strides are hyperparameters , finding the best combination requires cross-validation or architecture search ( Zoph & Le , 2017 ; Baker et al. , 2017 ; Tan et al. , 2019 ) , which rapidly become infeasible as the number of configurations grows exponentially with the number of downsampling layers . This led Zoph & Le ( 2017 ) not to search for strides in most of their experiments . Talebi & Milanfar ( 2021 ) and Jin et al . ( 2021 ) proposed a neural network that learns a resizing function for natural images , but the scaling factor ( i.e . the stride ) still required cross-validation . Thus , the nature of strides as hyperparameters — rather than trainable parameters — hinders the discovery of convolutional architectures and learning strides by backpropagation would unlock a virtually infinite search space . In this work , we introduce DiffStride , the first downsampling layer that learns its strides jointly with the rest of the network . Inspired by Rippel et al . ( 2015 ) , DiffStride casts downsampling in the spatial domain as cropping in the frequency domain . However , and unlike Rippel et al . ( 2015 ) , rather than cropping with a fixed bounding box controlled by a striding hyperparameter , DiffStride learns the size of its cropping box by backpropagation . To do so , we propose a 2D version of an attention window with learnable size proposed by Sukhbaatar et al . ( 2019 ) for language modeling . On five audio classification tasks , using DiffStride as a drop-in replacement to strided convolutions improves performance overall while providing interpretability on the optimal per-task receptive field . By integrating DiffStride into a ResNet-18 ( He et al. , 2016a ) , we show on CIFAR ( Krizhevsky et al. , 2009 ) and ImageNet ( Deng et al. , 2009 ) that even when initializing strides randomly , our model converges to the best performance obtained with the properly cross-validated strides of He et al . ( 2016a ) . Moreover , casting strides as learnable parameters allows us to propose a regularization that directly minimizes computation and memory usage . 2 METHODS . We first provide background on spatial and spectral pooling , and propose DiffStride for learning strides of downsampling layers . We focus on 2D CNNs since they are generic enough to be used for image ( LeCun et al. , 1989 ; Krizhevsky et al. , 2012 ; He et al. , 2016a ) and audio ( Amodei et al. , 2016 ; Kong et al. , 2020 ) processing ( taking time-frequency representations as inputs ) . However , these methods are equally applicable to the 1D ( e.g . time-series ) and 3D ( e.g . video ) cases . 2.1 NOTATIONS . Let x ∈ RH×W , its Discrete Fourier Transform ( DFT ) y = F ( x ) ∈ CH×W is obtained through the decomposition on a fixed set of basis filters ( Lyons , 2004 ) : F ( x ) mn = 1√ HW H−1∑ h=0 W−1∑ w=0 xhwe −2πi ( mhH + nw W ) , ∀m ∈ { 0 , . . . , H − 1 } , ∀n ∈ { 0 , . . . , W − 1 } . ( 1 ) The DFT transformation is linear and its inverse is given by its conjugate F ( . ) −1 = F ( . ) ∗ . The Fourier transform of a real-valued signal x ∈ RH×W being conjugate symmetric ( Hermitiansymmetry ) , we can reconstruct x from the positive half frequencies for the width dimension and omit the negative frequencies ( ymn = y∗ ( H−m ) modH , ( W−n ) modW ) . In addition , the DFT and its inverse are differentiable with regard to their inputs and the derivative of the DFT ( resp . inverse DFT ) is its conjugate linear operator , i.e . the inverse DFT ( resp . DFT ) . More formally , if we consider L : CH×W −→ R as a loss taking as input the Fourier representation y , we can compute the gradient of L with regard to x , by using the inverse DFT : x ∈ RH×W , y = F ( x ) , ∂L ∂x = F∗ ( ∂L ∂y ) = F−1 ( ∂L ∂y ) . ( 2 ) We denote by L the total number of convolution layers in a CNN architecture and each layer is indexed by l. The ◦ symbol represents the element-wise product between two tensors , b.c is the floor operation and ⊗ the outer product between two vectors . S represents the stride parameters , and sg is the stop gradient operator ( Bengio et al. , 2013 ; Yin et al. , 2019 ) , defined has the identity function during forward pass and with zero partial derivatives . 2.2 DOWNSAMPLING IN CONVOLUTIONAL NEURAL NETWORKS . A basic mechanism for downsampling representations in a CNN is strided convolutions which jointly convolve inputs and finite impulse response filters and downsample the output . Alternatively , one can disentangle both operations by first applying a non-strided convolution followed by a pooling operation that computes local statistics ( e.g . using an average , max ( Boureau et al. , 2010 ) ) before downsampling . In both settings , downsampling does not benefit from the global structure of its inputs and can discard important information ( Hinton , 2014 ; Saeedan et al. , 2018 ) . Moreover , and as observed by Graham ( 2014 ) , the integer nature of strides only allows for drastic reductions in resolution : a 2D-convolution with strides S = ( 2 , 2 ) reduces the dimension of its inputs by 75 % . Furthermore , stride configurations are cumbersome to explore as the number of stride combinations grows exponentially with the number of downsampling layers . This means that cross-validation can only explore a limited subset of the stride hyperparameter configurations . This limitation is likely to translate into lower performance , as Section 3.2 shows that an inappropriate choice of strides for a ResNet-18 architecture can account for a drop of > 18 % in accuracy on CIFAR-100 . 2.3 SPECTRAL POOLING . Energy of natural signals is typically not uniformly distributed in the frequency domain , with signals such as sounds ( Singh & Theunissen , 2003 ) , images ( Ruderman , 1994 ) and surfaces ( Kuroki et al. , 2018 ) concentrating most of the information in the lower frequencies . Rippel et al . ( 2015 ) build on this observation to introduce spectral pooling which alleviates the loss of information of spatial pooling , while enabling fractional downsizing factors . Spectral pooling also preserves low frequencies without aliasing , a known weakness of spatial/temporal convnets ( Zhang , 2019 ; Ribeiro & Schön , 2021 ) . We consider an input x ∈ RH×W and strides S = ( Sh , Sw ) ∈ [ 1 , H ) × [ 1 , W ) . First , the DFT is computed y = F ( x ) ∈ CH×W and for simplicity we assume that the center of this matrix is the DC component — the zero frequency . Then , a bounding box of size b HSh c × b W Sw c crops this matrix around its center to produce ỹ ∈ Cb H Sh c×b WSw c. Finally , this output is brought back to the spatial domain with an inverse DFT : x̃ = F−1 ( ỹ ) ∈ Rb H Sh c×b WSw c. In practice , x is typically a multichannel input ( i.e . x ∈ RH×W×C ) and the same cropping is applied to all channels . Moreover , since x is real-valued and thanks to Hermitian symmetry ( see Section 2.1 for more details ) , only the positive half of the DFT coefficients are computed , which allows saving computation and memory while ensuring that the output x̃ remains real-valued . Unlike spatial pooling that requires integer strides , spectral pooling only requires integer output dimensions , which allows for much more fine-grained downsizing . Moreover , spectral pooling acts as a low-pass filter over the entire input , only keeping the lower frequencies i.e . the most relevant information in general and avoiding aliasing ( Zhang , 2019 ) . However , and similarly to spatial pooling , spectral pooling is differentiable with respect to its inputs but not with respect to its strides . Thus , one still needs to provide S as hyperparameters for each downsampling layer . In this case , the search space is even bigger than with spatial pooling since strides are not constrained to integer values anymore .

Summary: This paper proposes an approach to learning optimal striding parameters in convolutional networks. The proposed approach, DiffStride, is a downsampling layer that builds on spectral pooling to allow for integer output dimensions, but arbitrary strides by cropping in the Fourier domain. Unlike spectral pooling, DiffStride relaxes the parameters of the cropping mask to be differentiable, and using the stop-gradient operator in the cropping. Results show that the proposed approach can work well as a drop-in replacement for normally fixed pooling layers. It is also demonstrated that from different random initialisations a variety of different pooling approaches can be learned that acheive similar performance. Use of a regularisation term the attempts to encourage time and space efficiency reduces this variability and allows accuracy to be traded off.