Learning Strides in Convolutional Neural Networks

1 INTRODUCTION . Convolutional neural networks ( CNNs ) ( Fukushima , 1980 ; LeCun et al. , 1989 ) have been the most widely used neural architecture across a wide range of tasks , including image classification ( Krizhevsky et al. , 2012 ; He et al. , 2016a ; Huang et al. , 2017 ; Bello et al. , 2021 ) , audio pattern recognition ( Kong et al. , 2020 ) , text classification ( Conneau et al. , 2017 ) , machine translation ( Gehring et al. , 2017 ) and speech recognition ( Amodei et al. , 2016 ; Sercu et al. , 2016 ; Zeghidour et al. , 2018 ) . Convolution layers , which are the building block of CNNs , project input features to a higher-level representation while preserving their resolution . When composed with non-linearities and normalization layers , this allows for learning rich mappings at a constant resolution , e.g . autogressive image synthesis ( van den Oord et al. , 2016 ) . However , many tasks infer high-level low-resolution information ( identity of a speaker ( Muckenhirn et al. , 2018 ) , presence of a face ( Chopra et al. , 2005 ) ) by integrating over low-level , high-resolution measurements ( waveform , pixels ) . This integration requires extracting the right features , discarding irrelevant information over several downsampling steps . To that end , pooling layers and strided convolutions aggressively reduce the resolution of their inputs , providing several benefits . First , they act as a bottleneck that forces features to focus on information relevant to the task at hand . Second , pooling layers such as low-pass filters ( Zhang , 2019 ) improve shift-invariance . Third , a reduced resolution implies a reduced number of floating-point operations and a higher receptive field in the subsequent layers . Pooling layers can usually be decomposed into two basic steps : ( 1 ) computing local statistics densely over the whole input ( 2 ) sub-sampling these statistics by an integer striding factor . Past work has mostly focused on improving ( 1 ) , by proposing better alternatives to max and average pooling that avoid aliasing ( Zhang , 2019 ; Fonseca et al. , 2021 ) , preserve the important local details ( Saeedan et al. , 2018 ) , or adapt to the training data distribution ( Gulcehre et al. , 2014 ; Lee et al. , 2016 ) . Ob- serving that integer strides reduce resolution too quickly ( e.g . a ( 2 , 2 ) striding reduces the output size by 75 % ) , Graham ( 2014 ) proposed fractional max-pooling , that allows for fractional ( i.e . rational ) strides , allowing for integration of more downsampling layers into a network . Similarly , Rippel et al . ( 2015 ) introduce spectral pooling which , by cropping its inputs in the Fourier domain , performs downsampling with fractional strides while emphasizing lower frequencies . While fractional strides give more flexibility in designing downsampling layers , they increase the size of an already gigantic search space . Indeed , as strides are hyperparameters , finding the best combination requires cross-validation or architecture search ( Zoph & Le , 2017 ; Baker et al. , 2017 ; Tan et al. , 2019 ) , which rapidly become infeasible as the number of configurations grows exponentially with the number of downsampling layers . This led Zoph & Le ( 2017 ) not to search for strides in most of their experiments . Talebi & Milanfar ( 2021 ) and Jin et al . ( 2021 ) proposed a neural network that learns a resizing function for natural images , but the scaling factor ( i.e . the stride ) still required cross-validation . Thus , the nature of strides as hyperparameters — rather than trainable parameters — hinders the discovery of convolutional architectures and learning strides by backpropagation would unlock a virtually infinite search space . In this work , we introduce DiffStride , the first downsampling layer that learns its strides jointly with the rest of the network . Inspired by Rippel et al . ( 2015 ) , DiffStride casts downsampling in the spatial domain as cropping in the frequency domain . However , and unlike Rippel et al . ( 2015 ) , rather than cropping with a fixed bounding box controlled by a striding hyperparameter , DiffStride learns the size of its cropping box by backpropagation . To do so , we propose a 2D version of an attention window with learnable size proposed by Sukhbaatar et al . ( 2019 ) for language modeling . On five audio classification tasks , using DiffStride as a drop-in replacement to strided convolutions improves performance overall while providing interpretability on the optimal per-task receptive field . By integrating DiffStride into a ResNet-18 ( He et al. , 2016a ) , we show on CIFAR ( Krizhevsky et al. , 2009 ) and ImageNet ( Deng et al. , 2009 ) that even when initializing strides randomly , our model converges to the best performance obtained with the properly cross-validated strides of He et al . ( 2016a ) . Moreover , casting strides as learnable parameters allows us to propose a regularization that directly minimizes computation and memory usage . 2 METHODS . We first provide background on spatial and spectral pooling , and propose DiffStride for learning strides of downsampling layers . We focus on 2D CNNs since they are generic enough to be used for image ( LeCun et al. , 1989 ; Krizhevsky et al. , 2012 ; He et al. , 2016a ) and audio ( Amodei et al. , 2016 ; Kong et al. , 2020 ) processing ( taking time-frequency representations as inputs ) . However , these methods are equally applicable to the 1D ( e.g . time-series ) and 3D ( e.g . video ) cases . 2.1 NOTATIONS . Let x ∈ RH×W , its Discrete Fourier Transform ( DFT ) y = F ( x ) ∈ CH×W is obtained through the decomposition on a fixed set of basis filters ( Lyons , 2004 ) : F ( x ) mn = 1√ HW H−1∑ h=0 W−1∑ w=0 xhwe −2πi ( mhH + nw W ) , ∀m ∈ { 0 , . . . , H − 1 } , ∀n ∈ { 0 , . . . , W − 1 } . ( 1 ) The DFT transformation is linear and its inverse is given by its conjugate F ( . ) −1 = F ( . ) ∗ . The Fourier transform of a real-valued signal x ∈ RH×W being conjugate symmetric ( Hermitiansymmetry ) , we can reconstruct x from the positive half frequencies for the width dimension and omit the negative frequencies ( ymn = y∗ ( H−m ) modH , ( W−n ) modW ) . In addition , the DFT and its inverse are differentiable with regard to their inputs and the derivative of the DFT ( resp . inverse DFT ) is its conjugate linear operator , i.e . the inverse DFT ( resp . DFT ) . More formally , if we consider L : CH×W −→ R as a loss taking as input the Fourier representation y , we can compute the gradient of L with regard to x , by using the inverse DFT : x ∈ RH×W , y = F ( x ) , ∂L ∂x = F∗ ( ∂L ∂y ) = F−1 ( ∂L ∂y ) . ( 2 ) We denote by L the total number of convolution layers in a CNN architecture and each layer is indexed by l. The ◦ symbol represents the element-wise product between two tensors , b.c is the floor operation and ⊗ the outer product between two vectors . S represents the stride parameters , and sg is the stop gradient operator ( Bengio et al. , 2013 ; Yin et al. , 2019 ) , defined has the identity function during forward pass and with zero partial derivatives . 2.2 DOWNSAMPLING IN CONVOLUTIONAL NEURAL NETWORKS . A basic mechanism for downsampling representations in a CNN is strided convolutions which jointly convolve inputs and finite impulse response filters and downsample the output . Alternatively , one can disentangle both operations by first applying a non-strided convolution followed by a pooling operation that computes local statistics ( e.g . using an average , max ( Boureau et al. , 2010 ) ) before downsampling . In both settings , downsampling does not benefit from the global structure of its inputs and can discard important information ( Hinton , 2014 ; Saeedan et al. , 2018 ) . Moreover , and as observed by Graham ( 2014 ) , the integer nature of strides only allows for drastic reductions in resolution : a 2D-convolution with strides S = ( 2 , 2 ) reduces the dimension of its inputs by 75 % . Furthermore , stride configurations are cumbersome to explore as the number of stride combinations grows exponentially with the number of downsampling layers . This means that cross-validation can only explore a limited subset of the stride hyperparameter configurations . This limitation is likely to translate into lower performance , as Section 3.2 shows that an inappropriate choice of strides for a ResNet-18 architecture can account for a drop of > 18 % in accuracy on CIFAR-100 . 2.3 SPECTRAL POOLING . Energy of natural signals is typically not uniformly distributed in the frequency domain , with signals such as sounds ( Singh & Theunissen , 2003 ) , images ( Ruderman , 1994 ) and surfaces ( Kuroki et al. , 2018 ) concentrating most of the information in the lower frequencies . Rippel et al . ( 2015 ) build on this observation to introduce spectral pooling which alleviates the loss of information of spatial pooling , while enabling fractional downsizing factors . Spectral pooling also preserves low frequencies without aliasing , a known weakness of spatial/temporal convnets ( Zhang , 2019 ; Ribeiro & Schön , 2021 ) . We consider an input x ∈ RH×W and strides S = ( Sh , Sw ) ∈ [ 1 , H ) × [ 1 , W ) . First , the DFT is computed y = F ( x ) ∈ CH×W and for simplicity we assume that the center of this matrix is the DC component — the zero frequency . Then , a bounding box of size b HSh c × b W Sw c crops this matrix around its center to produce ỹ ∈ Cb H Sh c×b WSw c. Finally , this output is brought back to the spatial domain with an inverse DFT : x̃ = F−1 ( ỹ ) ∈ Rb H Sh c×b WSw c. In practice , x is typically a multichannel input ( i.e . x ∈ RH×W×C ) and the same cropping is applied to all channels . Moreover , since x is real-valued and thanks to Hermitian symmetry ( see Section 2.1 for more details ) , only the positive half of the DFT coefficients are computed , which allows saving computation and memory while ensuring that the output x̃ remains real-valued . Unlike spatial pooling that requires integer strides , spectral pooling only requires integer output dimensions , which allows for much more fine-grained downsizing . Moreover , spectral pooling acts as a low-pass filter over the entire input , only keeping the lower frequencies i.e . the most relevant information in general and avoiding aliasing ( Zhang , 2019 ) . However , and similarly to spatial pooling , spectral pooling is differentiable with respect to its inputs but not with respect to its strides . Thus , one still needs to provide S as hyperparameters for each downsampling layer . In this case , the search space is even bigger than with spatial pooling since strides are not constrained to integer values anymore .

Summary: In this paper, the authors propose DiffStride, a technique for learning the stride of downsampling operations in neural networks by gradient descent. Specifically, similarily to SpectralPool, the feature map is transformed to the frequency domain by a Discrete Fourier Transform, where it is then cropped according to learnable parameters. The authors envision a simple soft-relaxation of the cropping in order to allow the gradient to flow towards cropping parameters. Performances against both standard and random stride policies are reported on a number of datasets, both for audio and image recognition. Moreover, the authors introduce a regularization objective that penalizes the use of small strides, in the interest of encouraging downsampling for improving computational and memory cost.