Learning Strides in Convolutional Neural Networks
Authors: Rachid Riad
Olivier Teboul
David Grangier
Neil Zeghidour
Venue: ICLR 2022 Oral (2022)
Keywords: Strides
Convolutional neural networks
Downsampling
Spectral representations
Fourier

Abstract: Convolutional neural networks typically contain several downsampling operators, such as strided convolutions or pooling layers, that progressively reduce the resolution of intermediate representations. This provides some shift-invariance while reducing the computational complexity of the whole architecture. A critical hyperparameter of such layers is their stride: the integer factor of downsampling. As strides are not differentiable, finding the best configuration either requires cross-validation or discrete optimization (e.g. architecture search), which rapidly become prohibitive as the search space grows exponentially with the number of downsampling layers. Hence, exploring this search space by gradient descent would allow finding better configurations at a lower computational cost. This work introduces DiffStride, the first downsampling layer with learnable strides. Our layer learns the size of a cropping mask in the Fourier domain, that effectively performs resizing in a differentiable way. Experiments on audio and image classification show the generality and effectiveness of our solution: we use DiffStride as a drop-in replacement to standard downsampling layers and outperform them. In particular, we show that introducing our layer into a ResNet-18 architecture allows keeping consistent high performance on CIFAR10, CIFAR100 and ImageNet even when training starts from poor random stride configurations. Moreover, formulating strides as learnable variables allows us to introduce a regularization term that controls the computational complexity of the architecture. We show how this regularization allows trading off accuracy for efficiency on ImageNet.

Published as a conference paper at ICLR 2022
LEARNING STRIDES IN CONVOLUTIONAL NEURAL NETWORKS
Rachid Riad" Olivier Teboul,, David Grangier & Neil Zeghidour ENSS, INRIA, INSERM, UPEC, PSL. Research University
Google Research
rachid. r riadßens. fr, fteboul, grangier, neilz neilz]@google.com
ABSTRACT
Convolutional neural networks typically contain several downsampling operators, such as strided convolutions or pooling layers, that progressively reduce the reso- lution of intermediate representations. This provides some shift-invariance while reducing the computational complexity of the whole architecture. A critical hy- perparameter of such layers is their stride: the integer factor of downsampling. As strides are not differentiable, finding the best configuration either requires cross- validation discrete optimization (e.g. architecture search), which rapidly be- come prohibitive as the search space grows exponentially with the number of downsampling layers. Hence, exploring this search space by gradient descent would allow finding better configurations at lower computational cost. This work introduces DiffStride, the first downsampling layer with learnable strides. Our layer learns the size of cropping mask in the Fourier domain, that effectively performs resizing in a differentiable way. Experiments on audio and image clas- sification show the generality and effectiveness of our solution: we use DiffStride as a drop-in replacement to standard downsampling layers and outperform them. In particular, we show that introducing our layer into ResNet- 18 architecture al- lows keeping consistent high performance on CIFARIO, 10, CIFAR 100 and ImageNet even when training starts from poor random stride configurations. Moreover, for- mulating strides as learnable variables allows us to introduce a regularization term that controls the computational complexity of the architecture. We show how this regularization allows trading off accuracy for efficiency on ImageNet.
INTRODUCTION
Convolutional neural networks (CNNs) (Fukushima, 1980; LeCun et al., 1989) have been the most widely used neural architecture across a wide range of tasks, including image classification (Krizhevsky et al., 2012: He et al., 2016a; Huang et al., 2017; Bello et al., 2021), audio pattern recog- nition (Kong et al., 2020), text classification (Conneau et al., 2017), machine translation (Gehring et al., 2017) and speech recognition (Amodei et al., 2016; Sercu et al., 2016; Zeghidour et al., 2018). Convolution layers, which the building block of CNNs, project input features to higher-level representation while preserving their resolution. When composed with non-linearities and normal- ization layers, this allows for learning rich mappings at constant resolution, e.g. autogressive image synthesis (van den Oord et al., 2016). However, many tasks infer high-level low-resolution infor- mation (identity of a speaker (Muckenhirn al., 2018), presence of face (Chopra al., 2005)) by integrating over low-level, high-resolution measurements (waveform, pixels). This integration requires extracting the right features, discarding irrelevant information over several downsampling steps. To that end, pooling layers and strided convolutions aggressively reduce the resolution of their inputs, providing several benefits. First, they act as a bottleneck that forces features to focus on in- formation relevant to the task at hand. Second, pooling layers such as low-pass filters (Zhang, 2019) improve shift- infftinvariincee. Third, a reduced resolution implics a reduced number of floating-poin operations and higher receptive field in the subsequent layers.
"This work conducted while interning Google.

---

Published as a conference paper at ICLR 2022
Pooling layers can usually be decomposed into two basic steps: (1) computing local statistics densely over the whole input (2) sub-samlinng these statistics by an integer striding factor. Past work has mostly focused on improving (1), by proposing better alternatives to max and average pooling that avoid aliasing (Zhang, 2019; Fonseca et al., 2021), preserve the important local details (Saeedan al., 2018), or adapt the training data distribution (Gulcehre et al., 2014; Lee et al., 2016). OB- serving that integer strides reduce resolution too quickly (e.g. a (2 striding reduces the output size by 75%). Graham (2014) proposed fractional max-pooling, that allows for fractional (i.e. ra- tional) strides, allowing for integration of more downsampling layers into network. Similarly, Rippel et al. (2015) introduce spectral pooling which, by cropping its inputs in the Fourier domain, performs downsampling with fractional strides while emphasizing lower frequencies.
While fractional strides give more flexibility in designing downsampling layers, they increase the size of an already gigantic search space. Indeed, as strides are hyperparameters, finding the best combination requires vross-validation or architecture search (Zoph & Le, 2017; Baker et al., 2017: Tan et al., 2019), which rapidly become infeasible as the number of configurations grows expo- nentially with the number of downsampling layers. This led Zoph & Le (2017) not to search for strides in most of their experiments. Talebi & Milanfar (2021) and Jin al. (2021) proposed neural network that learns a resizing function for natural images, but the scaling factor (i.e. the stride) still required cross-validation. Thus, the nature of strides as hyperparameters rather than trainable parameters - hinders the discovery of convolutional architectures and learning strides by backpropagation would unlock a virtually infinite search space.
In this work, we introduce DiffStride, the first downsampling layer that learns its strides jointly with the rest of the network. Inspired by Rippel et al. (2015), DiffStride casts downsampling in the spatial domain as cropping in the frequency domain. However, and unlike Rippel et al. (2015), rather than cropping with fixed bounding box controlled by a striding hyperparameter, DiffStride learns the size of its cropping box by backpropagation. To do so, we propose a 2D version of an attention window with learnable size proposed by Sukhbaatar et al. (2019) for language modeling. On five audio classification tasks, using DiffStride as a drop-in replacement to strided convolutions improves performance overall while providing interpretability on the optimal per-task receptive field. By integrating DiffStride into a ResNet-18 (He et al., 2016a), we show CIFAR (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009) that even when initializing strides randomly, our model converges to the best performance obtained with the properly cross-validated strides of He et al. (2016a). Moreover, casting strides as learnable parameters allows us to propose a regularization that directly minimizes computation and memory usage. We release our implementation of DiffStrrde!
METHODS
We first provide background on spatial and spectral pooling, and propose DiffStride for learning strides of downsampling layers. We focus on 2D CNNs since they are generic enough to be used for image (LeCun et al.. 1989; Krizhevsky et al., 2012; He al., 2016a) and audio (Amodei al., 2016; Kong et al., 2020) processing (taking fime-freyuency representations as inputs). However, these methods are equally applicable to the 1D (e.g. time-series) and 3D (e.g. video) cases.
2.1 NOTATIONS
Let x E x W its Discrete Fourier Transform (DFT) y = F(x) E Cxx is obtained through the decomposition on a fixed set of basis filters (Lyons, 2004):
H-1W-1
F(x) VHW
Σ
Σ
Thwe
2nd( mb ax Vm E (O...., H 1), (o....,
S0....,.... W W-1}.
1}.
h=0 00
(1)
The DFT transformation is linear and its inverse is given by its conjugate Fourier transform of a real-valued signal x E RHx being conjugate symmetric (Hermitian- symmetry), we can reconstruct x from the positive half frequencies for the width dimension and omit the negative frequencies (Jmn = yi mjmoddH ( -n)meow).. In addition, the DFT and its in- verse are differentiable with regard to their inputs and the derivative of the DFT (resp. inverse DFT)
= -())-=((()) The
httpps://ii thub. www.oooolooooooooooooooooooooooo e- resea /di ffstride
2

---

Published as a conference paper at ICLR 2022
is its conjugate linear operator, i.c. the inverse DFT (resp. DFT). More formally, if consider C : CM*W - IR as a loss taking as input the Fourier representation y, we can compute the gradient of L with regard to z, by using the inverse DFT:
ac
aL dy
aL
EER y F(z).
dx = F
(2)
Oy
We denote by L the total number of convolution layers in CNN architecture and each layer is indexed by I. The o symbol represents the element-wise product between two tensors, |-] is the floor operation and @ the outer product between two vectors. S represents the stride parameters, and sg is the stop gradient operator (Bengio et al., 2013; Yin et al., 2019), defined has the identity function during forward pass and with zero partial derivatives.
2.2 DOWNSAMPLING IN CONVOLUTIONAL NEURAL NETWORKS
A basic mechanism for downsampling representations in CNN is strided convolutions which jointly convolve inputs and finite impulse response filters and downsample the output. Alterna- tively, one disentangle both operations by first applying non-strided convolution followed by pooling operation that computes local statistics (e.g. using an average, max (Boureau et al., 2010)) before downsampling. In both settings, downsampling does not benefit from the global structure of its inputs and can discard important information (Hinton, 2014; Saeedan et al., 2018). Moreover, and as observed by Graham (2014), the integer nature of strides only allows for drastic reductions in resolution: a -D-convolution with strides S = (2, 2) reduces the dimension of its inputs by 75%. Furthermore, stride configurations are cumbersome explore as the number of stride combinations grows exponentially with the number of downsampling layers. This means that cross-validation can only explore a limited subset of the stride hyperparameter configurations. This limitation is likely to translate into lower performance, Section 3.2 shows that inappropriate choice of strides for ResNet-18 18 architecture can account for drop of >> 18% in accuracy on CIFAR-100.
2.3 SPECTRAL POOLING
Energy of natural signals is typically not uniformly distributed in the frequency domain, with sig- nals such as sounds (Singh & Theunissen, 2003), images (Ruderman, 1994) and surfaces (Kuroki et al., 2018) concentrating most of the information the lower frequencies. Rippel et al. (2015) build on this observation to introduce spectral pooling which alleviates the loss of information of spatial pooling, while enabling fractional downsizing factors. Spectral pooling also preserves low frequencies without aliasing, a known weakness of spatial/temporal convnets (Zhang, 2019; Ribeiro & Schön, 2021).
We consider input x E H&W and strides S = Sw) H) [1, W). First, the DFT is computed y = F(z) E CH and for simplicity we assume that the center of this matrix is the DC component - the zero frequency. Then, a bounding box of size S
H W crops this matrix S
C W. Finally, this output is brought back to the spatial domain with an inverse DFT: ã = F-) E R | In practice, x is typically a multi- channel input (i.e. x B and the same cropping is applied all channels. Moreover, since x is real-valued and thanks to Hermitian symmetry (see Section 2.1 for more details), only the positive half of the DFT coefficients are computed, which allows saving computation and memory while ensuring that the output E remains real-valued.
around its center to produce
Unlike spatial pooling that requires integer strides, spectral pooling only requires integer output di- mensions, which allows for much more fine mme-grained downsizing. Moreover, spectral pooling acts as a low-pass filter over the entire input, only keeping the lower frequencies i.e. the most relevant information in general and avoiding aliasing (Zhang, 2019). However, and similarly to spatial pool- ing, spectral pooling is differentiable with respect to its inputs but not with respect to its strides. Thus, one still needs to provide S hyperparameters for each downsampling layer. In this case, the search space is even bigger than with spatial pooling since strides are not constrained to integer values anymore.
2.4 STRDDE
To address the difficulty of searching stride parameters, we propose DiffStride, a novel downsam- pling layer that allows spectral pooling to learn its strides through backpropagation. To downsample
3

---

Published as a conference paper at ICLR 2022
1M
Ohop
Miropeed
Logond
Strides
In Smoothneee factor
y: Fonzle texadorm
5 Iuverae Fouzie teatafeet
Elenent vase pooduct
Crey Chopping rperabor
W: Window based on staides
Porwaed
maak
Backmard grodieat
X Stup Gradient
Spottal conndtoabee
25,
17, togut thoges
Figure I: DiffStride forward and backward pass, using a single-channee image. We only compute the positive half of DFT coefficients along the horizontal axis due conjugate symmetry. The zoomed frame shows the horizontal mask mask(s.. (s.. (n). Here S = (Sh, S = (2.6,3.1)
E HW DiffSttride performs cropping in the Fourier domain similarly to spectral pooling, How- ever, instead of using a fixed bounding box, DiftStride learns the box size via backpropagation. The learnable box W is parametrized by the shape of the input, smoothness factor R and the strides. We design this mask W the outer product between picted in the lower right corner of Figure 1), one along the horizontal axis and one along the vertical axis. These 1D masks are directly derived from the adaptive attention span introduced by Sukhbaatar et al. (2019) to learn the attention span of self-attention models for natural language processing, Ex- ploiting the conjugate symmetry of the coefficients, we only consider positive frequencies along the horizontal axis, while mirror the vertical mask around frequency are defined as follows:
differentiable ID masking functions (de-
Therefore, the two masks
H
H
mask SS. H. (m) min max
(R
0 m E [0, n),
ml),
(3)
25,
W
W
mask (5.. W. min
max
(R +
1 n E [0,
+
(4)
R
25
2
where S = (Sk, Sw) are the strides and R an hyperparameter that controls the smoothness of the mask. We build the 2D differentiable mask as the outer product between the two ID masks:
W(S,, Sw, H, W, R) ma H.M X mask
We use W in two ways: (1) we apply it to the Fourier representation of the inputs via element- wise product, which performs low-pass filtering (2) we crop the Fourier coefficients where the mask zero (i.e. the output has dimensions H LT, 2 x R
S + R.).
W
The first step is differentiable with respect to strides S, however the cropping operation is not. Therefore, we apply stop gradient operator (Bengio al., 2013) to the mask before cropping. This way, gradients can flow to the strides through the differentiable low-pass filtering operation, but not through the con-differentiable cropping. Finally, the cropped tensor transformed back into the spatial domain using inverse DFT. All these steps summarized by Algorithm and illustrated on a single channel image in the Figure 1.
During training constrain strides multi-channel input (i.e. x E @MXWXC) we learn the same strides S for all channels to ensure uniform spatial dimensions across channels. In spatial and spectral pooling, strides are typically
(Sh, Sw) to remain in [1, H) x [1, W). When is

---

Published as a conference paper at ICLR 2022
Algorithm 1: DiftStride layer
Inputs: : Input E RRHW strides S = (Sh, Sw) E [1, H) x [1, W), smoothness factor R.
Output: Downsampled output ă E ! +2 x 1.w +2 RJ
y
F(x)
Project input to the Fourier domain.
mask - W(S,, Swi H, W, Rt)
b Construct the mask. See Equation 5.
Ymasked
y o mask
> Apply the mask as a low-pass filter. Crop(ymaan Umasked sy mask)) & Crop the tensor with the mask after stopping gradients.
Mcropped
I K
(Veropped)
> Return to the spatial domain.
3x3 Soro
3x3 Saro
Cany
Sel Come Sindwee11. 1)
Simdeem12.2)
Stideemm2..2) 2)
Strcdeen!1 :)
Cifisonde with
DifStrdee
Stridhoo(SS..S..
Strcosm18,
Cony Sbidds--(t.1)
Relu
Sex homy
Strccss(1.11
Reln
Rehn
(a) Residual block with a strided convolution.
(h) Residual block with a shared DiffStride layer.
Figure 2: Comparison side by side of the shortcut blocks in classic ResNet architectures with strided convolutions, and with DifiSttrie that learns the strides of the block.
tied along the spatial axes (i.e. S. = Sa), which we can also do in DiffStride by sharing a single learnable stride on both dimensions. However, our experiments in Section 3 show that learning specific strides for the vertical and horizontal axis is beneficial, not only when processing time- frequency representations of audio, but also more surprisingly- when classifying natural images. Adding an hyperparameter R to each downsampling layer would conflict with the goal of removing strides as hyperparameters. Thus, not only we use single R value for all layers, but we found no significant impact of this choice and all our experiments use using a single 1D) mask allows deriving DiffStride in 1D CNNs, while performing the outer product between three ID masks allows applying DiffStride to 3D inputs.
4. While we focus on the 2D case,
2.4.1 RESIDUAL BLOCK WITH DIFFSTRIDE
Unlike systems that only feed outputs of the 1th layer to the (1 + 1)th (Krizhevsky et al., 2012), ResNets (He et al., 201 6a;b) introduce skip-connections that operate in parallel to the main branch. ResNets stack two types of blocks: (1) identity blocks that maintain the input channel dimension and spatial resolution and (2) shortcut blocks that increase the output channel dimension while reducing the spatial resolution with a strided convolution (see Figure 2a). We integrate DiffStride into these shorteut blocks by replacing strided convolutions by convolutions without strides followed by DiffStride. Besides, sharing DiffStride strides between the main and residual branches ensures that their respective outputs have identical spatial dimensions and can be summed (See Figure 2b).
2.4.2 REGULARIZING COMPUTATION AND MEMORY COST WITH DIFTSTRIDE
The number of activations in network depends on the strides and learning these parameters gives control over the space and time complexity of an architecture in a differentiable manner. This con- trasts with previous work, as measures of complexity such as the number of floating point operations (FLOPs) are typically not differentiable with respect to the parameters of model and searching for efficient architectures is done via high-level exploration (e.g. introducing separable convolutions (Howard et al., 2017)), architecture search (Howard et al., 2019; Tan & Le, 2019) or using continu- ous relaxations of complexity (Paria et al., 2020).
A standard 2D convolution with square kernel of size k2 and C" output channels has computa- tional cost of 222 x x x H x W when operating on x E # XC. Its memory usage- in terms of the number of activations store is C" x H x W. Considering fixed number of channels and kernel size, both the computational complexity and memory usage of a convolution layer are thus linear functions of its input size H W. This illustrates argument made in Section 1 that
5

---

Published as a conference paper at ICLR 2022
downsampling does not only improve performance by discarding irrelevant information, but also reduces the complexity of the upper layers. More importantly, in the context of DifiStride the input size H' x W of layer is determined as follows: H' W - H²- + 2 RJ x W'- st
+2 RJ,
which is differentiable with respect to the strides at the previous layer S'-1. Furthermore, also depends on spatial dimensions at the previous layer H' x W which themselves are a function of 5 By induction over layers, the total computational cost and memory usage are proportional to EEI St x Since in the context of DiffStride the kernel size and number of channels remain constant during training, we can directly regularize our model towards time and space effi ciency by adding the following regularizer to our training loss:
I-L. \= III
(6)
S S
where À is the regularization weight. In Section 3.2, we show that training on ImageNet with different values for À allows us to trade-off accuracy for efficiency in a smooth fashion.
3 EXPERIMENTS
We evaluate DiftStride on eight classification tasks, both on audio and images. For each compari- son, we keep the same architecture and replace strided convolutions by convolutions with no stride followed by DiffStride. To avoid the confounding factor of downsampling in the Fourier domain, we also compare our approach to the spectral pooling of Rippel et al. (2015), which only differs from DiffStride by the fact that its strides are not learnable.
3.1 AUDIO CLASSIFICATION
Experimental setup We perform single-task and multi-task audio classification on 5 tasks: acous- tic scene classification (Heittola et al., 2018), birdsong detection (Stowell et al., 2018), musical in- strumental classification and pitch estimation on the NSynth dataset (Engel et al., 2017) and speech command classification (Warden, 2018). The statistics of the datasets are summarized in Table A. The audio sampled at 16 kHz is decomposed into log-compressed mel-spectrograms with 64 chan- nels, computed with a window of 25 ms every 10 ms.
A 2D-CNN, based on (Tagliasacchi et al., 2019) takes these spectrograms as inputs and alternates blocks of strided convolutions along time ((3 x 1) kernel) and frequency ((1 x 3) kernel). Each strided convolution is followed by ReLU (Glorot et al., 2011) and batch normalization (loffe & Szegedy, 2015). The sequence of channels dimensions is defined as (64, 128, 256, 256, 512, 512) and the strides initialized as ((2, 2), (2. 2), (1, (2, (1, (2, 2) for all downsampling methods. The output of the CNN passes through a global max-pooling and feeds into a single linear classification layer for single-task, and multiple classification layers for multi-task classification. As examples vary in length, we train models on random 1s windows with ADAM (Kingma & Ba, 2015) and a learning rate of 10-4 4 for 1 M batches, with batch size 256. Evaluation is run by splitting full sequences into 1 s non-overlapping windows and averaging the logits over windows.
Results Table summarizes the results for single-task and multi-tast audio classification. In both settings, DiffStride improves over strided convolutions and spectral pooling, with strided convolu- tions only outperforming DiffStride for acoustic scene classification in the single task setting. Table 2 shows the strides learned by the first layer of DifiStride, which downsamples mel-spectrograms along frequency and time axes. Learning allows the strides to deviate from their initialization ((2, 2)) and to adapt to the task at hand. Converting strides to cut-off frequencies shows that the learned strides fall in range showed by behavioral studies and direct neural recordings (Hullett al., 2016; Flinker et al., 2019) to be necessary for e.g. speech intelligibility at 25 Hz (Elliott & Theunissen, 2009). Moreover, DiffStride learns different strides for the time and frequency axes. Table A.7 shows the benefits of learning per-dimension value rather than sharing strides. Another notable phenomenon is the per-task discrepancy on NSynth, with the pitch estimation requiring faster spec- tral modulations (as represented by higher cutt-off frequency along the frequency axis). Finally, multi-task models do not converge to the mean of strides, but rather to a higher value that passes more frequencies not negatively impact individual tasks.

---

Published as a conference paper at ICLR 2022
Setting
Single-task Spectral
Multi-task
Task
Strided Conv.
DiffStrride
Strided Conv.
Spectral
DiffStride
Acoustic scenes
99.1 + 0.2
98.6 + 0.1
98.6 1 0.2
97.7 1 0.4
97.7 = 0.7
97.7 + 0.3
Birdsong detection
78.8 + 0.3
79.7 + 0.3
81.3 0.1
77.3 0.2
77.8 - 0.3
78.6 1 0.5
Music (instrument)
72.6 + 0.3
72.9 1: 0.5
7.0
69.8 + 0.4
70.4 10.400.4 0.4
73.0
0.8
Music (pitch)
91.8
9.01
89.4 9.1
8.7
89.9
= 0.3
Speech commands
0.1
88.5 - 0.3
90.5 + 0.3
83.5 = 0.6
83.9 0.4
86.2 + 0.8
Mean Accuracy
85.0 + 9.3
86.0 + 9.2 88.3 + 8.7|
83.5 +: 10.0
83.5 = 9.6 85.0 + 8.9
Table 1:
Test accuracy (% = sd over 3 runs) for audio classification in the single (one model per task) and multi-tast (one model for all tasks) settings.
Learned Strides
Equivalent cut-off frequencies
Time
Frequency
Time (Hz)
Frequency (Cyc/Mel)
Acoustic scenes
1.89 0.05
1.99 0.03
26.25 + 0.63
0.2448 + 0.009
Birdsong detection
1.91 + 0.02
1.96 0.01
25.83 + 0.36
0.2500 + 0.000
Music (Instrument)
1.29 0.06
2.12 0.01
38.33 + 1.57
0.2292 + 0.009
Music (Pitch)
1.32
1.61 +
37.50 2.72
0.3021 + 0.018
Speech commands
1.97
1.95 0.01
25.00 = 0.00
0.2500 + 0.000
Multi-task model
0.01
1.79 0.03
34.17 = 0.30
0.2708 + 0.0074
Table Learned strides (% + sd over 3 runs) of the first layer for the single and multi-task models. The sampling rate of the input spectrogram being known (10 ms), we can convert the strides to upper cut-off frequencies (i.c. the maximum frequency kept by the sewpass-filter..
3.2 IMAGE CLASSIFICATION
Experimental setup We use the R8 (He et al., 201 architecture, comparing the original strided convolutions (see Figure 2a) to spectral pooling and DifSStride (both as in Figure 2b). We randomly sample 6 striding configurations for the three shortcut blocks of the ResNet-18, each stride being sampled in [1,3], with (2, 2) being the configuration of the original ResNet of He et al. (2016a). The horizontal and vertical strides are initialized equally at start. These random configurations simulate cross-validation of stride configurations to: (1) showcase the sensitivity of the architecture to these hyperparameters, (2) test our hypothesis that Difi'ttride can benefit from learning its strides to recover from a poor initialization. On Imagenet, as inputs are bigger than CIFAR we also allow the first ResNet-18 identity block to learn its strides which are by default.
We first benchmark the three methods on the two CIFAR datasets (Krizhevsky, 2009). CIFARIO consists of 32 x 32 images labeled in 10 classes with 6000 images per class. We use the official split, with 50,000 images for training and 10,000 images for testing. CIFAR100 uses the same images as CIFAR1O, but with a more detailed labelling with 100 classes. We also compare the ResNet- 18 architectures on the ImageNet dataset (Deng et al., 2009), which contains 1,000 classes. The models are trained on the official training split of the Imagenet dataset (1.28M images) and we report our results on the validation set (50k images). Here, we evaluate performance in terms of top-1 and top-5 accuracy. We train on all datasets with stochastic gradient descent (SGD) (Bottou et al., 1998) with learning rate of 0.1, batch size of 256 and a momentum (Qian, 1999) of 0.9. On CIFAR, we train models for 400 epochs dividing the learning rate by 10 at 200 epochs and again by 10 at 300 epochs, with a weight decay of 5.10 For CIFAR, we apply random cropping on the input images and left-right random flipping. On ImageNet, we train with a weight decay of 1.10-3 for 90 epochs, dividing the learning rate by 10 at epochs 30, 60 and 80. We apply random cropping on the input images as in (Szegedy et al., 2015) and left-right random flipping.
Results We report the results on the CIFAR datasets and Imagenet in Tables 3 and 4 respectively, with the accuracy of our baseline ResNet-18 (first row, Strided Conv.) being consistent with pre- vious work (Bianco et al., 2018). First, we observe that strides are indeed critical hyperparameters for the performance of a standard ResNet-18 dropping from 66.8% average to 48.2% between the best and worst configurations. Remarkably, spectral pooling is much more robust bad initializations than strided convolutions, even though
the three datasets, with the accuracy on CIFAR100
7

---

Published as a conference paper at ICLR 2022
CIFARTO
CIFAR100
Init. Strides
Strided Cony.
Spectral
Dift'ttrid
Strided Conv.
Spectral
DiftStride
(2, 2, 2)
91.4 0.2
92.4 + 0.1 92.5
0.1
66.8 0.2
73.7 - 0.1
73.4 = 0.5
(2, 2, 3)
90.5 0.1
92.8 9.2
0.1
0.54
73.7 0.2
73.5 1.55000
(1, 3, 1)
90.0 0.4
91.1
9.1
70.3 0.5
7.2
(3, 1, 3)
0.1
90.9
9.1
0.8
6..4
0.4
(3, 1)
8.1
9.2
0.3
0.2
73.4 +.3
(3, 2)
82.0 0.6
89.2 0.2
92.3
0.1
4.2
66.6 0.5
73.6 0.4
Mean accuracy
87.7 77.7:3444
91.1
92.4
0.2
59.1
1 6.7 70.6 + 2.6 73.5 = 0.3
Table 3: Accuracies (% + sd over 3 runs) on CIFARIO and CIFAR100. First column represents strides at each shortcut block, (2, 2, 2) being the configuration of (He et al., 2016a). For reference, the state-of-the-are on CIFARIO (CIFARI00) 100) (Dosovitskiy et al., 2020) ((Foret al., 2020)) with
an accuracy of 99.5% (96.1%).
3.5
Layers
5
4
25
3.0
5q
3
2.5
51
15-
5
2.0
10
40
60
80
1
5
5W
S/
5
S
sst
ns.
apoch
(a) Trajectories of the different strides (S (5'))E(1.223)) for a single run.
(b) Distribution of learned strides in ResNet-18 for the different initializations.
(c) Distribution of the final global striding factors for the different initializations.
Figure 3: Learning dynamics of DifiSttrid on the CIFAR10 dataset.
its strides are also fixed. However, DiffStride is overall much more robust to poor choices of strides,
converging consistently to a high accuracy on the three datasets, with variance over initializations that is lower by an order of magnitude. This shows that backpropagation allows DiffStride to find a better configuration during training avoiding cross-validation which would require 6,561 experi- ments for testing all combinations of strides in [1, 3] on Imagenet. Tables A.5 and A.6 confirm these observations on the EfficientNet-BO (Tan & Le, 2019) architecture.
Learning dynamics and equivalence classes Figure illustrates the learning dynamics of Diff- Stride on CIFARIO. Figure 3a plots the strides as a function of the epoch for a run with the baseline (2,2,2) configuration as initialization. The strides all deviate from their initialization while con- verging rapidly, with the lower layers keeping more information while higher layers downsample more drastically. Interestingly, we discover equivalence classes: despite converging to the same accuracy (as reported in Table 3) the various initializations yield very diverse strides configurations convergence, both in terms of total striding factor (defined as the product of strides, see Figure 3c) and of repartition of downsampling factors along the architecture (see Figure 3b). We obtain similar conclusions on CIFAR 100 and Imagenet (see Figures A. and A.2). In the non-regularized case, it could seem ie. dropping more information through cropping. highlights that loss optimization is a trade- off between preserving information (no striding, cropping) and downscaling such that the next convolution kernel accesses a wider spatial context.
that minimizing the training loss yields positive stride updates,
Regularizing the complexity The existence of equivalence classes suggests that DiffStride can find more computationally efficient configurations for a same accuracy. We thus train DiffStride on ImageNet using the complexity regularizer defined in Equation 6, with À varying between 0.1 and 10, always initializing strides with the baseline ((1, 1), (2, 2), (2, 2), (2, 2)). Figure 4 plots accuracy versus computational complexity (as measured by the value of the regularization term at convergence) of DiffStride. For comparison, we also plot the models with strided convolutions with the random initializations of Table 4, showing that DiffSttride finds configurations with lower computational cost for the same accuracy. Some of these are quite extreme, e.g. with = 10 model converges to strides ((10.51, 32.23), (1.20, 2.68), (1.20, 2.04), (1.96, 4.53)) for 58.57%
8

---

Published as a conference paper at ICLR 2022
Top-1
Top-5
Init. Strides
Strided Conv.
Spectral
DiftStride
Strided Conv.
Spectral
DiftStride
(1. 2, 2,2) 2)
68.65 + 0.26
69.01 +: 0.19
69.66 + 0.06
88.5 + 0.15
88.48 = 0.02
89.07 + 0.03
(1, 1, 1)
69.79 0.15
69.88 + 0.05
68.22 = 0.07
89.43 0.18
89.15 0.07
88.10 = 0.08
(1, 3, 3)
68.86 0.28
68.63
0.08
69.41 = 0.16
88.64 0.15
88.42 0.01
88.98 0.04
(2, 2, 2, 3)
63.45 0.09
67.16 = 0.17
69.53 0.08
1.00
87.25 0.06
89.05 = 0.05
(2, 3, 1,2)
65.35 0.03
66.35
+ 0.24
69,42 0.06
86.27
86.67 = 0.15
88.91 = 0,05
(3, 3, 2,3)
57.11 0.11
64.44
+ 0.01
69.43 + 0.11
80.42 0.11
85.22
0.09
89.03 + 0.02
Mean accuracy
65.53
4.49 67.58 + 1.88
69.28 + 0.50 86.39 + 3.15 87.53 = 1.36
88.85 + 0.35
Table Top-1 and top-5 accuracies (% + sd over 3 runs) on Imagenet, 2, being the config- uration of (He et al., 20)6a). For reference, state-of-the-art on Imagenet is (Dai et al., 2021) with a top-1 accuracy of 90.88%.
70
60
with rendors Initi leation
65
Beshee-13 10 with strided convolutions strides
0.0
0.5
1.0
1.5
2.0
Regularization term
Figure 4: Top-1 accuracy (%) on the Imagenet validation set as a function of the regularization term defined in equation 6, after training with À [0.1, 10).
top-1 accuracy. When training ResNet-18 18 with strided convolutions using the closest integer strides (i.e. ((11,32) (1, (1, 2). (2, 5))), the model converges to 24.54% top-1 accuracy. This suggests that performing pooling in the spectral domain more robust to aggressive downsampling, which corroborates the remarkable advantage of spectral pooling over strided convolutions when using poor strides choices in Tables 3 and 4 despite both models having fixed strides.
Limitations Pooling the spectral domain comes at higher computational cost than strided con- volutions it requires (I) computing non-strided convolution and (2) a DFT and its inverse (see Table A.2). This could be alleviated by computing the convolution the Fourier domain as an element-wise multiplication and summation tained by replacing the DFT by real-valued counterpart, such as the Hartley transform (Zhang & Ma, 2018), which would remove the need for complex-valued operations that may be poorly optimized in deep learning frameworks. We also observe no benefits of DiftStride when training DenseNets (Huang al., 2017), see Tables A.3 and A.4. We hypothesize that this is due to the limited number of downsampling layers, which reduces the space of stride configurations to few, equivalent ones when sampling strides in [1:3], 3). Finally, some hardware (e.g. TPUs) require a static computation graph. As DiffStride changes the spatial dimensions of intermediate representations- and thus the computation graph- between each gradient update, we currently only train on GPUs.
channels. Further improvements could be ob-
4 CONCLUSION AND FUTURE WORK
We introduce DiffStride the first downsampling layer with learnable strides. We show on audio and image classification that DiffStride can be used as drop-in replacement to strided convolutions, removing the need for cross-validating strides. As we observe that our method discovers multiple cqually-accurate stride configurations, we introduce a regularization term to favor the most compu- tationally advantageous. In future work, we will extend the scope of applications of DiffStride, to e.g. ID and 3D architectures. Moreover, learning strides by backpropagation opens new avenues in designing adaptive convolutional architectures, such multi-scale models that would learn to oper- ate at various scales in parallel by using independent branches with separate instances of DiffStride, or by predicting strides parameters of DiffStride on a per-example basis.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGMENTS
Authors thank Félix de Chaumont Quitry for the useful discussions and assistance throughout this project, well as the reviewers of ICLR 2022 for their feedback that helped improving this manuscript. Authors also thank Oren Rippel, Jasper Snoek, and Ryan P. Adams for their inspiring work spectral pooling.
REPRODUCIBILITY STATEMENT
We describe DiftStride in details in the text as well as with Algorithm 1 and Figure 1. We men- tion all relevant hyperparameters reproduce our experiments, as well as describe audio datasets in A.1. Moreover, we release Tensorflow 2.0 code for training a Pre-Act ResNet-18 with strided convolutions, spectral pooling or DiftStride on CIFARIO and CIFARI00, with DiftStride being implemented as stand-alone, reusable Keras layer. This open-source code can be found at https: /tte:://gihule googleeooeeeeoeoooooooooooo reeeerreeeereeeeeeeeeeee diffstride.
REFERENCES
Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jingliang Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Qiang Cheng, Guoliang Chen, et al. Deep speech 2: End-to- end speech recognition in english and mandarin. In International conference on machine learning, pp. 173-182. PMLR, 2016. 1,2
Bowen Baker, Otkrist Gupta, Nikhil Naik, and Ramesh Raskar. Designing neural network architec- tures using reinforcement learning. In 5th International Conference Learning Representationss ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL httsst http://pppeeevveeeeee/ee ew net. forrmmiidss) c2cvqee. 2
Irwan Bello, William Fedus, Xianzhi Du, Ekin Dogus Cubuk, A. Srinivas, Tsung-Yi Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. ArXiv, abs/2103.07579, 2021. 1
Yoshua Bengio, Nicholas Léonard, and Aaron Courville. Estimating or propagating gradients through stochastic neurons for conditional computation. arXiv preprint arXiv: 3388.33322 2013. 3, 4
Simone Bianco, Rémi Cadène, Luigi Celona, and Paolo Napoletano. Benchmark analysis of repre- sentative deep neural network architectures. IEEE Access, 6:64270-64277, 2018.
Léon Bottou et al. Online learning and stochastic approximations. On-line learning in neural networks, 17(9):142, 1998.
Y-Lan Boureau, Jean Ponce, and Yann LeCun. A theoretical analysis of feature pooling in visual recognition. In Proceedings the 27th international conference on machine learning (ICML-10), pp. 111-118, 2010.
S. Chopra, R. Hadsell, and Y. LeCun. Learning similarity metric discriminatively, with application to face verification. 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'0S))
vol. 1, 2005. 1
Alexis Conneau, Holger Schwenk, Loic Barrault, and Yann Lecun. Very deep convolutional net- works for text classification. In Proceedings of the 15th Conference of the European Chap- ter of the Association for Computational Linguistics: Volume 1, Long Papers, pp. 1107- 1116, Valencia, Spain, April 2017, Association for Computational Linguistics. URL https: //aclanthology ottp://-/////111111111111 1117-111....1.
Zihang Dai, Hanxiao Liu, Quoe V Le, and Mingxing Tan. Coatnet: Marrying convolution and attention for all data sizes. arXiv preprint arXiv:2/06. 04800, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scabl scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. leee, 2009. 2,
10

---

Published as a conference paper at ICLR 2022
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani. Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. In International Confer- ence on Learning Representations, 2020.
Taffeta M Elliott and Frédéric E Theunissen. The modulation transfer function for speech intelligi- bility. PLoS computational biology, (33):e1000002, 2009.
Jesse Engel, Cinjon Resnick, Adam Roberts, Sander Dieleman, Douglas Eck, Karen Simonyan, and Mohammad Norouzi. Neural Audio Synthesis Musical Notes with WaveNet Autoencoders. Technical report, apr 2017. URL http::ttt///.. http://axvvvvoro..o
17011 01279.
Adeen Flinker, Werner Doyle, Ashesh D Mehta, Orrin Devinsky, and David Poeppel. Spectrotem- poral modulation provides unifying framework for auditory cortical asymmetries. Nature human behaviour, 344):393-405, 2019.
Eduardo Fonseca, Andrés Ferraro, and Xavier Serra. Improving sound event classification by in- creasing shift invariance in convolutional neural networks. ArXiv, abs/2107.00623. 2021. 2
Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness-aware minimiza- tion for efficiently improving generalization. In International Conference on Learning Represen- tations, 2020. 8
K. Fukushima. Neocognitron: A self self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36:193-202, 1980. 1
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin. Convolutional sequence to sequence learning. In International Conference on Machine Learning, pp. 1243- 1252. PMLR, 2017.
Xavier Glorot, Antoine Bordes, and Yoshua Bengio. Deep sparse rectifier neural networks. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp- 315-323. JMLR Workshop and Conference Proceedings, 2011. 6
Benjamin Graham. Fractional max-pooling. arXiv preprint arXiv: 1412.607, 2014. 2,
Caglar Gulcehre, Kyunghyun Cho, Razvan Pascanu, and Yoshua Bengio. Learned- norm pooling for deep feedforward and recurrent neural networks. In Joint European Conference on Machine Learning and Knowledge Discovery Databases, pp. 530-546. Springer, 2014. 2
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016a. 7,
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings deep residual networks. In European conference computer vision, pp. 630-645. Springer, 2016b. 5
Toni Heittola, Annamaria Mesaros, and Tuomas Virtanen. TUT Urban Acoustic Scenes 2018, De- velopment dataset, April 2018. URL https::///oii
521.:......1sno/i.. 1228142
Geoffrey Hinton. What's wrong with convolutional nets. MIT Brain and Cognitive Sciences-Fall Colloquium Series, Dec 2014a. URL http ..., 2014.7
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications. ArXiv, abs/1704.04861. 2017. 5
Andrew G. Howard, Mark Sandler, Grace Chu, -iang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, Quoc V. Le, and Hartwig Adam. Searching for mobilenetv3. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp- 1314-1324, 2019.
11

---

Published as a conference paper at ICLR 2022
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp.
2017. 1, 9, 15
Patrick W Hullett, Liberty S Hamilton, Nima Mesgarani, Christoph E Schreiner, and Edward F Chang. Human superior temporal gyrus organization spectrotemporal modulation tuning de- rived from speech stimuli. Journal of Neuroscience, 36(6):2014-2033 2016. 6
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456.. PMLR, 2015. 6
Chen Jin, Ryutaro Tanno, Thomy Mertzanidou, Eleftheria Panagiotaki, and Daniel C. Alexander. Learning to downsample for segmentation of ultra-high resolution images, 2021.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. ICLR (Poster), 2015. 6
Qiuqiang Kong, Yin Cao, Turab lqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. JEEE/ACM Transac- fions on Audio, Speech, and Language Processing, 28:2880-2894, 2020. 2
A. Krizhevsky. Learning multiple layers of features from tiny images. 2009. 7
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep volutional neural networks. Advances in neural information processing systems, 15:1097-1105, 2012. 2,
Alex Krizhevsky et al. Learning multiple layers of features from tiny images. 2009.
Scinob Kuroki, Masataka Sawayama, and Shin' ya Nishida. Haptic texture perception 3d 3d-printed surfaces transcribed from visual natural textures. International Conference on Human Haptic Sensing and Touch Enabled Computer Applications, pp. 102-112. Springer, 2018. 3
Yann LeCun, Bernhard Boser, John Denker, Donnie Henderson, Richard Howard, Wayne Hubbard, and Lawrence Jackel. Handwritten digit recognition with a back proekk--ppgaaionn network. Advances in neural information processing systems, 2, 1989. 1,2
Chen- Yu Lee, Patrick W Gallagher, and Zhuowen Tu. Generalizing pooling functions in convo- lutional neural networks: Mixed, gated, and tree. Artificial intelligence and statistics, pp- 464-472. PMLR, 2016. 2
Richard G. Lyons. Understanding Digital Signal Processing (2nd Edition). Prentice Hall PTR, USA, 2004. ISBN 0131089897. 2
Hannah Muckenhirn, M. Magimai. -Doss, and S. Marcel. Towards directly modeling raw speech signal for speaker verification using cnns. 2018 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 4884 4888, 2018.
Biswajit Paria, Chih- Kuan Ych, N. Xu, B. Póczos, Pradeep Ravikumar, and I. Yen. Minimizing flops to learn efficient sparse representations. ArXiv, abs/2004.05665 2020. 5
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural networks: the official journal of the International Neural Nenwork Society, 12 1:145-151, 1999.
Antônio Ribeiro and Thomas B. Schön. How convolutional neural networks deal with aliasing. In 2021 IEEE International Conference Acoustics, Speech and Signal Processing. ICASSP, IEEE, 2021.
Oren Rippel, Jasper Snock, and Ryan Adams. Spectral representations for convolutional neural networks. In Proceedings ofthe the 28th International Conference Neural Information Processing Systems- Volume 2, pp. 1449-2457, 2015. 2,6,
Daniel L Ruderman. The statistics of natural images. Network: computation neural systems, (4):517, 1994. 3
12

---

Published as a conference paper at ICLR 2022
Faraz Sacedan, Nicolas Weber, Michael Goesele, and Stefan Roth. Detail- Petail-preserving pooling in deep networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9108-9116, 2018. 2,3
Tom Sercu, Christian Puhrsch, Brian Kingsbury, and Yann LeCun. Very deep multilingual convolu- tional neural networks for Ivesr. In 2016 IEEE international conference on acoustics, speech and signal processing (ICASSP). pp.
IEEE, 2016.
Nandini C Singh and Frédéric E Theunissen. Modulation spectra of natural sounds and ethological theories of auditory processing. The Journal of the Acoustical Society of America, 1 11(())3394 3411, 2003. 3
Dan Stowell, - Mike Wood, - Hanna Pamula, Yannis Stylianou, and Hervé Glotin. Automatic acoustic detection of birds through deep learning: the first Bird Audio Detection challenge. Tech- nical report, 2018. URL https: /eattss:/arxivv /arxivaivvv/vv////////aaa//aiiia org/pdft 1807 05812. pdf. 6
Sainbayar Sukhbaatar, Édouard Grave, Piotr Bojanowski, and Armand Joulin. Adaptive attention span in transformers. In Proceedings of the 57th Annual Meeting of the Association for Compu- tational Linguistics, pp. 331-335, 2019.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1-9, 2015.
7
Marco Tagliasacchi, Beat Gfeller, Félix de Chaumont Quitry, and Dominik Roblek. Self-supervised audio representation learning for mobile devices. arXiv preprint arXiv: /11.//99........ 1196, 2019.
Hossein Talebi and Peyman Milanfar. Learning to resize images for computer vision tasks. ICCV, 2021. 2
Mingxing Tan and Quoe Le. Efficientnet: Rethinking model scaling for convolutional neural net- works. In International Conference on Machine Learning, pp. 6105-6114. PMLR, 2019. 5, 8,
Mingxing Tan, Bo Chen, Ruoming Pang, Vijay Vasudevan, Mark Sandler, Andrew Howard, and Quoc V Le. Mnasnet: Platform-aware neural architecture search for mobile. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2820-2828, 2019.
Aäron van den Oord, Nal Kalchbrenner, Lasse Espeholt, K. Kavukcuoglu, Oriol Vinyals, and A. Graves. Conditional image generation with pixelenn decoders. In NIPS, 2016.
Pete Warden. Speech Commands: A Dataset for Limited- Vocaaullary Speech Recognition. Technical report, 2018. URL https: aaxiv, wwwpwwwwwwwpp////./.../..... http://dd1110000 03209 pdf. 6
P Yin, Lyu, S Zhang, Osher, YY Qi, and J Xin. Understanding straight-through estimator in train- ing activation quantized neural nets. In International Conference on Learning Representations, 2019. 3
Neil Zeghidour, Qiantong Xu, Vitaliy Liptchinsky, Nicolas Usunier, Gabriel Synnaeve, and Ronan Collobert. Fully convolutional speech recognition. arXiv preprint arXiv: 2018.
Hao Zhang and Jianwei Ma. Hartley spectral pooling for deep learning. arXiv preprint
arXiv:1810.04028, 2018. 9
Richard Zhang. Making convolutional networks shift-invariant again. In ICML, 2019. 1, 2,
Barret Zoph and Quoc V. Le. Neural architecture search with reinforcement learning. 2017, URL https:/ ptttps://ariiye wrw./bo..s....oooooo............ 611, 01578.
13

---

Published as a conference paper at ICLR 2022
A
APPENDIX
Table A.I: Datasets used for audio classification. Default train/test splits are always adopted.
Task
Name
Classes
Train examples
Test examples
Acoustic scenes
TUT Urban 2018
10
7,829
810
Birdsong detection
DCASE2018
32,129
3,561
Music (instrument)
Nsynth
11
289,205
12,678
Music (pitch)
Nsynth
128
289,205
12,678
Speech commands
Speech commands
35
84,771
10,700
30
25
4
20
15
2
10
/=3
ss /=1
$/1
S 5 S2
S S3
W
/=1
(a) Distribution of learned strides in ResNet-18 18 for the different initializations.
(b) Distribution of the final global striding factors for the different initializations.
Figure A.1: Learned strides by DiffStride on the CIFAR100 dataset.
15
10.0
7.5
10
5.0
5
2.5
1= 4
n. 1=1
A
$ S 57 S S SA (a) Distribution learned strides ResNet-18 for the different initializations.
w
1=1
(b) Distribution the final global striding factors for the different initializations.
Figure A.2: Learned strides by DiffStride the Imagenet dataset.
Analysis of strides learned CIFAR100 and ImageNet In Figure A. (Figure A.2), we show distributions learned strides and the global striding factor at convergence on CIFAR (Im- agenet), starting from random stride initializations. On CIFAR 100, we observe equivalence classes, ie. model that learns various stride configurations for a same accuracy. On Imagenet, even though also observe significant variance of the global striding factor, models tend to downsample only in the upper layers. Striding late in the architecture comes at a higher computational cost, which furthermore justifies regularizing DiffStride to reduce complexity shown in Section 3.2.
14

---

Published as a conference paper at ICLR 2022
Time and space complexity in practice While Figure 4 reports theoretical estimates of com- putational complexity based on stride configurations, both spectral pooling and DiffStride require computing DFT and its inverse. Moreover, DiffStride requires accumulating gradients with respect to the strides during training. Table A.2 reports the duration and peak memory usage of the multi- task architecture described in 3.1, for single batch. Replacing strided convolutions with spectral pooling increases the wall time by 32% due to the DFT and inverse DFT, while the peak memory usage almost unaffected. DiffStriie furthermore increases the wall time (by 43% w.r.t strided con- volutions) as the backward pass is more expensive. Similarly, it almost doubles the peak memory usage. However, in inference, DiftStride does not need to compute and store gradients w.r.t. the strides, thus the time and space complexity become identical that of spectral pooling.
Strided Conv.
Spectral
DiftStride
Time/step
1.0
1.32
1.43
Training
Peak memory (GB)
1.0
1.02
1.98
Time/step
1.0
1.32
1.32
Inference
Peak memory (GB)
1.0
1.02
1.02
Per-step time and peak memory usage of Spectral Pooling and DiffStride relative to strided convolutions, on V100 GPU. During training, a "step"" is the forward and backward pass for single batch, while inference it only involves forward pass.
Table A.2:
DenseNet experiments CIFAR We also evaluate DiffStride in DenseNet (Huang al., 201 7), especially the DenseNet-Bo architecture with depth of 121 and a growth rate of 32. The DenseNet architecture halves spatial dimensions during transition blocks. We replace the 2D average pooling in the transition blocks by spectral pooling or DiffStride. The considered architecture for DenseNet has two downsampling steps. We similar experiment as in 3.2 with random strides between the dense blocks on the two CIFAR datasets. We observe that initializing strides randomly does not af- feet the performance of the standard Densenet-BC architecture with average pooling. Consequently, DiffStride does not improve over alternatives.
Init. Strides
Average Pooling
Spectral
DiffStride
(2, 2)
92.3 + 0.2
91.5 0.2
91.5 + 0.1
(1, 2)
91.8 0.2
90.5 0.5
91.1 0.3
(1,
92.0 + 0.2
91.0
9..6
(2, 3)
92.0 + 0.3
92.1 0.2
92.2 + 0.1
(3, 1) Mean accuracy
91.6 + 0.2
91.5 + 0.4
91.7 + 0.2
91.9 0.3
91.3 0.6
91.6 0.4
Table A.3: Accuracies (% + sd over 3 runs) for CIFARI0 for each downsampling method with the DenseNet-BC architecture, First column represents strides at each transition block, (2, 2) being the configuration of (Huang al., 2017).
Init. Strides
Average Pooling
Spectral
DiffStride
(2,
69.1 + 0.6
68.6 + 0.6
68.2 = 0.2
(1, 2)
67.7 + 0.3
67.2 + 0.3
68.0 + 0).
(1,
67.8 0.4
67.3 0.2
68.0 + 0.7
(2, 3)
67.6 + 0.3
69.0 + 0.8
69.0 + 0.2
(3, 1)
68.4 + 0.6
69.1 0.4
69.7 + 0.6
Mean accuracy
68.1 + 0.7
68.2 + 1.0
68.6 + 0.8
Table A.4: Accuracies (% + sd over runs) for CIFAR100 for each downsampling method with the DenseNet-BC architecture. First column represents strides at each transition block, (2, 2) being the configuration of (Huang al., 2017).
15

---

Published as a conference paper at ICLR 2022
EfficientNet experiments CIFAR We evaluate DiffSStrie in EfficientNet-BO architecture (Tan & Le, 2019), a lightweight model discovered by architecture search. This architecture has seven strided convolutions. Unlike Tan & Le (2019), train from scratch on CIFAR, which explains the lower accuracy of the baseline. As the model has seven downsampling layers, rescale the images from 32 32 128 128, and only sample strides in [1: We run a similar experiment as in 3.2 with random strides on the two CIFAR datasets. Consistently with the results obtained with a ResNet-18, spectral pooling much more robust to poor strides than strided convolutions, with DiffStride outperforming all alternatives.
do not pre-train ImageNet, but rather
Init. Strides
Average Pooling
Spectral
DifiStride
1,2, 11,1,2,2,211
87.2 + 0.1
90.4 0.2
91.1 + 0.0
89.7 + 0.1
90.9 + 0.3
90.9 + 0.1
2,2,2,2,2,5
83.7 0.2
90.0 + 1.0
90.8 + 0.1
1,2,,,12,111
89.2 0.2
90.4 = 0.4
91.1 + 0.1
Mean accuracy
87.5 + 2.5
90.4 + 0.6
90.9 + 0.1
Table A.5: Accuracies (% + sd over runs) for CIFARIO for each downsampling method with the EfficientNet-BO architecture. First column represents strides at each strided convolution, with (1, 2,, 2, 1, 1) being the configuration of (Tan Le, 2019).
Init. Strides
Average Pooling
Spectral
Diff'Stride
2, 2, 1,
55.2 + 0.3
66.0 + 0.6
66.6 = 0.5
(1, 2. 1. (1,
62.0 + 0.7
66.4 + 0.6
66.6 = 0.3
2)
46.8 + 2.0
65.9 + 0.5
66.3 0.7
(2, 1, 2.
2,1
60.4 + 0.1
65.5 + 0.1
67.0 = 0.1
Mean accuracy
56.1 + 6.3
65.9 + 0.5
66.6 + 0.5
Table A.6: Accuracies (% + sd over runs) for CIFARI00 for each downsampling method with the EfficientNet-B0 architecture. First column represents strides each strided convolution, with 1,,,,,,,1,,11, being the configuration of (Tan & Le, 2019).
Ablation study: learning a per-dimension stride or a shared one We perform multi-task audio classification with either learning single stride value for each DifiStride layer, different one for the time and frequency axes. The overall performance across tasks is improved when learning a different stride value for each dimension (See Table A.7).
Setting
Multi-task
Task
Shared stride
Different strides
Acoustic scenes
97.4 + 0.3
97.7 + 0.3
Birdsong detection
79.7 + 1.0
78.6 = 0.5
Music (instrument)
70.7 + 0.5
73.0 + 0.8
Music (pitch)
86.7 + 0.5
89.9 + 0.3
Speech commands
86.8 1 0.1
86.2 1 0.8
Mcan Accuracy
84.3 - 9.2
85.0 + 8.9
Table A.7: Test accuracy (% + sd over 3 runs) for audio classification in the multi-task (one model for all tasks) with DifdSt when learning a single stride value (Shared stride) per layer, or different one for each dimension (Different strides).
16


Reviews:
[{"review_id": "_kTtOcHDv3", "reviewer": "Reviewer_34rN", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Positives\n---------\n\nOverall I feel that the paper presents a really neat idea well. Besides a few minor issues (see below) the paper was enjoyable to read and describes the ideas well. There is thorough experimentation on a range of tasks and model architectures that demonstrate the power of the proposed approach. \n\n\nConcerns\n--------\n\nReally, the only major concern I have is around the rather limited discussion of the limitations. In particular, I'd at least like some discussion (or thoughts) on why it doesn't improve densenet performance, and I'd like to see experimental results presented on the impact of the use of DiffStride during training and inference, both in terms of measurable memory usage and impact on training & inference wall time. Obviously such results can be caveated with a statement that the implementation could be improved as per the existing discussion in the limitations section.\n\n \nMinor points\n------------\n\nPlease check and fix algorithm 1 - I believe line 4 should be using the output from the filtering with the mask, rather than the raw FFT result. It would also be better not to reuse the $\\tilde{y}$ symbol on lines 4 and 5, and it would also be helpful if the symbols (the variants of the intermediate computations $y$, $\\tilde{y}$}) were clearly labelled on Fig 1.\n", "summary_of_the_paper": "This paper proposes an approach to learning optimal striding parameters in convolutional networks. The proposed approach, DiffStride, is a downsampling layer that builds on spectral pooling to allow for integer output dimensions, but arbitrary strides by cropping in the Fourier domain. Unlike spectral pooling, DiffStride relaxes the parameters of the cropping mask to be differentiable, and using the stop-gradient operator in the cropping. Results show that the proposed approach can work well as a drop-in replacement for normally fixed pooling layers. It is also demonstrated that from different random initialisations a variety of different pooling approaches can be learned that acheive similar performance. Use of a regularisation term the attempts to encourage time and space efficiency reduces this variability and allows accuracy to be traded off.", "summary_of_the_review": "Overall, as can be seen from my review I'm satisfied that this paper makes a good contribution.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "OvSCimqQCz7", "reviewer": "Reviewer_e1mG", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "PROs\n- The paper is very well written and pretty much clear at a first read. All the introduced components are properly motivated intuitively, and figures and algorithms really help the reader in understanding.\n- The authors excel in framing their work within related art. Specifically, they discuss both standard alternatives such as pooling, as well as alternatives based on fractional stride. For all of them, issues are properly pointed out such that the motivation behind this paper is very clear.\n- The paper is, to the best of my effort, technically sound. In terms of novelty, one might argue it being a bit incremental over SpectralPool, as it substitutes the cropping hyperparameters with learnable values. However, i) this step is not trivial, as the cropping operation is non-differentiable, and the authors introduce a solution to that and ii) this modification enables a significant improvement in performances, registered across a number of datasets.\n- The authors support their proposal by carrying out experiments on multiple datasets. Specifically, 6 datasets are employed for audio recognition and 3 (CIFAR-10, CIFAR-100 and Imagenet) are used for image recognition. In my opinion, for a paper of this type, attacking a very fundamental and obiquitous operator in modern architectures, being able to showcase improvements on non-toy datasets such as Imagenet is remarkable and noteworthy.\n- For every experiment, the authors report the mean and standard deviation over multiple trials, strenghtening the reliability of the conclusions.\n- The authors properly discuss limitations of their work, by highlighting the computational cost, some failure cases on DenseNets and implementation challenges on specific types of hardware.\n\n---\nCONs\n\nI think this paper is very solid, and I consider the following points as minor concerns.\n- In Sec 3.1, the authors claim that as \"DiffStride learns different strides for the time and frequency axes\", and as such \"this justifies using a different parameter for each dimension rather then sharing strides\". However, this validation is flawed. Just because the model takes advantage of this flexibility, it doesn't necessarily mean that it is beneficial in the end. This might be the case if we completely trusted gradient descent to reach global optimums of the cost function, which is not the case. Indeed, the final stride configuration might depend much on the initialization, as also suggested by the experiment in Fig. 3. Therefore, to validate such a claim the authors should simply include a baseline model where DiffStride is applied with shared striding parameters for the time and frequency axes.\n- Fig. 4 would be much clearer if on the x axis MACs or FLOPs were represented, instead of the regularization term. This would help quantify how much computation DiffStride can save without losing significant performances. As it is, the value of the regularization term dos not tell much about actual computational cost.\n- To my understanding, within experiments the authors substitute a few downsampling layers (not even all of them) in a network with DiffStride. As a reader, I would have expected that every convolutional layer would be equipped with its own learnable striding. I grasp that this may not be practical but I don't have a clear view of the reason. The authors should consider adding some motivation behind this choice.\n- Another point that may be worth discussing as an extension/future work. May the DiffStride technique be employed within a conditional computation framework, by predicting the striding parameters conditioned on the current example? I think that this strategy might succeed especially when aiming at optimizing the computational cost of a model. I do not expect such an extention to be implemented and tested within the rebuttal period. I would just like the authors' opinion about that (potential, concerns etc).\n- In Tab. 1, the Speaker Id dataset provides no information whatsoever, as all methods score a perfect accuracy of $100.0 \\pm 0.0$. I suggest the authors to remove this dataset, as it is apparently beyond trivial, and including it downgrades the quality of the experiment rather than increasing it.\n- On CIFAR-100, the authors show that SpectralPool is significantly better than strided convs even with the same downsampling hyperparameters. Can the authors provide an intuition for this behavior?  ", "summary_of_the_paper": "In this paper, the authors propose DiffStride, a technique for learning the stride of downsampling operations in neural networks by gradient descent. Specifically, similarily to SpectralPool, the feature map is transformed to the frequency domain by a Discrete Fourier Transform, where it is then cropped according to learnable parameters. The authors envision a simple soft-relaxation of the cropping in order to allow the gradient to flow towards cropping parameters. Performances against both standard and random stride policies are reported on a number of datasets, both for audio and image recognition. Moreover, the authors introduce a regularization objective that penalizes the use of small strides, in the interest of encouraging downsampling for improving computational and memory cost.", "summary_of_the_review": "Overall, I recommend the paper for acceptance. The contribution is novel and clearly motivated, and experimental results are encouraging across different datasets. I think this paper can be of interest for many within the ICLR community. There are some improvement points as discussed above, but in my opinion the strengths clearly outnumber the weaknesses.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "GOIhiSwFvyN", "reviewer": "Reviewer_ubnz", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "+ The paper is well-written and the proposed formulation is sound. \n\n+ Also extensive experiments are provided on several datasets to demonstrate the benefits of the proposed method.\n\n\n- Authors demonstrate that the proposed method can recover from different initial strides and learn the optimal one. However, my most important concern is the lack of appropriate comparisons with neural architecture search approaches, since - in my view - these are the most direct competitors of the proposed method. Therefore, I would expect experiments and appropriate discussions between the cost and benefits between the proposed method and neural architecture search approaches.\n\n- The proposed method seems to target only residual architectures. Discussion on how the proposed method could be used with other architectures would be beneficial\n\n- Also, providing experiments with different architectures (apart from resnet) on the same dataset would also further improve the confidence on the obtained results.\n\n- What is the actual overhead of the using the proposed method in real applications? How much does the training time and memory usage increases?", "summary_of_the_paper": "In this work the authors introduce a differentiable stride formulation, which allows for learning the stride value. To this end, they propose learning the size of a cropping mask in the Fourier domain, which allows for learning how to perform resize in a differentiable way. Authors present experiments on several datasets on different domains (image, audio), including large scale datasets.", "summary_of_the_review": "Overall, I think this is a good paper with an interesting approach on differential stride learning. There is no theoretical discussion and I would like to see some additional experiments, however I feel that this work is slightly above acceptance threshold, mainly given the novelty of the application.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "A5qFbtqx4jI", "reviewer": "Reviewer_BTMH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The idea is interesting and makes sense to me. The writing is clear and easy to understand.  However, from the experiment results (especially table3 and 4), it seems that the proposed method has marginal improvement compared to regular strided conv and spectral pooling baselines. It seems that the default setting can already achieve very good results on ImageNet and how it will affect the model performance when the model is large enough remains unclear. The benefit of learnable strides is not fully demonstrated. It would be great if the authors could implement one or more future works in Sec. 4 to showcase its capability further and show the proposed module's overheadein detail to let the readers better understand its limitation.\n\n", "summary_of_the_paper": "This paper proposes DiffStride as a drop-in replacement to standard downsampling layers. It extends previous work Spectral Poolnig and learns the size of the cropping box in the frequency domain by backpropagation. Experiments are conducted on audio and image classification, and the results show that the model can learn non-integer stride and adapt different initial stride well.", "summary_of_the_review": "Overall I think this paper presents a novel idea to learn stride in the downsampling layer. However, the current results are not good enough to showcase its effectiveness by making the stride learnable. The authors may want to find more ways, as discussed in Sec.4, to show its value.\n\n-- Post rebuttal\nAfter reading the authors' response, I raise my rating to 8.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a method to learn the stride of downsampling in deep networks using a  gradient-based learning approach. The main idea is to work in the frequency domain and to learn the cropping mask in that domain. The authors also introduce a regularization for applications seeking computationally and memory efficiency. The authors investigate the interest of the approach on a number of datasets with audio and image data. 

The reviewers praised the paper, appreciating the elegance of the approach and the effort made to thoroughly evaluate it. The reviewers also appreciated the clarity of the exposition and the care in the reporting of the results. The reviewers also expressed some concerns about several choices of the design (stride sharing) and the lack of detail in some experiments (computational /memory efficiency). Finally, the reviewers wished the paper had more theoretical grounding.

The authors submitted detailed responses to the reviewers' comments. After reading the responses, updating the reviews, and discussion, the reviewers found that the responses were ‘reinforcing [their] initial assessments' and their several concerns were satisfactorily addressed. Moreover several of the reviewer’s suggestions clearly already led to an improved manuscript with very thorough experimental evaluation and simpler approaches for stride sharing. 

The paper proposed an elegant, learning-based, approach to one of the most important design choices in deep network architecture design: the strides in the convolutions. The authors provided a careful and thorough experimental evaluation, and moreover improved it during the review process following the reviewer’s feedback. 

Accept, definitely. 
Decision: Accept (Oral)