Title: MAKING CONVOLUTIONAL NETWORKS SHIFT-INVARIANT AGAIN

Abstract: Modern convolutional networks are not shift-invariant, despite their convolutional nature: small shifts in the input can cause drastic changes in the internal feature maps and output. In this paper, we isolate the cause -the downsampling operation in convolutional and pooling layers -and apply the appropriate signal processing fix -low-pass filtering before downsampling. This simple architectural modification boosts the shift-equivariance of the internal representations and consequently, shift-invariance of the output. Importantly, this is achieved while maintaining downstream classification performance. In addition, incorporating the inductive bias of shift-invariance largely removes the need for shift-based data augmentation. Lastly, we observe that the modification induces spatially-smoother learned convolutional kernels. Our results suggest that this classical signal processing technique has a place in modern deep networks.

1. INTRODUCTION:

Deep convolutional neural networks (CNNs) are designed to perform high-level tasks and be robust to low-level nuisance factors. For example, small shifts in the input should simply shift the internal feature maps (shift-equivariance), and leave the output relatively unaffected (shift-invariance). This property has been explicitly engineered through convolutional and pooling layers, where the same function is applied on a local region across the image in a sliding window fashion. However, recent work (Engstrom et al., 2017; Azulay & Weiss, 2018) has found that small shifts can drastically change the output of a classification network. Why is this the case? Shift-invariance is lost when spatial resolution is lost, for example, from pooling layers. Our insight is that conventional strided-pooling, as shown in Fig. 1 (top), is inherently composed of two operations: (1) evaluating the pooling operator densely (without striding), and (2) downsampling. Naive downsampling loses shift-equivariance, as high-frequency components of the signal alias into low-frequencies. This phenomenon is commonly illustrated in movies, where wheels appear to spin backwards, due to the frame rate not meeting the Nyquist sampling criterion (known as the Stroboscopic effect). Separating these operations is important, as it allows us to keep the pooling operation, while applying the appropriate fix to the downsampling operation.

We propose to add the signal processing tool of low-pass filtering before downsampling, as shown in Fig. 1 (bottom) . By low-pass filtering, the high-frequency components of the signal are reduced, reducing aliasing and better preserving shift-equivariance. This ultimately cascades into better shiftinvariance in the output. We show example classification instabilities in Fig. 2 .

A potential concern is that over-aggressive low-pass filtering can result in heavy loss of information. However, we find that with a reasonable selection of low-pass filter weights, we can maintain classification performance while increasing shift-invariance. Furthermore, we show that without shift-based data augmentation, incorporating this inductive bias actually improves performance.

We find that the learned filters also naturally become smoother after adding the blurring layer. These results indicate that incorporating this small modification not only induces shift-invariance, but causes the network to learn a smoother feature extractor.

In summary, our contributions are as follows: We low-pass filter between the operations. This keeps the original pooling operation, while antialiasing the appropriate signal. This equivalent analysis and modification can be applied to any strided layer, such as convolution.

• We isolate the cause for loss of shift-invariance -downsampling. Separating the downsampling from pooling enables us to keep the desired pooling, while fixing the loss of shift-equivariance.

We propose to low-pass filter before downsampling, a common signal processing technique. • We validate on a classification task, and demonstrate increased shift-equivariance in the features and shift-invariance in the output. • In addition, we observe large improvements in classification performance when training without shift-augmentation, indicating more efficient usage of data.

2. RELATED WORK:

Local connectivity and weight sharing have been a central tenet of neural networks, including the Neocognitron (Fukushima & Miyake, 1982) , LeNet (LeCun et al., 1998) and modern networks such as Alexnet (Krizhevsky et al., 2012) , VGG (Simonyan & Zisserman, 2014) , ResNet (He et al., 2016) , and DenseNet (Huang et al., 2017) . In biological systems, local connectivity was famously discovered observed in a cat's visual system by Hubel & Wiesel (1962) . Recent work has strived to build in additional types of invariances, such as rotation, reflection, and scaling (Sifre & Mallat, 2013; Bruna & Mallat, 2013; Esteves et al., 2017; Kanazawa et al., 2014; Worrall et al., 2017; Cohen & Welling, 2016) . Our work focusses on the elusive goal of shift-invariance.

Though properties such as shift-equivariance have been engineered into networks, what factors and invariances does an emergent representation actually learn? Analysis of deep networks have included qualitative approaches, such as showing patches which activate hidden units (Girshick et al., 2014; Zhou et al., 2014) , actively maximizing hidden units (Mordvintsev et al., 2015) , and mapping features back into pixel space (Dosovitskiy & Brox, 2016a; b; Mahendran & Vedaldi, 2015; Zeiler & Fergus, 2014; Nguyen et al., 2017; Hénaff & Simoncelli, 2015) . Our analysis is focused on a specific, low-level property and is complementary to these qualitative approaches.

A more quantitative approach for analyzing networks is measuring representation or output changes (or robustness to changes) in response to manually generated perturbations to the input, such as image transformations (Goodfellow et al., 2009; Lenc & Vedaldi, 2015; Azulay & Weiss, 2018) , geometric transforms (Ruderman et al., 2018; Fawzi & Frossard, 2015) , and CG renderings with various shape, poses, and colors (Aubry & Russell, 2015) . A related line of work is in adversarial examples, where directed perturbations in the input can result in large changes in the output. These perturbations can be directly on pixels (Goodfellow et al., 2014a; b) , a single pixel (Su et al., 2017) , small deformations (Xiao et al., 2018) , or even affine transformations (Engstrom et al., 2017) . We aim make the network robust to the simplest of these types of attacks and perturbations: shifts. Both Hénaff & Simoncelli (2015) and Azulay & Weiss (2018) identify that modern deep networks ignore the Nyquist sampling criterion when downsampling. In our work, we propose and empirically validate an easily adoptable fix which minimally perturbs the existing network architecture.

Classic hand-engineered computer vision and image processing representations, such as SIFT (Lowe, 1999) , wavelets, and image pyramids (Burt & Adelson, 1987; Adelson et al., 1984) also extract features in a sliding window manner, often with some subsampling factor. As discussed in Simoncelli et al. (1992) , literal shift-equivariance cannot hold when with subsampling. Shiftequivariance can be recovered if features are extracted densely, for example textons (Leung & Malik, 2001 ), the Stationary Wavelet Transform (Fowler, 2005) , and DenseSIFT (Vedaldi & Fulkerson, 2010) . Deep networks can also be evaluated densely, by removing striding and making appropriate changes to subsequent layers by using á trous/dilated convolutions (Chen et al., 2014; 2018; Yu & Koltun, 2015) . This comes at great computation and memory cost. Our work investigates achieving shift-equivariance with minimal additional computation, by blurring before subsampling.

Blurring before downsampling is fundamental technique in signal processing (Oppenheim et al., 1999) , image processing (Gonzalez & Woods, 1992) , computer graphics (Foley et al., 1995) , and vision (Szeliski, 2010) . In deep learning, average pooling (LeCun et al., 1990 ) is a form of blurring. Scherer et al. (2010) finds max-pooling to more effective than variants of blurred-downsampling, under the assumption that they are alternatives. Conversely, we show that they are compatible.

3. METHODS:



3.1. PRELIMINARIES:

Deep convolutional networks as feature extractors Let an image with resolution H × W be represented by X ∈ R H×W ×3 . An L-layer CNN can be expressed as a feature extractor F l (X) ∈ R H l ×W ×C l , with layer l ∈ [0, L], spatial resolution H l × W l and C l channels. Each feature map can also be upsampled to original resolution, F l (X) ∈ R H l ×W l ×C l .

Shift-equivariance and shift-invariance A representation F is shift-equivariant if shifting the input produces a shifted feature map, meaning that shifting and feature extraction are commutable. We more rigorously define the Shift function in Eqn. 4. A representation is shift-invariant if shifting the input results in an identical representation.

EQUATION

EQUATION

For modern classifiers, layer l = 0 is the raw pixels, and final layer L is a probability distribution over D classes, F L ∈ ∆ 1×1×D . The net typically progressively reduces spatial resolution, until all resolution is lost and features are of shape R 1×1×C l . A common technique, such as used in (Lin et al., 2013; He et al., 2016; Huang et al., 2017) , is to average across the entire convolutional feature map spatially, and use fully-connected layers in all subsequent layers, which can be expressed as 1 × 1 convolutions (Long et al., 2015) . In such a setting, as proven by Azulay & Weiss (2018) , shift-invariance on the output will necessarily emerge from shift-equivariance in the convolutional features.

Modulo-N shift-equivariance/invariance In some cases, the definitions in Equations 1, 2 may hold only when shifts (∆h, ∆w) are integer multiples of N. We refer to these scenarios as modulo-N shift-equivariance or invariance. For example, modulo-2 shift-invariance means that even-pixel shifts of the input result in an identical representation, but odd-pixel shifts may not.

3.2. CONVENTIONAL POOLING VS PROPOSED POOL-BLUR-DOWNSAMPLE:

Conventional strided pooling breaks shift-equivariance In Fig. 3 , we show an example 1-D signal

[0, 0, 1, 1, 0, 0, 1, 1]. Max-pooling (kernel k = 2, stride s = 2) will result in [0, 1, 0, 1]. Simply shift- ing the input by one index results a dramatically different answer of [1, 1, 1, 1]. Shift-equivariance is lost.

As seen in the bottom-left, both of these results are inherently downsampling from an intermediate signal -the input signal densely max-pooled (k = 2, s = 1). We can write a maxpooling layer as a composition of two functions, max-pooling densely evaluated, followed by naive downsampling: MaxPool k,s (X) = Downsample s (MaxPool k,1 (X)). Max-pooling preserves shiftequivariance (when evaluated densely), but naive downsampling does not.

Blurring before downsampling better preserves shift-equivariance We propose to low-pass filter the intermediate signal before downsampling, as shown in Fig. 3 (bot-right). We define our MaxPool-BlurDownsample operator below.

EQUATION

Sampling from the low-pass filtered signal gives [.5, 1, .5, 1] and [.75, .75, .75, .75] (Fig. 3 bot-right).

These are closer to each other and better representations of the intermediate signal.

The method allows for a choice of blur kernel. In image processing, small kernels are often used across applications such as edge detection (Canny, 1986) and image pyramids (Adelson et al., 1984) .

We try a number of kernels, ranging from size 2 × 2 to 7 × 7. As the blur kernels are separable, it

pixels [32] conv1_2 [32] pool1 (dense) [32] pool1 (ds) [16] conv2_2 [16] pool2 (dense) [16] pool2 (ds) [8] conv3_2 [8] pool3 (dense) [8] pool3 (ds) [4] conv4_2 [4] pool4 (dense) [4] pool4 (ds) [2] conv5_2 [2] pool5 (dense) [2] pool5 (ds) [1] classifier [1] softmax [1]

(a) MaxPool (Baseline) can be implemented as a series of two convolutions (vertical blur followed by horizontal), and added computation scales linearly with k blur , rather than quadratically.

pixels [32] conv1_2 [32] pool1 (dense) [32] pool1 (ds) [16] conv2_2 [16] pool2 (dense) [16] pool2 (ds) [8] conv3_2 [8] pool3 (dense) [8] pool3 (ds) [4] conv4_2 [4] pool4 (dense) [4] pool4 (ds) [2] conv5_2 [2] pool5 (dense) [2] pool5 (ds) [1] classifier [1] softmax [1]

4. EXPERIMENTS:

4.1 EXPERIMENTAL SETUP Data, architecture, training schedule We test on CIFAR10 classification (Krizhevsky & Hinton, 2009) , which consists of 50k training and 10k testing images at resolution 32 × 32. We use the VGG13 architecture (Simonyan & Zisserman, 2014) from the PyTorch framework (Paszke et al., 2017) 1 and will make code available.

Each block consists of 2 Conv-BatchNorm-ReLU chunks, followed by MaxPool, doubling feature channels and halving spatial resolution until all resolution is lost. A final softmax predicts a probability vector. We use stochastic gradient descent (SGD) with momentum 0.9 and batch size 128. We train for 100 epochs at initial learning rate 0.1 and 50 additional epochs at 0.01 and 0.01.

Low-pass filter kernels:

We try a number of standard low-pass filters, shown in Table 1 , ranging from size 2 × 2 to 7 × 7. All filters allow the DC signal pass and suppress (or completely kill) the highest frequency. Variations in filters correspond to tradeoffs between location of the cutoff frequency, slope of the cutoff, and variation of lobes in the passband and stopband. These properties are well-studied in the context of finite impulse response (FIR) filter design. However, it is unclear which types of filters are best suited for deep networks, so we empirically investigate their effects.

Circular convolution and shifting Edge artifacts are an important consideration. When an image is shifted, information is necessarily lost on one side, and has to be filled in on the other. In all our experiments, we use circular shifting and convolution. When the convolutional kernel hits the edge, it wraps to the other side. When shifting, pixels are "rolled" off the edge to the other side.

[Shift ∆h,∆w (X)] h,w,c = X (h-∆h)%H,(w-∆w)%W,c , where % is the modulus function (4) This modification minorly decreases classification performance, 93.8% vs 93.4% with data augmentation. This could potentially be mitigated by additional padding, at the expense of memory and computation. But more importantly, this methodology affords us a clean testbed. Any loss in shift-equivariance or invariance is purely due to characteristics of the feature extractor.

4.2. ANALYSIS:

We measure shift-equivariance/invariance in three ways, targeting different aspects. We first focus on the shift-equivariance of the internal layers. We then check on the agreement of the hard output classification. Finally, we measure how much the soft predicted probability itself varies.

1. Feature distance (lower is better). We test how close shift-equivariance and invariance are to being fulfilled by computing d(Shift ∆h,∆w ( F(X)), F(Shift ∆h,∆w (X))) and d( F(X), F(Shift ∆h,∆w (X)) (left & right-hand sides of Eq. 1, 2), respectively. We use cosine distance, which is commonly used for deep features (Kiros et al., 2015; Zhang et al., 2018) . 2. Classification consistency (higher is better). Perhaps of greatest interest is the actual decisions the classifier makes. We can measure its consistency by checking how often the network outputs the same classification, given the same image with two different shifts: E (X,h1,w1,h2,w2) 1{arg max P (Shift h1,w1 (X)) = arg max P (Shift h2,w2 (X))}.

3. Classification variation (lower is better). The metric above looks at the hard classification, discounting classifier confidence. Similar to Azulay & Weiss (2018) , we trace the variation in probability of correct classification, given different shifts. We can capture the variation across all possible shifts: V ar h,w ({P correct class (Shift h,w (X))}}).

Table 1 shows results across a number of different low-pass filters, training with and without data augmentation. We dissect the results below.

How shift-equivariant are deep features? In Fig. 4 (top), we compute distance from shiftequivariance, as a function of all possible shift-offsets (∆h, ∆w) and layers. MaxPool layers are broken into two components -before and after downsampling. Pixels are trivially shift-equivariant, as are all layers before the first downsampling. Once downsampling occurs in pool1(ds), shiftequivariance is lost. However, modulo-N shift-equivariance still holds, and each subsequent downsampling doubles the factor.

Additionally, we observe that before the downsampling operation, the pooling layer first increases shift-equivariance (e.g., conv3 2 to pool3(dense)). This is consistent with the long-held intuition that pooling build invariances inside the network (LeCun et al., 1990) and isolates the downsampling operation as the culprit behind loss of shift-equivariance.

Does blurring before downsampling achieve better shift-equivariance? In Fig. 4 (bottom), we add a blurring filter to the MaxPool layers, as proposed in Section 3, and again plot shiftequivariance maps for each layer. Shift-equivariance is clearly better preserved. In particular, the severe drop-offs in downsampling layers do not occur. Improved shift-equivariance throughout the network cascades into more consistent classifications in the final softmax layer.

Some selected examples are in Fig. 2 . Our method stabilizes the classifications. In Fig. 6 , we show the distribution of classification variations, before and after adding in the low-pass filter. Even a small 2 × 2 filter, immediately variation. As the filter size is increased, the output classification variation decreases. This has a larger effect when training without data augmentation, but is still observable when training with data augmentation.

Does shift-invariance degrade performance? Our method produces more shift-equivariant feature maps and consequently, more shift-invariant outputs. However, does this come at a cost?

We study the output classification consistency versus classification accuracy. In Fig. 5 (left), we show results, trained without shift-based data augmentation. Training with the baseline MaxPooling gives accuracy 91.6% and consistency 88.1%. Our proposed change -with a 5 × 5 triangle filter improves accuracy to 93.3% and consistency to 98.2%. This indicates that low-pass filtering does not destroy the signal, or make learning harder. On the contrary, preserving shift-equivariance serves as "built-in" augmentation, indicating more efficient data usage. 1 2 3 Figure 7 : Total Variation (TV) by layer. We compute average smoothness of learned conv filters per layer (lower is smoother). Baseline MaxPool is in black, and adding additional blurring is shown in colors. Note that the learned convolutional layers become smoother, indicating that a smoother feature extractor is induced. The Binomial-7 filter produces consistently strong results, in both consistency and accuracy.

c o n v 1 _ 1 c o n v 1 _ 2 c o n v 2 _ 1 c o n v 2 _ 2 c o n v 3 _ 1 c o n v 3 _ 2 c o n v 4 _ 1 c o n v 4 _ 2 c o n v 5 _ 1 c o n v 5 _ 2 Layer 0.0 0.2 0.4 0.6 0.8 1.0 1.2 1.4 1.6 Average Filter TV MaxPool [1] Delta MaxPoolBlurDS [1 1] Rect-2 MaxPoolBlurDS [

In principle, networks can learn to be shift-invariant from data. Does adding shift-based data augmentation remove the benefit from method? Shift-based data augmentation with the baseline network results in consistency of 96.6%, lower than our method trained without data augmentation. In addition, as seen in Fig. 5 (right), applying out method with data augmentation provides an immediate jump in classification consistency, while maintaining accuracy. From there, a clear tradeoff appears -higher amounts of shift-invariance can be achieved at the cost of decreased accuracy. For example, very large rectangular filters over-aggressively smooth the signal. Downstream applications may favor one factor over another, and the choice of filter allows one to explore this space. Fig. 6 investigates the distribution of classification variations. Training with data augmentation with the baseline network reduces variation (black lines on both plots). Our method reduces variation in both scenarios. More aggressive filtering further decreases variation.

How do the learned convolutional filters change with the proposed modification? We measure spatial smoothness using the normalized Total Variation (TV) metric proposed in Ruderman et al. (2018) . Our proposed change smooths the internal feature maps for purposes of downsampling. As shown in Fig. 7 , this induces smoother learned filters throughout the network. Adding in more aggressive blur kernels further decreases the TV (increasing smoothness). This indicates that our modification actually induces a smoother feature extractor overall. How does the proposed method affect timing? In Tab. 2, we show the added time each element of the proposed method takes: evaluating the MaxPool layer at stride 1 instead of stride 2, and running a blurring filter.

Since the blurring filters are separable, time increases linearly with filter size. The largest filter adds 12.3% per forward pass. This is significantly cheaper than evaluating multiple forward passes in an ensembling approach (1024× computation to evaluate every shift), or evaluating each layer more densely by exchanging striding for dilation (4×, 16×, 64×, 256× computation for conv2-conv5, respectively). These timings are on our VGG13 network setup. With deeper networks, the relative added computation decreases.

5. CONCLUSIONS AND DISCUSSION:

In summary, we show that shift-invariance is lost through a deep network, as downsampling in pooling layers do not meet the Nyquist criteria. We propose a simple architectural modification, following signal processing principles, to improve shift-equivariance. This change allows the network architecture designer to keep their pooling layer of choice untouched.

We achieve higher consistency while maintaining classification performance. In addition, we show large improvements in both performance and consistency when training without data augmentation. This is potentially applicable to online learning scenarios, where the data distribution is changing. Future directions include exploring the potential benefit to downstream applications, such as nearest-neighbor retrieval, improving temporal consistency in video models, robustness to adversarial examples, and high-level vision tasks such as detection. Another possible future direction is learning the downsampling kernels. Overall, our experiments indicate that this classical signal processing technique has a place in modern deep networks.

APPENDIX A DENSENET ARCHITECTURE:

Blurring before downsampling can be applied to any strided layer in any network. We provide an additional experiment using the DenseNet architecture (Huang et al., 2017) . In Fig. 8 , we show classification consistency vs. accuracy, similar to Fig. 5 for VGG13 in the main paper. Comparison to VGG13 for Baseline Network We use the DenseNet-40-12 architecture, from a publicly available implementation.2 Relative to VGG13, DenseNet achieves higher performance (94.4% vs 93.8% ), despite using fewer parameters (1M vs 9M). DenseNet also starts with higher shift-invariance (97.7% vs. 96.6%) for two reasons: (a) fewer downsampling layers (2 vs 5) and (b) already using blurring before downsampling, in the form of AveragePool layers, equivalent to using a Rectangle (2) filter. We investigate the effects of replacing this 2 × 2 filter.

Results on DenseNet Our method improves the DenseNet results, and confirms the findings in the main paper. In some cases, results are actually stronger. The primary findings are:

• As seen in Fig. 8 (left), using a stronger low-pass filter, such as Binomial (5, 7) without data augmentation, provides competitive performance compared to the baseline trained with data augmentation. For Binomial (7), performance is actually better in both consistency and accuracy. • As seen in Fig. 8 (right), when training with data augmentation, using filters such as Rect (3), Triangle (3,5), and Binomial (5,7) not only increase consistency, as expected, but also slightly increases accuracy, surprisingly.

The observations for DenseNet corroborate the results from VGG13 in the main paper, further demonstrating the effectiveness of blurring before downsampling.

B ROBUSTNESS TO SHIFT-BASED ADVERSARY:

In the main paper, we show that using the proposed PoolBlurDownsample method increases the classification consistency, while maintaining accuracy. A logical consequence is increased accuracy in presence of a shift-based adversary. We empirically confirm this in Fig. 9 for VGG13 on CI-FAR10. We compute classification accuracy as a function of maximum adversarial shift. A max shift of 2 means the adversary can choose any of the 25 positions within a 5 × 5 window. For the classifier to "win", it must correctly classify all of them correctly. Max shift of 0 means that there is no adversary. Conversely, a max shift of 16 means the image must be correctly classified at all 32 × 32 = 1024 positions.

Our primary observations are as follows:

• As seen in Fig. 9 (left), the baseline network (gray) is very sensitive to the adversary.

• Adding larger Binomial filters (from red to purple) increases robustness to the adversary. In fact, Binomial ( 7) filter (purple) without augmentation outperforms the baseline (black) with augmentation. • As seen in Fig. 9 (right), adding larger Binomial filters also increases adversarial robustness, even when training with augmentation.

These results corroborate the findings in the main paper, and demonstrate a use case: increased robustness to shift-based adversarial attack. 

C EFFECT OF BLURRING BEFORE POOLING:

In our proposed method, we break the strided-pooling operation into two, and blur in between. This allows us to directly blur before downsampling, which has solid theoretical backing in sampling theory (Oppenheim et al., 1999) , and is commonly used in image processing (Gonzalez & Woods, 1992) , graphics (Foley et al., 1995) , and computer vision (Szeliski, 2010) . Here, we empirically investigate blurring before pooling instead. Fig. 10 shows the results by applying blurring first (shown in the gray points), in comparison our proposed method (colored polygons, as shown before in Fig. 5 ). We make the following observations:

• In Fig. 10 (right), when training with augmentation, blurring before pooling reduces performance for all filters. For almost all filters (with few exceptions), both classification accuracy and consistency are significantly reduced. • In Fig. 10 (left), when training without augmentation, the lower performing filters actually perform better when blurring before filtering. For the better filters, however, blurring before pooling lowers performance (similar shift-invariance, but lower accuracy).

The signal pre-pooling is undoubtedly related to the signal post-pooling. Thus, blurring before pooling provides "second-hand" anti-aliasing, and still increases shift-invariance over the baseline. Though it does empirically help in certain circumstances (the lower-performing filters, without augmentation), the best performing filters use the proposed PoolBlurDownsample ordering. We use select standard low-pass filters to empirically test. The filter weights are shown in Tab. 1 in the main paper. Note that weights are normalized to sum to 1. Rectangle, Triangle, and Binomial filters are discussed in textbooks such as (Szeliski, 2010) . Window and Least Squares are more advanced FIR filter design techniques.

• Rectangle: a moving average, often referred to as a box filter. The filter is a vector of length ones. For example, Rect-2, is [1, 1]. This filter, followed by subsampling, is equivalent to the AveragePooling layer. • Triangle: linearly decreases weight of neighboring values. This is equivalent of applying box filtering twice. For example, Triangle-3 is [1, 2, 1], is two Rect-2 [1, 1] filters convolved together, and Triangle-5 is [1, 2, 3, 2, 1], is two Rect-3 [1, 1, 1] filters convolved together.

• Binomial: Filter used in Laplacian Pyramids (Burt & Adelson, 1987) ; [1, 1] filter convolved with itself repeatedly. Note that Binomial-2,3 is equivalent to Rectangle-2 and Triangle-3, respectively.

• Window: filter produced using the window method (firwin), as described in "7.4 Optimum Approximations for FIR Filters" in Oppenheim et al. (1999) .

• Least Squares: least squares error minimization (firls), from Python scipy.signal toolbox, as described in Selesnick (2005) .

E.2 FILTER SEPARABILITY:

As discussed in Section 3.2, our filters are separable. Consider a G ∈ R. If G is rank-1, it can be decomposed (or separated) into G y G x , where G y ∈ R K×1 and G x ∈ R 1×K . This is an important consideration when convolving G with signal X ∈ R H×W .

G * X = (G y * G x ) * X = G y * (G x * X), where * is convolution (5)

:

https://github.com/pytorch/vision/blob/master/torchvision/models/vgg.py

https://github.com/andreasveit/densenet-pytorch

The left-hand side, evaluating the blur with 2-D convolution takes H × W × K × K multiply-adds, with runtime scaling quadratically with K 2 . Meanwhile, evaluating a horizontal and vertical blur sequentially takes H × W × K multiply-adds each, scaling linearly by K.

Figures and Tables:

Figure1: (Top) Pooling does not preserve shift-equivariance. It is functionally equivalent to densely evaluated pooling followed by naive downsampling. The latter operation ignores the Nyquist sampling theorem and loses shift-equivariance. (Bottom) We low-pass filter between the operations. This keeps the original pooling operation, while antialiasing the appropriate signal. This equivalent analysis and modification can be applied to any strided layer, such as convolution.

Figure 2: Classification stability for selected images. Predicted probability of the correct class changes when shifting the image. The baseline (black) exhibits chaotic behavior, which is stabilized by our method (blue).

Figure 3: Toy example of sensitivity to shifts. We illustrate how downsampling affects shift-equivariance with a toy example. (Top-Left) An input toy signal is in light gray; max-pooled (k = 2, s = 2) toy signal is in blue. (Top-Right) Simply shifting the input and then max-pooling provides a completely different answer (red). (Bot-Left) The blue and red points are inherently sampled from densely max-pooled (k = 2, s = 1) intermediate signal (thick black). (Bot-Right) We instead sample from the low-passed intermediate signal, shown in green and magenta, better preserving shift-equivariance.

Figure 4: Shift-equivariance throughout the network. We compute feature distance between left and righthand sides of the shift-equivariance condition in Equation 1. Each point in each heatmap is a shift (∆h, ∆w). Layer resolution is in [brackets]; in the last three, shift-equivariance is equivalent to shift-invariance. Layers pix-pool1(dense) have perfect equivariance (distance 0 at all shifts, shown by blue). Red is half mean distance between two random different images, and is adjusted depending on the layer. (a) On the baseline, shift-equivariance is reduced each time downsampling takes place. Modulo-N shift-equivariance holds, with N doubling with each downsampling. (b) With our proposed change, shift-equivariance is better maintained, and the resulting classfication (softmax) layer is more shift-invariant.

Figure 6: Distribution of per-image classification variation. We show the distribution of classification variation in the test set, (left) without and (right) with data augmentation at training. Lower variation means more consistent classifications (and increased shift-invariance). Training with data augmentation drastically reduces variation in classification. Adding filtering further decreases variation.

Figure 9: Robustness to shift-based adversarial attack. Classification accuracy as a function of the number of pixels an adversary is allowed to shift the image. Applying our proposed filtering increases robustness, both without (left) and with right data augmentation.

Figure10: Blurring before pooling. Blurring before the max-pooling (gray points) for different filters, as compared to their PoolBlurDownsample counterparts (colored polygons). The poorer performing filters, when training without data augmentation, observe an increase in performance. For almost all filters when training with data augmentation, and for the higher-performing filters training without data augmentation, performance is significantly reduced, often in both accuracy and consistency. Directly blurring the downsampled signal (after the pooling layer), as proposed in the main paper, is more effective.

Figure 5: Classification consistency vs. classification. Networks trained (left) without and (right) with shiftbased data augmentation, using various filters. Up (more consistent) and to the right (more accurate) is better.Number of sides corresponds to number of filter taps used (e.g., diamond for 4-tap filter); colors correspond to different methods for generating FIR filters. We highlight filters Rectangle (4), Triangle (5), and Binomial (5-7), which perform consistently well in both metrics and settings. Classification consistency and classification. Results across blurring filters and training scenarios (without and with data augmentation). We evaluate classification accuracy without shifts (Test accuracy -None) and on random shifts (Test accuracy -Random), as well as classification consistency. Highlighted filters perform consistently well in both metrics and settings, as more easily seen in Fig.5.

Table content: <table><tr><td/><td colspan="4">Train without Data Augmentation</td><td/><td/><td/><td/><td>Train with Data Augmentation</td></tr><tr><td/><td colspan="2">Baseline (trn w/o data aug) Baseline (trn w/ data aug)</td><td>Filt size [2] Filt size [3]</td><td>Filt size [4] Filt size [5]</td><td colspan="2">Filt size [6] Filt size [7]</td><td/><td colspan="2">Filt type [Rectangle] Filt type [Triangle]</td><td>Filt type [Pascal] Filt type [Window]</td><td>Filt type [Least-Sq]</td></tr><tr><td>0.02 0.00 0.02 0.90 0.92 0.94 0.96 Classification Consistency 0.04 0.98</td><td/><td/><td/><td/><td/><td colspan="2">0.970 0.975 0.980 0.985 0.990</td><td/></tr><tr><td>0.04 0.88</td><td colspan="5">0.910 0.915 0.920 0.925 0.930 0.935 Classification Accuracy</td><td colspan="2">0.965</td><td colspan="2">Classification Accuracy 0.905 0.910 0.915 0.920 0.925 0.930 0.935 0.940</td></tr><tr><td/><td colspan="2">0.04</td><td>0.02</td><td/><td/><td>0.00</td><td/><td/><td>0.02</td><td>0.04</td></tr><tr><td/><td/><td/><td/><td/><td colspan="5">Train with no augmentation</td><td>Train with augmentation</td></tr><tr><td/><td>Filter shape</td><td># Taps</td><td colspan="2">Weights</td><td colspan="5">Test accuracy Classification Test accuracy Classification</td></tr><tr><td/><td/><td/><td/><td/><td colspan="4">None Rand</td><td>Consistency</td><td>None Rand</td><td>Consistency</td></tr><tr><td colspan="2">Delta (baseline)</td><td>1</td><td>[1]</td><td/><td colspan="2">91.6</td><td colspan="2">87.4</td><td>88.1</td><td>93.4</td><td>93.7</td><td>96.6</td></tr><tr><td/><td>Rectangle</td><td>2</td><td colspan="2">[1, 1]</td><td colspan="2">92.8</td><td colspan="2">89.3</td><td>90.5</td><td>93.9</td><td>93.8</td><td>97.6</td></tr><tr><td/><td>Rectangle</td><td>3</td><td colspan="2">[1, 1, 1]</td><td colspan="2">93.4</td><td colspan="2">91.8</td><td>94.5</td><td>93.6</td><td>93.7</td><td>97.9</td></tr><tr><td/><td>Rectangle</td><td>4</td><td colspan="2">[1, 1, 1, 1]</td><td colspan="2">93.2</td><td colspan="2">92.9</td><td>97.7</td><td>93.4</td><td>93.4</td><td>98.5</td></tr><tr><td/><td>Rectangle</td><td>5</td><td colspan="2">[1, 1, 1, 1, 1]</td><td colspan="2">92.2</td><td colspan="2">92.1</td><td>98.3</td><td>92.4</td><td>92.5</td><td>98.7</td></tr><tr><td/><td>Rectangle</td><td>6</td><td colspan="2">[1, 1, 1, 1, 1, 1]</td><td colspan="2">91.4</td><td colspan="2">91.2</td><td>97.3</td><td>91.4</td><td>91.5</td><td>98.9</td></tr><tr><td/><td>Rectangle</td><td>7</td><td colspan="2">[1, 1, 1, 1, 1, 1, 1]</td><td colspan="2">90.8</td><td colspan="2">90.7</td><td>98.8</td><td>90.5</td><td>90.5</td><td>99.0</td></tr><tr><td/><td>Triangle</td><td>3</td><td colspan="2">[1, 2, 1]</td><td colspan="2">93.1</td><td colspan="2">91.4</td><td>93.9</td><td>93.6</td><td>93.5</td><td>98.0</td></tr><tr><td/><td>Triangle</td><td>5</td><td colspan="2">[1, 2, 3, 2, 1]</td><td colspan="2">93.3</td><td colspan="2">93.0</td><td>98.2</td><td>93.3</td><td>93.2</td><td>98.6</td></tr><tr><td/><td>Triangle</td><td>7</td><td colspan="2">[1, 2, 3, 4, 3, 2, 1]</td><td colspan="2">92.3</td><td colspan="2">92.3</td><td>98.8</td><td>92.4</td><td>92.3</td><td>99.0</td></tr><tr><td/><td>Binomial</td><td>4</td><td colspan="2">[1, 3, 3, 1]</td><td colspan="2">93.0</td><td colspan="2">91.1</td><td>93.2</td><td>93.4</td><td>93.3</td><td>98.1</td></tr><tr><td/><td>Binomial</td><td>5</td><td colspan="2">[1, 4, 6, 4, 1]</td><td colspan="2">93.2</td><td colspan="2">92.6</td><td>96.3</td><td>93.1</td><td>93.2</td><td>98.4</td></tr><tr><td/><td>Binomial</td><td>6</td><td colspan="2">[1, 5, 10, 10, 5, 1]</td><td colspan="2">93.0</td><td colspan="2">92.4</td><td>96.9</td><td>93.4</td><td>93.3</td><td>98.6</td></tr><tr><td/><td>Binomial</td><td>7</td><td colspan="2">[1, 6, 15, 20, 15, 6, 1]</td><td colspan="2">93.0</td><td colspan="2">93.0</td><td>98.1</td><td>93.2</td><td>93.2</td><td>98.8</td></tr><tr><td/><td>Window</td><td>3</td><td colspan="2">[1, 1.57, 1]</td><td colspan="2">93.3</td><td colspan="2">91.5</td><td>94.2</td><td>93.5</td><td>93.5</td><td>98.0</td></tr><tr><td/><td>Window</td><td>6</td><td colspan="2">[-1, 1.67, 5, 5, 1.67, -1]</td><td colspan="2">92.9</td><td colspan="2">90.2</td><td>91.0</td><td>93.4</td><td>93.5</td><td>98.1</td></tr><tr><td/><td>Window</td><td>7</td><td colspan="2">[-1, 0, 3, 4.71, 3, 0, -1]</td><td colspan="2">92.4</td><td colspan="2">91.1</td><td>94.0</td><td>93.4</td><td>93.5</td><td>97.8</td></tr><tr><td colspan="2">Least Squares</td><td>3</td><td colspan="2">[1, 1,63, 1]</td><td colspan="2">93.1</td><td colspan="2">91.4</td><td>93.8</td><td>93.7</td><td>93.8</td><td>98.0</td></tr><tr><td colspan="2">Least Squares</td><td>7</td><td colspan="4">[-1, 0, 3.80, 6.13, 3.80, 0, -1] 92.7</td><td colspan="2">91.0</td><td>93.9</td><td>93.4</td><td>93.5</td><td>97.9</td></tr></table>

Timing analysis We test the average speed of a forward pass on a GTX1080Ti Nvidia GPU for a batch size of 100 of 32 × 32 image with the VGG13 network.

Table content: <table/>

Figure8: Classification consistency vs. classification for DenseNet Same test in as in Fig.5, but with DenseNet(Huang et al., 2017) instead of VGG13(Simonyan & Zisserman, 2014). We show networks trained (left) without and (right) with shift-based data augmentation, using various blurring filters. Consistency is computed by computing classification of an image with two random shifts, and checking for agreement. Up (more consistent) and to the right (more accurate) is better. Number of sides corresponds to number of filter taps used (e.g., triangle for 3-tap filter); colors correspond to different methods for generating FIR filters.

Table content: <table><tr><td/><td colspan="2">Train without Data Augmentation</td><td/><td colspan="2">Train with Data Augmentation</td></tr><tr><td/><td>Filt size [2], Baseline (w/o data aug) Filt size [2], Baseline (w/ data aug)</td><td>Filt size [1] Filt size [3]</td><td>Filt size [5] Filt size [7]</td><td>Filt type [Delta] Filt type [Rectangle]</td><td>Filt type [Triangle] Filt type [Binomial]</td></tr><tr><td>0.04 0.02 0.00 0.02 0.04 0.92 0.93 0.94 0.95 0.96 0.97 Classification Consistency 0.98 0.99</td><td colspan="2">0.920 0.925 0.930 0.935 0.940 0.945 Classification Accuracy</td><td>0.9725 0.9750 0.9775 0.9800 0.9825 0.9850 0.9875 0.9900 0.9925</td><td colspan="2">Classification Accuracy 0.939 0.940 0.941 0.942 0.943 0.944 0.945</td></tr><tr><td/><td>0.04</td><td>0.02</td><td>0.00</td><td>0.02</td><td>0.04</td></tr></table>

Peer review (iclr2019):
 This work shows that adding a simple blurring into max pooling layers can address issues of image classification instability under small image shifts. In general this work presents a simple and easy to implement solution to a common problem of CNNs and even though it lacks more thorough theoretical analysis of this problem from the signal processing perspective (such as minimal size of the blurring kernel for fulfilling the Nyquist-Shannon sampling theorem), it seems to provide ample empirical evidence. Pros: + The introduction and motivation is really well written and Figure 3 provides a clear visualisation main max pooling operator issues. + The proposed method is really simple and shows promising results on the CIFAR dataset. With random shifts, authors had to tackle cropping with circular shifts. As it can cause artifacts in the data, authors also provide baseline performances on the original data (used for both training and testing). + Authors provide a thorough evaluation, ranging from comparing hidden representations to defining consistency metrics of the classified classes. This work is lacking in the experimental section due to some missing details and few inconsistencies. I believe the most of my concerns can be relatively easily fixed/clarified in an update of this submission. Major issues, which if fixed would improve the rating: - It is not correct to average test accuracy and test consistency as both measures are different quantities, especially when using them for ranking. The difference between accuracy of different methods are considerably smaller than differences in the classification consistency. - It is not clear how many shifts are used for computing the "Random Test Accuracy" and the "Classification Accuracy". Also whether the random shifts are kept constant between evaluated networks and evaluation metrics. - Authors do not address the question what is the correct order of operations for the blurring. E.g. would the method empirically work if blurring was applied before max pooling? Do the operations commute? - The selection of the filters is rather arbitrary, especially regarding the 1D FIR filters. The separability of these filters should be discussed. - I believe authors should address how this work differs to [1], as it also tests different windowing functions for pooling operators, even though in different tasks. Minor issues, which would be nice to fix however which do not influence my rating: * Section 3.1 - And L-Layer deep *CNN*, H_l x W x C_l -> H_l x W_l x C_l * Section 3.1. Last paragraph - I would not agree with the statement that in CNNs the shift invariance must necessarily emerge upon shift equivariance. If anything, this may hold only for the last layer of a network without fully connected layers and with average pooling of the classifier output (ResNet/GoogleNet like networks). * Explicitly provide the network architecture as [Simonyan14] does not test on CIFAR and cannot use Batch normalisation. * It would be useful to add citation for the selected FIR filters. * The flow of section 4.2. can be improved to help readability. The three metrics should be first motivated before their introduction. Metric 2. paragraph - the metric is defined below, not above. * It would be interesting to see what would be the performance if the blurring filters were trained as well (given some sensible initialisation). * One future direction would be to verify that this approach generalises to larger networks as well. It might be worth to discuss this in the conclusions. [1] Scherer, Dominik, Andreas Müller, and Sven Behnke. "Evaluation of pooling operations in convolutional architectures for object recognition." Artificial Neural Networks–ICANN 2010. Springer, Berlin, Heidelberg, 2010. 92-101.