Making Convolutional Networks Shift-Invariant Again

Modern convolutional networks are not shift-invariant , despite their convolutional nature : small shifts in the input can cause drastic changes in the internal feature maps and output . In this paper , we isolate the cause – the downsampling operation in convolutional and pooling layers – and apply the appropriate signal processing fix – low-pass filtering before downsampling . This simple architectural modification boosts the shift-equivariance of the internal representations and consequently , shift-invariance of the output . Importantly , this is achieved while maintaining downstream classification performance . In addition , incorporating the inductive bias of shift-invariance largely removes the need for shift-based data augmentation . Lastly , we observe that the modification induces spatially-smoother learned convolutional kernels . Our results suggest that this classical signal processing technique has a place in modern deep networks . 1 INTRODUCTION . Deep convolutional neural networks ( CNNs ) are designed to perform high-level tasks and be robust to low-level nuisance factors . For example , small shifts in the input should simply shift the internal feature maps ( shift-equivariance ) , and leave the output relatively unaffected ( shift-invariance ) . This property has been explicitly engineered through convolutional and pooling layers , where the same function is applied on a local region across the image in a sliding window fashion . However , recent work ( Engstrom et al. , 2017 ; Azulay & Weiss , 2018 ) has found that small shifts can drastically change the output of a classification network . Why is this the case ? Shift-invariance is lost when spatial resolution is lost , for example , from pooling layers . Our insight is that conventional strided-pooling , as shown in Fig . 1 ( top ) , is inherently composed of two operations : ( 1 ) evaluating the pooling operator densely ( without striding ) , and ( 2 ) downsampling . Naive downsampling loses shift-equivariance , as high-frequency components of the signal alias into low-frequencies . This phenomenon is commonly illustrated in movies , where wheels appear to spin backwards , due to the frame rate not meeting the Nyquist sampling criterion ( known as the Stroboscopic effect ) . Separating these operations is important , as it allows us to keep the pooling operation , while applying the appropriate fix to the downsampling operation . We propose to add the signal processing tool of low-pass filtering before downsampling , as shown in Fig . 1 ( bottom ) . By low-pass filtering , the high-frequency components of the signal are reduced , reducing aliasing and better preserving shift-equivariance . This ultimately cascades into better shiftinvariance in the output . We show example classification instabilities in Fig . 2 . A potential concern is that over-aggressive low-pass filtering can result in heavy loss of information . However , we find that with a reasonable selection of low-pass filter weights , we can maintain classification performance while increasing shift-invariance . Furthermore , we show that without shift-based data augmentation , incorporating this inductive bias actually improves performance . We find that the learned filters also naturally become smoother after adding the blurring layer . These results indicate that incorporating this small modification not only induces shift-invariance , but causes the network to learn a smoother feature extractor . In summary , our contributions are as follows : Strided Pooling Shift-equivariance lost ; heavy aliasing max ( ) ( 1 ) Dense Pooling Preserves shift-equivariance max ( ) ( 2 ) Naïve Downsampling Shift-eq . lost ; heavy aliasing max ( ) max ( ) • We isolate the cause for loss of shift-invariance – downsampling . Separating the downsampling from pooling enables us to keep the desired pooling , while fixing the loss of shift-equivariance . We propose to low-pass filter before downsampling , a common signal processing technique . • We validate on a classification task , and demonstrate increased shift-equivariance in the features and shift-invariance in the output . • In addition , we observe large improvements in classification performance when training without shift-augmentation , indicating more efficient usage of data . 2 RELATED WORK . Local connectivity and weight sharing have been a central tenet of neural networks , including the Neocognitron ( Fukushima & Miyake , 1982 ) , LeNet ( LeCun et al. , 1998 ) and modern networks such as Alexnet ( Krizhevsky et al. , 2012 ) , VGG ( Simonyan & Zisserman , 2014 ) , ResNet ( He et al. , 2016 ) , and DenseNet ( Huang et al. , 2017 ) . In biological systems , local connectivity was famously discovered observed in a cat ’ s visual system by Hubel & Wiesel ( 1962 ) . Recent work has strived to build in additional types of invariances , such as rotation , reflection , and scaling ( Sifre & Mallat , 2013 ; Bruna & Mallat , 2013 ; Esteves et al. , 2017 ; Kanazawa et al. , 2014 ; Worrall et al. , 2017 ; Cohen & Welling , 2016 ) . Our work focusses on the elusive goal of shift-invariance . Though properties such as shift-equivariance have been engineered into networks , what factors and invariances does an emergent representation actually learn ? Analysis of deep networks have included qualitative approaches , such as showing patches which activate hidden units ( Girshick et al. , 2014 ; Zhou et al. , 2014 ) , actively maximizing hidden units ( Mordvintsev et al. , 2015 ) , and mapping features back into pixel space ( Dosovitskiy & Brox , 2016a ; b ; Mahendran & Vedaldi , 2015 ; Zeiler & Fergus , 2014 ; Nguyen et al. , 2017 ; Hénaff & Simoncelli , 2015 ) . Our analysis is focused on a specific , low-level property and is complementary to these qualitative approaches . A more quantitative approach for analyzing networks is measuring representation or output changes ( or robustness to changes ) in response to manually generated perturbations to the input , such as image transformations ( Goodfellow et al. , 2009 ; Lenc & Vedaldi , 2015 ; Azulay & Weiss , 2018 ) , geometric transforms ( Ruderman et al. , 2018 ; Fawzi & Frossard , 2015 ) , and CG renderings with various shape , poses , and colors ( Aubry & Russell , 2015 ) . A related line of work is in adversarial examples , where directed perturbations in the input can result in large changes in the output . These perturbations can be directly on pixels ( Goodfellow et al. , 2014a ; b ) , a single pixel ( Su et al. , 2017 ) , small deformations ( Xiao et al. , 2018 ) , or even affine transformations ( Engstrom et al. , 2017 ) . We aim make the network robust to the simplest of these types of attacks and perturbations : shifts . Both Hénaff & Simoncelli ( 2015 ) and Azulay & Weiss ( 2018 ) identify that modern deep networks ignore the Nyquist sampling criterion when downsampling . In our work , we propose and empirically validate an easily adoptable fix which minimally perturbs the existing network architecture . Classic hand-engineered computer vision and image processing representations , such as SIFT ( Lowe , 1999 ) , wavelets , and image pyramids ( Burt & Adelson , 1987 ; Adelson et al. , 1984 ) also extract features in a sliding window manner , often with some subsampling factor . As discussed in Simoncelli et al . ( 1992 ) , literal shift-equivariance can not hold when with subsampling . Shiftequivariance can be recovered if features are extracted densely , for example textons ( Leung & Malik , 2001 ) , the Stationary Wavelet Transform ( Fowler , 2005 ) , and DenseSIFT ( Vedaldi & Fulkerson , 2010 ) . Deep networks can also be evaluated densely , by removing striding and making appropriate changes to subsequent layers by using á trous/dilated convolutions ( Chen et al. , 2014 ; 2018 ; Yu & Koltun , 2015 ) . This comes at great computation and memory cost . Our work investigates achieving shift-equivariance with minimal additional computation , by blurring before subsampling . Blurring before downsampling is fundamental technique in signal processing ( Oppenheim et al. , 1999 ) , image processing ( Gonzalez & Woods , 1992 ) , computer graphics ( Foley et al. , 1995 ) , and vision ( Szeliski , 2010 ) . In deep learning , average pooling ( LeCun et al. , 1990 ) is a form of blurring . Scherer et al . ( 2010 ) finds max-pooling to more effective than variants of blurred-downsampling , under the assumption that they are alternatives . Conversely , we show that they are compatible . 3 METHODS . 3.1 PRELIMINARIES . Deep convolutional networks as feature extractors Let an image with resolution H × W be represented by X ∈ RH×W×3 . An L-layer CNN can be expressed as a feature extractor Fl ( X ) ∈ RHl×W×Cl , with layer l ∈ [ 0 , L ] , spatial resolution Hl ×Wl and Cl channels . Each feature map can also be upsampled to original resolution , F̃l ( X ) ∈ RHl×Wl×Cl . Shift-equivariance and shift-invariance A representation F̃ is shift-equivariant if shifting the input produces a shifted feature map , meaning that shifting and feature extraction are commutable . We more rigorously define the Shift function in Eqn . 4 . Shift∆h , ∆w ( F̃ ( X ) ) = F̃ ( Shift∆h , ∆w ( X ) ) ∀ ( ∆h , ∆w ) ( 1 ) 0 1 2 3 4 5 6 7 8 0.0 0.5 1.0 1.5 Si gn al ( S pa tia l D om ai n ) original signal Max-Pooling ( shift-0 ) 0 1 2 3 4 5 6 7 8 0.0 0.5 1.0 1.5 Si gn al ( S pa tia l D om ai n ) original signal ( shift-1 ) Max-Pooling ( shift-1 ) 0 1 2 3 4 5 6 7 8 0.0 0.5 1.0 1.5 Si gn al ( S pa tia l D om ai n ) original signal Max-Pooling Densely Max-Pooling ( shift-0 ) Max-Pooling ( shift-1 ) 4 3 2 1 0 1 2 3 4 0 2 4 6 8 M ag ni tu de ( F ou rie r D om ai n ) Max-Pooling Densley Max-Pooling ( shift-0 ) Max-Pooling ( shift-1 ) 0 1 2 3 4 5 6 7 8 0.0 0.5 1.0 1.5 Si gn al ( S pa tia l D om ai n ) Max-Pooling Densely MPD + LPF MaxPool-Blur-DS ( shift-0 ) MaxPool-Blur-DS ( shift-1 ) 4 3 2 1 0 1 2 3 4 0 2 4 6 8 M ag ni tu de ( F ou rie r D om ai n ) Max-Pooling Densley MPD + LPF MaxPool-Blur-DS ( shift-0 ) MaxPool-Blur-DS ( shift-1 ) Figure 3 : Toy example of sensitivity to shifts . We illustrate how downsampling affects shift-equivariance with a toy example . ( Top Left ) An input toy signal is in light gray ; max pooled ( k = 2 , s = 2 ) toy signal is in blue . ( Top-Right ) Simply shifting the input and then max-pooling provides a completely different answer ( red ) . ( Bot-Left ) The blue and red points are inherently sampled from densely max-pooled ( k = 2 , s = 1 ) intermediate signal ( thick black ) . ( Bot-Right ) We instead sample from the low-passed intermediate signal , shown in green and magenta , better preserving shift-equivariance . A representation is shift-invariant if shifting the input results in an identical representation . F̃ ( X ) = F̃ ( Shift∆h , ∆w ( X ) ) ∀ ( ∆h , ∆w ) ( 2 ) For modern classifiers , layer l = 0 is the raw pixels , and final layer L is a probability distribution over D classes , FL ∈ ∆1×1×D . The net typically progressively reduces spatial resolution , until all resolution is lost and features are of shape R1×1×Cl . A common technique , such as used in ( Lin et al. , 2013 ; He et al. , 2016 ; Huang et al. , 2017 ) , is to average across the entire convolutional feature map spatially , and use fully-connected layers in all subsequent layers , which can be expressed as 1 × 1 convolutions ( Long et al. , 2015 ) . In such a setting , as proven by Azulay & Weiss ( 2018 ) , shift-invariance on the output will necessarily emerge from shift-equivariance in the convolutional features . Modulo-N shift-equivariance/invariance In some cases , the definitions in Equations 1 , 2 may hold only when shifts ( ∆h , ∆w ) are integer multiples of N. We refer to these scenarios as moduloN shift-equivariance or invariance . For example , modulo-2 shift-invariance means that even-pixel shifts of the input result in an identical representation , but odd-pixel shifts may not .

Summary: From a theoretical point of view, one might be tempted to believe that deep CNNs are translation equivariant and their predictions are translation invariant. In practice, this is not necessarily true. The authors propose to augment standard deep CNNs with low-pass filters to reduce this problem. The results seem promising for an older VGG architecture.