Convolutional Neural Networks are not invariant to translation, but they can learn to be

1 INTRODUCTION . The equivalence of an objects across different viewpoints is considered a fundamental capacity of human vision recognition ( Hummel , 2002 ) . This is mediated by the inferior temporal cortex , which appears to provide the bases for scale , translation , and rotation invariance ( Tanaka , 1996 ; O ’ Reilly & Munakata , 2019 ) . Taking inspiration from biological models ( LeCun et al. , 1998 ) , Artificial Neural Networks have been endowed with convolution and pooling operations ( LeCun et al. , 1998 ; 1990 ) . It is often claimed that Convolutional Neural Networks ( CNNs ) are less susceptible to irrelevant sources of variation such as image translation , scaling , and other small deformations ( Gens & Domingos , 2014 ; Xu et al. , 2014 ; LeCun & Bengio , 1995 ; Fukushima , 1980 ) . While it is difficult to overstate the importance of convolution and pooling operations in deep learning , their ability to make a network invariant to image transformations has been overestimated : for example , Gong et al . ( 2014 ) showed that CNNs achieve neither rotation nor scale invariance . Similarly , multiple studies have reported highly limited translation invariance ( Kauderer-Abrams , 2017 ; Gong et al. , 2014 ; Azulay & Weiss , 2019 ; Chen et al. , 2017 ; Blything et al. , 2020 ) . It is important to understand the reason for the misconception regarding the ability of CNNs to be invariant to translation . We believe this is due to two misunderstanding . Firstly , it is commonly assumed that CNNs are ‘ architecturally ’ invariant to translation ( that is , the invariance is built in the architecture through pooling and/or convolution ) . For example : “ [ CNNs ] have an architecture hard-wired for some translation-invariance while they rely heavily on learning through extensive data or data augmentation for invariance to other transformations ” ( Han et al. , 2020 ) , and “ Most deep learning networks make heavy use of a technique called convolution ( LeCun , 1989 ) , which constrains the neural connections in the network such that they innately capture a property known as translational invariance . This is essentially the idea that an object can slide around an image while maintaining its identity ; a circle in the top left can be presumed , even absent direct experience ) to be the same as a circle in the bottom right. ” ( Marcus , 2018 ) , see also LeCun & Bengio ( 1995 ) ; Gens & Domingos ( 2014 ) ; Xu et al . ( 2014 ) ; Marcos et al . ( 2016 ) . In fact , the convolution operation is translationally equivariant , not invariant , meaning that a transformation applied to the input is transferred to the output ( Lenc & Vedaldi , 2019 ) . Even when this point is made , such in LeCun & Bengio ( 1995 ) , is it still assumed that the equivariance is enough to support an important degree of translation invariance . For example , LeCun & Bengio ( 1995 ) write : “ Once a feature has been detected its exact location becomes less important as long as its approximate position relative to other features is preserved ” . As a matter of fact , equivariance and invariance are mutually exclusive functions ( a representation can not support both ) , and accordingly , any invariance supported by a network must be coded into the fully connected part rather than in the equivariant convolutional layers . Moreover , perfect equivariance can be lost in the convolutional layers ( Azulay & Weiss , 2019 ; Zhang , 2019 ) through subsequent sub-sampling ( implemented with pooling and striding operations , commonly used in almost any CNN ) . Therefore , overall , most modern CNNs are neither architecturally invariant nor perfectly equivariant to translation . The other reason for overestimating the extent that CNNs are invariant to translation resides in the failure to distinguish between trained and online translation invariance ( see Bowers et al . 2016 ) . Trained invariance refers to the ability to correctly classify unseen instances of a class in trained location . For instance , a network trained on the whole visual field on identifying instances of dogs , will be able to identify a new image of a dog across multiple locations . This feature is obtained by data-augmentation : jittering the training samples so that the network is trained on items across different locations ( Kauderer-Abrams , 2017 ; Furukawa , 2017 ) . However , this should not be considered a form of translation invariance , as it is simply a case of identifying a test image ( a novel image of a dog ) at a trained location . More interesting is the concept of ‘ online ’ translation invariance : learning to identify an object at one location immediately affords the capacity to identify that object at multiple other1 . Online translation invariance is generally measured by training a network on images placed on a certain location ( generally the center of a canvas ) , and then testing with the same images placed on untrained location . In many reports CNNs performed at chance level on untrained locations ( Kauderer-Abrams , 2017 ; Gong et al. , 2014 ; Azulay & Weiss , 2019 ; Chen et al. , 2017 ; Blything et al. , 2020 ) . This problem has been tackled with several architectural changes : Sundaramoorthi & Wang ( 2019 ) suggested a solution based on Gaussian-Hermite basis ; Bruna & Mallat ( 2012 ) used a wavelet scattering network model ; Jaderberg et al . ( 2015 ) added a new module that can account for any affine transformation ; Blything et al . ( 2020 ) used Global Average Pooling . Without having to resort to new architectures , two works have recently contrasted the previous findings , obtaining a high degree of online translation invariance : Han et al . ( 2020 ) found that a CNN exhibited an almost perfect online translation invariance on a Korean Characters recognition task , but it did not compare these results with the previous literature . Blything et al . ( 2020 ) also found perfect online translation invariance when a VGG16 network was pretrained on ImageNet but found almost no invariance when the same ( vanilla ) network was untrained . This latter finding explains Han et al . results , as they also used a pretrained network when assessing online translation invariance . Together these results hint to the fact that translationally invariant representations do not need to be built inside the network architecture , but can be learned . In the current work , we further explore this idea . 2 CURRENT WORK . In this work we focus on ‘ online ’ translation invariance on a classic CNN , using VGG16 ( Simonyan & Zisserman , 2014 ) as a typical convolutional network . We show how , even though classic CNNs are not ‘ architectural ’ invariance , they can ‘ learn ’ to be invariant to translation by extracting latent features of their visual environmenta ( the dataset ) . Learning is used in the sense that the invariance is coded within the network weights , optimized through backpropagation , and not hard-wired in the network architecture , such as Sundaramoorthi & Wang ( 2019 ) or Bruna & Mallat ( 2012 ) . 1 ‘ Trained translation invariance ’ in which images can be identified across the canvas after training exemplar images across many locations is not to be confused with our approach of ‘ training ’ translation invariance , in which a network is trained to exhibit ‘ online ’ translation invariance We trained on environments in which the key characteristic was that items ’ categories were independent on their position ( for CNNs learning the categories based on their position , see Semih Kayhan & van Gemert 2020 ) . Our main contribution is finding that by pretraining on such environments , CNNs would indeed learn to be invariant to translation . Why is this important ? First , it is important to know how CNNs work , and there is currently confusion about how and when CNNs support translation invariance . Second , a network that learns deep characteristic of its visual environment such as being invariant to translation , rotation , etc. , is able to accelerate subsequent training , and accordingly , it is important to understand the conditions that foster invariance . In addition , because CNNs have been recently suggested as a model for the human brain ( Richards et al. , 2019 ; Ma & Peters , 2020 ; Kriegeskorte , 2015 ; Zhuang et al. , 2020 ) , it is important to understand if and how they can learn fundamental perceptual properties of human vision , of which invariance to translation is one ( Blything et al. , 2020 ; Bowers et al. , 2016 ; Koffka , 2013 ) . 3 OVERVIEW OF THE EXPERIMENTS . Blything et al . ( 2020 ) found that using a VGG16 network pretrained on ImageNet would result in an almost perfect translation invariance on a different dataset in which items were trained only on one location . They compared these results with a vanilla network , that is a non-pretrained network , which showed a lack of translation invariance . Here we replicate these findings with a wider variety of datasets ( Section 3.2 ) . We then show that it is possible to obtain similar results using much simpler artificial datasets in which objects were fully-translated across the canvas , but with some limitations due to the difference between the pretraining and the fine-tuning datasets ( Section 3.3 ) . As a sanity check , we showed that pretraining on the whole canvas is not enough to acquire translation invariance , but the network must be pretrained on fully-translated objects ( Section 3.4 ) . Next , we report studies that assess translation invariance learned from partial information in the environment , that is , whether invariance can be generalized to the whole visual field when the network was only trained on one area of the visual field , and whether invariance extends to all classes when only a subset of classes was trained at all locations ( Section 3.5 ) . We found that generalization failed in these cases . Finally , we report a cosine similarity analysis that provides some insight into the learned internal network representations ( Section 3.6 ) , and shows that the poor performance on some conditions in Section 3.3 was likely due to catastrophic forgetting/interference . 3.1 DATASETS . We used six datasets spanning a high range of complexity . From the more complex to the less complex2 , we used : EMNIST ( Cohen et al. , 2017 ) ; FashionMNIST ( FMNIST ) , from Xiao et al . ( 2017 ) , Kuzushiji MNIST ( KMNIST ) , from Clanuwat et al . ( 2018 ) ; MNIST , from ( LeCun et al. , 1998 ) ; and two versions of the Leek dataset used in Blything et al . : one contained only 10 images instead of 24 of the original dataset ( Leek10 ) . The other contained only two images from the original dataset ( Leek2 ) , disjointed from the images used for Leek10 . Representative examples ( and , for Leek10 and Leek2 , the entire datasets ) are shown in Figure 1B . We did not apply any data-augmentation to the datasets ( apart translating the items on the canvas , as explained below ) .

Summary: This paper addresses the problem of how convolutional neural networks (CNNs) achieve translation invariance, and the authors argue that this invariance es mostly learned from suitable datasets, rather than a result of the architecture. In particular, ImageNet-pretrained networks have learned to be invariant to translation, and fine tuned. The experiments are performed in MNIST-like datasets evaluating classification performance at different locations. The authors conclude that invariance is achieved when the CNN is trained with the different objects being presented at different locations across the canvas, and that the invariance can be forgotten after subsequent training.