Convolutional Neural Networks are not invariant to translation, but they can learn to be
Authors: Valerio Biscione
Jeffrey Bowers
Venue:  (2021)
Keywords: Invariance
Convolutional Networks
Translation
Internal Representations

Abstract: When seeing a new object, humans can immediately recognize it across different retinal locations: we say that the internal object representation is invariant to translation. It is commonly believed that Convolutional Neural Networks (CNNs) are architecturally invariant to translation thanks to the convolution and/or pooling operations they are endowed with. In fact, several works have found that these networks systematically fail to recognise new objects on untrained locations. In this work we show how, even though CNNs are not 'architecturally invariant' to translation, they can indeed 'learn' to be invariant to translation. We verified that this can be achieved by pretraining on ImageNet, and we found that it is also possible with much simpler datasets in which the items are fully translated across the input canvas. Significantly, simply training everywhere on the canvas was not enough. We investigated how this pretraining affected the internal network representations, finding that the invariance was almost always acquired, even though it was some times disrupted by further training due to catastrophic forgetting/interference. 
 These experiments show how pretraining a network on an environment with the right 'latent' characteristics (a more naturalistic environment) can result in the network learning deep perceptual rules which would dramatically improve subsequent generalization.

Under review as a conference paper at ICLR 2021
CONVOLUTIONAL NEURAL NETWORKS ARE NOT INVARIANT TO TRANSLATION, BUT THEY CAN LEARN TO BE
Anonymous authors Paper under double-bling blind review
ABSTRACT
When seeing a new object, humans immediately recognize it across differ- ent retinal locations: we say that the internal object representation is invariant to translation. It is commonly believed that Convolutional Neural Networks (CNNs) are architecturally invariant to translation thanks to the convolution and/or pool- ing operations they endowed with. In fact, several works have found that these networks systematically fail to recognise new objects on untrained locations. In this work we show how, even though CNNs are 'architecturally invariant" to translation, they can indeed 'lcarn' to be invariant to translation. We verified that this can be achieved by pretraining on ImageNet, and we found that is also possible with much simpler datasets in which the items are fully translated across the input canvas. Significantly, simply training everywhere on the can- vas was not enough. We investigated how this pretraining affected the internal network representations, finding that the invariance was almost always acquired, even though it was some times disrupted by further training due to catastrophic forgetiig/interferenee. These experiments show how pretraining a network on environment with the right 'latent' characteristics (a more naturalistic envi- ronment) can result in the network learning deep perceptual rules which would dramatically improve subsequent generalization.
INTRODUCTION
The equivalence of an objects across different viewpoints considered a fundamental capacity of human vision recognition (Hummel (002). This is mediated by the inferior temporal cortex, which appears provide the bases for scale, translation, and rotation invariance (Tanaka [996] O O'Reilly
& Munakata, 2019). Taking inspiration from biological models (LeCun al.) 1998). Artificial Neural Networks have been endowed with convolution and pooling operations (LeCun et al. 1998
1990 It is often claimed that Convolutional Neural Networks (CNNs) are less susceptible to irrel- evant sources of variation such as image translation, scaling, and other small deformations (Gens & Domingos 2014 Xu al. 2014 LeCun & Bengio 1995 Fukushima 1980). While it is difñ- cult to overstate the importance of convolution and pooling operations in deep learning, their ability make network invariant to image transformations has been overestimated: for example, [oog ct al. (2014) showed that CNNs achieve neither rotation nor scale invariance. Similarly, multiple
studies have reported highly limited translation invariance (Kauderer-Abramt
2017
Gong et al.
2014 Azulay & Weiss 2019
Chen et al. 2017 Blything et al.) 2020).
is important to understand the reason for the misconception regarding the ability of CNNs to be invariant to translation. We believe this is due to two misunderstanding. Firstly, it is commonly assumed that CNNs are 'architecturally" invariant to translation (that is, the invariance is built in the architecture through pooling and/or convolution). For example: "[CNNs] have an architecture hard-wired for some translation-invaiaane data or data augmentation for invariance to other transformations" (Han et al., 2020). and "Most deep learning networks make heavy use of a technique called convolution (LeCun 1989), which constrains the neural connections in the network such that they innately capture property known as translational invariance. This is essentially the idea that an object can slide around an image while maintaining its identity; circle in the top left can be presumed, even absent direct experience) be
while they rely heavily on learning through extensive

---

Under review as a conference paper at ICLR 2021
the same as circle in the bottom right." (Marcus 2018), see also LeCun & Bengio (9955; Gens & Domingos (2014); :: Xu et al. (2014); Marcos et al. (2016).
In fact, the convolution operation is translationally equivariant, not invariant, meaning that trans- formation applied to the input is transferred to the output (Lene & Vedaldi, 2019). Even when this point is made, such in LeCun & Bengio (1995), is it still assumed that the equivariance is enough to support an important degree of translation invariance. For example, [eCCun & Bengio (1995) write: "Once feature has been detected its exact location becomes less important as long as its approximate position relative to other features is preserved". As matter of fact, equivariance and invariance are mutually exclusive functions (a representation cannot support both), and accordingly, any invariance supported by a network must be coded into the fully connected part rather than in the equivariant convolutional layers. Moreover, perfect equivariance can be lost in the convolu- tional layers (Azulay & Weiss 2019 Zhang 2019) through subsequent sub-sampling (implemented with pooling and striding operations, commonly used in almost any CNN). Therefore, overall, most modern CNNs are neither architecturally invariant nor perfectly equivariant to translation.
The other reason for overestimating the extent that CNNs are invariant to translation resides in the failure to distinguish between trained and online translation invariance (see [Bowers et al. 2016). Trained invariance refers to the ability to correctly classify unseen instances of a class in trained location. For instance, a network trained on the whole visual field on identifying instances of dogs, will be able to identify a new image of a dog across multiple locations. This feature is obtained by aata-auugentation: jittering the training samples that the network is trained on items across different locations (Kauderer- Abrrams 2017 Furukawa 2017). However, this should not be consid- ered form of translation invariance, as simply case of identifying a test image (a novel image of a dog) at a trained location. More interesting is the concept of 'online' translation invariance: learning to identify an object one location immediately affords the capacity to identify that object at multiple other[]
Online translation invariance is generally measured by training a network on images placed on certain location (generally the center of canvas), and then testing with the same images placed on untrained location. In many reports CNNs performed at chance level on untrained locations [Kauderer-Abrams] 2017, Gong al. 2014] Azulay & Weiss, 2019. Chen et al. 2017: Blything ct al. 2020). This problem has been tackled with several architectural changes: Sundaramoorthi & Hermite basis; [Bruna & Mallat (2012) used wavelet scattering network model: Jaderberg et al. (2015) added a new module that can account for any affine transformation: Blything et al. (2020) used Global Average Pooling.
Wang (2019) suggested solution based on Gaussian Hluusiann-HHmitte
Without having to resort to new architectures, two works have recently contrasted the previous findings, obtaining a high degree of online translation invariance: HHan et al. (2020) found that CNN exhibited an almost perfect online translation invariance on a Korean Characters recognition task, but it did not compare these results with the previous literature. [Blthhng et al. (020) also found perfect online translation invariance when a VGG16 network was pretrained on ImageNet but found almost no invariance when the same (vanilla) network was untrained. This latter finding explains Han et al.] results, as they also used a pretrained network when assessing online translation invariance. Together these results hint to the fact that translationally invariant representations do not need to be built inside the network architecture, but can be learned. In the current work, we further explore this idea.
CURRENT WORK
In this work we focus on 'online' translation invariance on a classic CNN, using VGG16 (Simonyan & Zisserman 2014) as a typical convolutional network. We show how, even though classic CNNs are not "architectural" invariance, they can 'learn' to be invariant to translation by extracting latent features of their visual environmenta (the dataset). Learning is used in the sense that the invariance is coded within the network weights, optimized through backpropagation, and not hard-wired in the network architecture, such as Sundaramoorthi Wang (2019) Bruna & Mallat 2012).
1 "Trained translation invariance' in which images can be identified across the canvas after training exemplar images across many locations is not to be confused with our approach of "training" translation invariance, in which network is trained to exhibit 'online" translation invariance
2

---

Under review as a conference paper at ICLR 2021
We trained environments in which the key characteristic was that items' categories were indepen- dent on their position (for CNNs learning the categories based on their position, see Semih Kayhan & van Gemert 2020). Our main contribution is finding that by pretraining such environments, CNNs would indeed learn to be invariant to translation.
Why is this important? First, it is important to know how CNNs work, and there is currently con- fusion about how and when CNNs support translation invariance. Second, a network that learns deep characteristic of its visual environment such as being invariant translation, rotation, etc., is able to accelerate subsequent training, and accordingly, it is important to understand the conditions that foster invariance. In addition, because CNNs have been recently suggested as a model for the
human brain (Richards et al. [2019] Ma & Peters 2020; Kriegeskorte| [2015] Zhuang et al. (020),
is important to understand and how they can learn fundamental perceptual properties human vision, of which invariance to translation is one (Blything et al., 2020 Bowers et al. 2016 Kofika 2013
OVERVIEW OF THE EXPERIMENTS
Blything et al. (2020) found that using a VGGI6 network pretrained on ImageNet would result in an almost perfect translation invariance different dataset in which items were trained only on one location. They compared these results with a vanilla network, that is a non non-pretrained network, which showed lack of translation invariance. Here we replicate these findings with a wider variety of datasets (Section (.22) We then show that it is possible to obtain similar results using much simpler artificial datasets in which objects were fully-translated across the canvas, but with some limitations due to the difference between the pretraining and the fine-tuning datasets (Section B.3). As a sanity check, we showed that pretraining on the whole canvas is not enough to acquire translation invari- ance, but the network must be pretrained on fully--rann translatdd objects (Section 3.4.. Next, we report studies that assess translation invariance learned from partial information in the environment, that is, whether invariance can be generalized to the whole visual field when the network was only trained on one area of the visual field, and whether invariance extends to all classes when only a subset of classes was trained at all locations (Section 3.5). We found that generalization failed in these cases. Finally, we report a cosine similarity analysis that provides some insight into the Iearned internal network representations (Section 3.6). and shows that the poor performance some conditions in Section 3.3 was likely due to catastrophic forgettin//mterferenne.
3.1 DATASETS
We used six datasets spanning a high range of complexity. From the more complex to the less com- plex we used: EMNIST (Cohen et al. 2017); FashionMNIST (FMNIST), from Xiao et al. (011)) Kuzushiji MNIST (KMNIST), from Clanuwat et al. (2018); MNIST, from (LeCun et al. 1998); and
two versions of the Leek dataset used in Blything et al. one contained only 10 images instead of 24 of the original dataset (Leek10). The other contained only two images from the original dataset (Leek2), disjointed from the images used for Leek Representative examples (and, for Leek and Leek2, the entire datasets) are shown in Figure B.. We did not apply any data-augmeenattion to the datasets (apart translating the items on the canvas, explained below).
3.2 EXPERIMENT 1: PRETRAINING ON IMAGENET
The experimental design is shown in Figure T we tested VGG16 network either pretrained on ImageNet or not pretrained (vanilla). Both networks were then re- ce-triined on a 1-location dataset, that is, datasets where items from all classes were presented only on one location. The network that was pretrained on ImageNet would therefore use the learned parameters as initialization for the new training with the 1-location dataset (this is commonly referred as fine tue-ttning, Girshick et al. 2014), whereas the vanilla network would start from scratch with Kaiming Initialization (He et al.] 2016) We used Adam optimizer with a fixed learning rate of 0,001. The -location dataset used items from
Forr EMNIST, FashionMNIST, KMNIST and MNIST we based our judgment of complexity on the ben- hcode honn/task/imman
chamrk in and I Leek2 as the casiest to classify due to the lack of intra-class variability, as each class is composed of just one image
classi i ficat ion) We deemed Leck
3

---

Under review as a conference paper at ICLR 2021
the six datasets described in Section 3.1 but with each item resized to 50 50 pixels and placed on the leftmost-centered location on a black canvas 224 x 224. Therefore no translation was used for this dataset. Both networks were then tested on the same items placed across the whole canvas (we did not place the items at the edge of the canvas to avoid cropping). Therefore, the networks tested their ability recognise trained objects unseen locations ('online" translation invariance). We repeated each condition 5 times.
Protrain
Pretrained ImageNet Pretraining
A
Olmagehott
Chance level
Vanils
130
Vanilia VEG15
Accuracyy (%0
B
Train
RYNST
KHNIST
EHNIST
FHMIST
KNNIST
NNIST
Leok10
Levkc?
Z 2
3
5
Leck20
Leek2
O
Cir (tragehaat
100
VGG36
Train
Figure 1: A. Representation of the general experimental design used here and in following experi- ments. B. Samples of the six datasets used here. Leek 10 and Leek2 are shown in their entirety. C. Results from experiment 1. The barplot indicates the average accuracy across the 5 runs. Dashed lines indicates chance, and error lines represents one standard deviation. The heatmaps at the bot- tom show the average accuracy across all classes for each dataset at each location in the canvas (interpolated across 19 x 19 test points used).
The results are shown Figure DC. A network pretrained on ImageNet was able to recognize with high accuracy objects from the 1-location datasets when tested unseen locations, whereas vanilla network was not able to do so (Figure CC. barplot). For the pretrained condition, accuracy seemed to slightly increase with simpler datasets. We better understand the extent of networks' invariance to translation by plotting its accuracy across the whole canvas: we tested the network on items centered across grid of 19 19 points equally distributed upon the canvas. By interpolating the results across the tested area we obtained a heatmap like those shown in FigureTC. bottom. We can see that the case of the vanilla network (bottom row), for all datasets, high accuracy was achieved only at trained location (the leftmost-centere one). In fact, performance started degrading after a displacement around pixels, and was chance at 30 pixels. the other hand, pretrained model (upper row) was extremely accurate everywhere on the canvas.
3.3 EXPERIMENT 2: BEYOND IMAGENET
We hypothesised that the network pretrained on ImageNet learned translation invariance because of the data augmentation performed during pretraining that involved translating subsets of the input images across the visual field of the network. If this is correct, we could hypothetically make network learn to invariant to translation with any simple datasets which the items occur all across the canvas. There obviously other features that network extract from ImageNet that are not related to image translation, but here we will focus on translation only.
We pretrained VGG16 network on fully-translated datasets: items from datasets in Section 3.1 were resized to 50 x 50 pixels and randomly placed anywhere across a 224 x 224 black canvas. Similarly to the previous experiment,
then fine-tuned the network each 1-location dataset, and tested it on the same items, dally-translated The experimental setup is illustrated in Figure ZA Again, if the network has learned to be invariant translation, would be able recognise objects
4

---

Under review as a conference paper at ICLR 2021
everywhere on the canvas, without need to be trained every location. The resulting accuracy, averaged across the whole canvas and across five repetitions for each (pretrain, fine-tune) pair, are shown in Figure 2B. Notice that the accuracy are normalized across datasets that 0 is chance level. In most cases, the networks were able to correctly classify items from the 1-location datasets when seen in untrained locations (unlike in Figure [ where able to perform above chance in untrained locations). However, this is not true for all combination of pretrained and fine-tuned datasets. We observed a pattern in which networks pretrained with a "com- plex" fully-tun uully-translated dataset obtained high performances when fine-tuned on 1 1-location "simple" datasets, but the opposite was not true (for example, pretraining on Leek2 resulted poor perfor- mance when fine-tuned on any other dataset). By probing the networks' internal representation, we show in Section 3.6 how even networks pretrained on simple datasets had acquired translation invariance, but it was then disrupted by fine-tuning. Before that, we want to confirm that networks that acquired translation invariance did so because they experienced fully translated items, and not simply because they had been trained across the whole canvas.
that vanilla network was
A
B
Fine-Tandd (1-location)
Fine Tune
That
88.0
38.3
89.8
78.7
78.3
60.3
83.5
Nomaaliz Accuracy
Probrain
AF
37.5
00.6
98.1
00.6
73.1
72.5
8
36.9
Lack2
25
0.5
100.0
level
Figure 2: A. Experimental design for Experiment 2. B. Results in terms of normalized accuracy (0 random chance, 100 is perfect performance). Standard deviation for each condition shown in Section A.
3.4 EXPERIMENT TRAINING ON THE WHOLE CANVAS NOT ENOUGH
It is conceivable that in the previous experiments, and in similarly designed experiments in the literature (Kauderer-Abree vanilla networks failed show invariance to translation because they were tested on locations where they had not seen any items. In which case, pretrained networks succeeded not because they had acquired the deep property of translation invariance from the visual environment, but simply because they had been trained on the whole canvas. To test this hypothesis we separated the canvas in equilateral areas (88x58 pixels), and within each area, only 2 of the total classes presented. The
2017 Gong et al. 2014 Chen et al. 2017 Blything al. 2020),
EMNIST18 Pretrain
MNIST Fine-Tuning
B
EMNIST18
100
Test
C
Accuracy
B
9
3
Figure 3: Experimental design for Experiment 3 and results. A. After fine-tuning on MNIST, the network did not generalize untrained locations. When tested on the same dataset (EMNIST18), but without using classes segregation, letters were only recognised when presented the area they were trained on, so mean accuracy was low. In both cases, the heatmaps are averaged across all classes.

---

Under review as a conference paper at ICLR 2021
items were randomly centered anywhere within their area (in such a way that part of the object could sometime slightly overlap another area, but the objects were never cropped). Therefore the objects were subjected to limited translation. Once trained with this setup, we fine-tuned the network 1-location dataset and tested the same dataset on the whole canvas (like Section [32 and 3.3 We used the EMNIST dataset with the first 18 categories (EMNIST18) for pretraining, and MNIST for I-location fine-tuning and fully- fully-translated testing dataset, because they were the datasets that most consistently resulted in good translation invariance the previous experiment, Results are shown in Figure 3 We also tested the pretrained network on a trals-traasltted version of EMNISTI8. 18, that is, without class segregation, and without fine-tunie fine-tuning (Figure ()guureBB.. Even though the network was trained on items everywhere on the canvas, it did not acquire the ability to generalize on unseen locations with neither EMNIST18 nor newly trained objects (MNIST). This is strong demonstration that the network needs to be trained on an environment where objects are translated across the whole canvas in order to learn to be invariant to translation, and that simply training on the whole canvas is not enough.
3.5 EXPERIMENT TRAINING ON LIMITED TRANSLATIONS NOT ENOUGH
In the previous experiments we assessed the degree of online translation invariance that a network exhibited with items trained at one location. We showed that network would display translation invariance only if it had been pretrained on a fully-translated dataset. Here we explore alternative visual environments that may allow network to learn this fundamental principle.
In the following conditions, we used characters from EMNIST, and tested on a fully-translated EMNIST dataset. Whereas in the previous experiments we fine-tuned on new dataset, here we are interested in generalizing on the same items the network is trained on, so we do not use fine-tuning (however, testing on fine-tuned dataset examined in Section A.2). We divided the canvas quadrants and treated the upper-right quadrant in a special way in the following three conditions.
In Condition the upper-right quadrant was left empty, and all the other classes were trained on the rest on the canvas. When tested, the network was only partially able to generalize on the untrained area (Figure left).
In Conditions 2 and 3 we explored the possibility that training different classes on different areas would allow the network to recognize the classes outside their training area. Unlike the experiments in Section 3.3 in which new classes were trained at one location after the tully-translated pretraining had occurred, here we train, at the same time, dally-translated classes and partially translated classes.
In Condition 2 trained the letter T to Z of the EMNIST datasets the upper-right quadrant, and the remaining letters on the rest of the canvas. When tested on a fully-translate version that dataset (in which there was class segregation) the network failed: the letters would be recognized only in the respective areas (Figure 4 right).
In Condition we slightly modified the previous experiment by training subset of classes across the whole canvas, while limiting another subset to the upper-right quadrant. We split the experiment in sub-conditions, varying the number of fully-translated classes and limited translated ones. In all sub-conditions the network did not achieve full translation invariance on classes that were presented only on the upper-right quadrant (more details in Section A.2.3 and Figure
Condition
Condition
100
EMNIST
EMNIST
4
9
Q Test
Protrain
Pretrein
Test
All closses
Figure 4: Experimental design and results of condition and Experiment 4. On Condition 1, the heatmap shows certain degree of generalization untrained area, but not complete. On Condition 2, the network only accurately predicted classes on the area in which they were present during training.

---

Under review as a conference paper at ICLR 2021
The overall conclusion from these experiments is that, in order to acquire the property of being invariant to translation, a network needs to be pretrained on fully-translated objects, for all classes. Further details on these experiments, including the results when fine-tuned on new dataset, provided in Section A.2
3.6 EXPERIMENT 5: INTERNAL REPRESENTATION WITH COSINE SIMILARITY ANALYSIS
We found Experiment in Section 3.3 that CNNs would some times not display invariance to simple
translation after fine-tuning. This seemed to happen when they had been pretrained dataset and fine-tuned on complex one, and could be explained by the phenomenon of inter- ference and catastrophic forgetting (Furlanello al. 2016 McCloskey & Cohen 1989) in which training on new tasks degrades previously acquired capabilities. If this was true, it would mean that network could learn be invariant translation with any translated dataset, regardless of its complexity, and this ability could be retained with techniques that prevent catastrophic forgetting (Beaulieu et al. 2020; Javed & White 2019 Li & Hoiem 2016).
In order to test the hypothesis that catastrophic interference may play role, computed the cosine similarity between the activations of the penultimate networks' layers when given as input an image at the leftmost-centered location and an images at different levels of displacement from that location. This metric indicates the degree to which translated objects share the same internal representation (we show the results of applying the cosine similarity analysis across the whole network in Section A.3.2).
We firstly checked that the cosine similarity analysis correctly captured the increasing similarity of internal representation with translated images for a vanilla network, a network pretrained on Ima- geNet, and a network pretrained 3.3). Results are shown in Figure 5A. Notice that the vanilla and pretrained models were never trained on the items used compute the cosine similarity. For the vanilla network, the similarity quickly dropped, indicating that more translated objects have very different internal representations. The other two networks have very high similarity even for highly translated objects. This is an- other confirmation that networks trained in such environments did develop a general ability of being invariant to translation.
the tested dataset (we used the networks trained in Sections 3.2
We then measured the amount of catastrophic forgetting by computing the cosine similarity of each pretrained network (on each dataset Section 3.T) before and after fine-tuning on the 1-location datasets. Results for all conditions, at different experimental stages, are shown in Figure 5B. For most datasets, the cosine similarity was high for every dataset before fine-tuning, even for untrained
A
B
141
1.0
1.
0.9
Cloite Sinlarty
Casine Simianty
0.8
0.8
No suinig
Trained Inapelial
0.7
Trahnd Dataset
0.5
NNIT
Look10
Leo82
125 Hortzontal Translation (pieets)
Thaned dasosots
datasets
Train
Pretrahh
Tine-Ture
eoken Sim. each Dataset ofter training imageliet
Vanilla
VSG15
Cos, Sim. with no training
Figure 5: A. Cosine similarity across several horizontal displacements. different stages of the experimental setup for experiments Section 3.3 figure indicates the different stages which the analysis is performed.
B. Cosine similarity at The bottom part of the
7

---

Under review as a conference paper at ICLR 2021
items. After fine-tuning, however, the cosine similarity would sometime go down to un un-trained level. In Figure 10 in Section A.3.1 we show that the change is consistent with the drop in accuracy
saw in Section 33
DISCUSSION
Our experiments show that translation invariance can be learned in CNNs that lack built-in architec- tural invariance, and raise the possibility that a whole range of perceptual capabilities can be learned rather than built in the architecture. This work also suggest that, certain cases, the architecture may be less important than the visual world the network is trained on. In fact, we verified that a simple fully-connected network (without convolutions) can indeed learn translation invariance in some conditions (see Section A.4 in the Appendix).
Neural networks performance is often compared human performance (Baker et al. 2018 Han it al., 2019; Srivastava et al. 2219 Ma & Peters, 2020). A fundamental feature of human per- ception is that it supports widespread generalization, including combinatorial generalization (e.g., identifying and understanding images composed of novel combination of known features). Current CNNs are poor at generalizing to novel environments (Geirhos et al. (2018), especially when combi- natorial generalization is required (Vankov & Bowers 2020; Hummel 2013). Here we have shown that CNNs are able to extract latent principles of translation invariance from their visual world and to re-use them to identify novel stimuli with very different visual forms in untrained locations. An interesting question the extent to which other forms of generalization and other fundamental prin- iaaaatiii,,ookaaaaa Koffka 2013) be learned in standard CNNs trained on the appropriate datasets, and what sorts of generalization requires architectural in- novations. Without the right training environment, is not surprising that CNNs fail to capture the cognitive capacity of the human visual system (Funke et al. 2020), and the only way to address this fundamental question is to train models under more realistic conditions.
ciples of perception (e.g.. Gestalt principles of organization,
Although training on more naturalistic datasets may lead to better and more human- like forms of generalization, is also worth highlighting how our artificial hand-crafted environments, such as our fully-translated datasets, could be used to train networks to acquire particular perceptual regularity in spite of the architecture not directly supporting that representation. This may prove to be a useful technique for learning in more complex environments: Instead of having a network learning all possible visual configurations (through data augmentation) of a given dataset, the network can be pretrained extremely simple datasets that embed fundamental perceptual principles of the environment. When facing a complex dataset, the network only needs to learn the basic configuration of the new objects, and extrapolate the others through learned perceptual properties. This approach would clearly need to address the problem of catastrophic forgetting since, as we have seen, it can seriously impair generalization of learned perceptual rules when applied new objects (Li & Hoiem] 2016: Furlanello et al. 20166. This approach would resemble Curriculum learning (Bengio al.) 2009), but instead of progressively learning new classes, the network would progressively learn new perceptual properties. We are not aware of any study adopting this approach.
CONCLUSION
We have shown that, even though standard CNNs are not architecturally invariant to translation, they can learn be by training on dataset that contains this regularity. We have also shown how such property is retained when fine-tuning on simpler dataset, but lost for more complex ones, and this appears to reflect the role of catastrophic interference in constraining translation invariance rather than a failure to learn invariance from these simple datasets. More generally, we suggest that by training artificial networks with non-naturalistic datasets, we are constraining their ability to learn deep fundamental rules of perception. By using more naturalistic environments, and an approach that would avoid catastrophic forgetting, we could unlock a greater capacity of generalization.
ACKNOWLEDGMENTS
We would like to thank Gaurav Malhotra for helpful comments and suggestions.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize so poorly to small image transformations? Journal of Machine Learning Research, 20:1-25, 2019.
Nicholas Baker, Hongjing Lu, Gennady Erlikhman, and Philip J. Kellman. Deep convolutional networks do not classify based on global object shape. PLoS Computational Biology, 14(12): 1-43, 2018.
Shawn Beaulieu, Lapo Frati, Thomas Miconi, Joel Lehman, Kenneth O. Stanley, Jeff Clune, and Nick Cheney. Learning to Continually Learn. arXiv preprint arXiv: 2002.09571, 2020.
Yoshua Bengio, Jerome Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In ACM International Conference Proceeding Series, volume 382, pp. 1-8, New York, New York, USA, 2009. ACM Press.
Ryan Blything, Valerio Biscione, Ivan Vankov, Casimir H. Ludwig, and Jeffrey S. Bowers. Extreme on-line translation tolerance in human and machine object recognition. arXiv preprint arXiv: 2009. 12855, 2020.
Jeffrey S. Bowers, Ivan Vankov, and Casimir J.H. Ludwig. The visual system supports online translation invariance for object identification. Psychonomic Bulletin and Review, 23(2):432-455 2016.
Joan Bruna and Stéphane Stephane Mallat. Invariant Scattering Convolution Networks, IEEE Trans- actions on Pattern Analysis and Machine Intelligence, 35(8):187--1886, 3 2012.
Francis X. Chen, Gemma Roig, Leyla Isik, Xavier Boix, and Tomaso Poggio. Eccentricity dependent deep neural networks: Modeling invariance in human vision. AAAI Spring Symposium Technical Report, SS- -:541-546, 2017.
Dan Claudiu Cireşan, Ueli Meier, Luca Maria Gambardella, and Jürgen Schmidhuber. Deep, big, simple neural nets for handwritten digit recognition. Neural Computation, 22(12)3307-3322, 2010.
Tarin Clanuwat, Mikel Bober-Irizar, Asanobu Kitamoto, Alex Lamb, Kazuaki Yamamoto, and David
Ha. Deep Learning for Classical Japanese Literature. arXiv preprint arXiv: 1812.01718, 12 2018.
Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. EMNIST: an extension of MNIST to handwritten letters. arXiv preprint arXiv: 0702.05373, 2 2017.
Kunihiko Fukushima. Neocognitron: A self-organizing neural network model for mechanism of pattern recognition unaffected by shift in position. Biological Cybernetics, 36(4):193-202, 1980.
Christina M Funke, Judy Borowski, Karolina Stosio, Wieland Brendel, Thomas S A Wallis, and Matthias Bethge. The Notorious Difficulty of Comparing Human and Machine Perception. arXiv preprint arviv: 2004. 1.04.09406v1. 2020.
Tommaso Furlanello, Jiaping Zhao, Andrew M. Saxe, Laurent Itti, and Bosco S. Tjan. Active Long Term Memory Networks. arXiv preprint arXiv: 1606. 02355, 6 2016.
Hidetoshi Furukawa. Deep Learning for Target Classification from SAR Imagery: Data Augmenta- 8 2017.
tion and Translation Invariance. arXiv preprint arXiv:
Robert Geirhos, Carlos R Medina Temme, Jonas Rauber, Heiko H Schütt, Matthias Bethge, Fe- lix A Wichmann, Carlos M Temme, Jonas Rauber, Heiko Schütt, Matthias Bethge, and Fe- lix A Wichmann. Generalisation in humans and deep neural networks. In S Bengio, H Wallach, H Larochelle, K Grauman, Cesa-Bianchi, and Garnett (eds.), Advances Neural Information Curran Associates, Inc., 2018.
Processing Systems 31, pp. 7538-7550.
Robert Gens and Pedro M Domingos. Deep Symmetry Networks. In Z Ghahramani, M Welling, Cortes, D Lawrence, and K Q Weinberger (eds.), Advances in Neural Information Processing Systems 27, pp. 2537-2545. Curran Associates, Inc., 2014.

---

Under review as a conference paper at ICLR 2021
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accu- rate object detection and semantic segmentation. In Proceedings of the IEEE Computer Society Conference Computer Vision and Pattern Recognition, pp. 580-587. IEEE Computer Society, 2014.
Yunchao Gong, Liwei Wang, Ruiqi Guo, and Svetlana Lazebnik. Multi-scale Orderless Pooling of Deep Convolutional Activation Features. Lecture Notes Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 8695 LNCS(PART 7):392 407,3 2014.
Chihye Han, Wonjun Yoon, Gihyun Kwon, Seungkyu Nam, and Daeshik Kim. Representation of White- and Black-Box Adversarial Examples in Deep Neural Networks and Humans: A Func- tional Magnetic Resonance Imaging Study. arXiv preprint arXiv: 1905.02422, 2019.
Yena Han, Gemma Roig, Gad Geiger, and Tomaso Poggio. Scale and translation for novel objects in human vision. Scientific Reports, 10(1):1-13, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, volume 6eleee, pp. 770-778. IEEE Computer Society, 12 2016.
JE Hummel. Dynamic Binding in a Neural Network for Shape Recognition John E. Hummel and Irving Biederman. Psychological review, 99(3):480-517, 2002.
John E Hummel. Object Recognition. In Oxford library of psychology. The Oxford handbook of cognitive psychology, number August, pp. 1-16. 2013.
Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu. Spatial transformer networks. Advances in Neural Information Processing Systems, 1005:Ja:22011-22055, 2015.
Khurram Javed and Martha White. Meta-Learning Representations for Continual Learning. In H Wallach, H Larochelle, A Beygelzimer, F d textquuuteee Alché-Bus Mucé-Bucc, E Fox, and R Gamett (eds.), Advances in Neural Information Processing Systems 32, pp. 1820-1830. Curran Asso- ciates, Inc., 2019.
Eric Kauderer-Abrams Quantifying Translation-Invarianee in Convolutional Neural Networks. arXiv preprint arXiv: 1801.01450vl, 12 2017.
K. Koffka. Principles of gestalt psychology. Taylor and Francis, 2013.
Nikolaus Kriegeskorte. Deep Neural Networks: A New Framework for Modeling Biological Vision and Brain Information Processing, Annual Review of Vision Science, 1(1):417-446, 11 2015.
Yann LeCun. Generalization and network design strategies. 143-155, 1989.
Connectionism in perspective, 19:
Yann LeCun and Yoshua Bengio. Convolutional Networks for Images, Speech, and Time-Series. The handbook of brain theory and neural networks, 3361, 1995.
Yann LeCun, Bernhard E Boser, John Denker, Donnie Henderson, R E Howard, Wayne E Hub- bard, Lawrence D Jackel, Le Cun, Jackel Henderson, Y Le Cun, John S Denker, Donnie Hender- son, R E Howard, Wayne E Hubbard, and Lawrence D Jackel. Handwritten Digit Recognition with a Back-Propagation Network. In S Touretzky (ed.), Advances in Neural Information Pro- cessing Systems 2, pp. 396-404. Norgan-Kaufmann 1990.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEFE, 86(11):2278-2323 1998.
Karel Lenc and Andrea Vedaldi. Understanding Image Representations by Measuring Their Equiv- ariance and Equivalence. International Journal of Computer Vision, 127(5)456-476,5 5 2019.
Zhizhong Li and Derek Hoiem. Learning without forgetting. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 9908 LNCS, pp. 614-629. Springer Verlag, 2016.
10

---

Under review as a conference paper at ICLR 2021
Wei Ji Ma and Benjamin Peters. A neural network walks into lab: towards using deep nets as models for human behavior. arXiv preprint arXiv: 2005.02181, pp. 1-39, 2020.
Diego Marcos, Michele Volpi, and Devis Tuia. Learning rotation invariant convolutional filters for texture classification. Proceedings International Conference on Pattern Recognition, volume 0, pp. 2012 2017. Institute of Electrical and Electronics Engineers Inc., 1 2016.
Gary Marcus. Deep Learning: Critical Appraisal. arXiv preprint arXiv: 1801.00631, 2018.
Michael McCloskey and Neal J. Cohen. Catastrophic Interference in Connectionist Networks: The Sequential Learning Problem. Psychology of Learning and Motivation Advances in Research and Theory, 24(C):109-166) 1989.
Randall C. O'Reilly and Yuko Munakata. Computational Explorations in Cognitive Neuroscience. Computational Explorations in Cognitive Neuroscience, 2019.
Blake A. Richards, Timothy Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, Colleen J. Gillon, Danijar Hafner, Adam Kepecs, Nikolaus Kriegeskorte, Peter Latham, Grace W. Lind- say, Kenneth D. Miller, Richard Naud, Christopher C. Pack, Panayiota Poirazi, Pieter Roelfsema, João Sacramento, Andrew Saxe, Benjamin Scellier, Anna C. Schapiro, Walter Senn, Greg Wayne, Daniel Yamins, Friedemann Zenke, Joel Zylberberg, Denis Therien, and Konrad P. Kording, A deep learning framework for neuroscience. Nature Neuroscience, 22(11):1761-1771 2019.
Osman Semih Kayhan and Jan van Gemert. On Translation Invariance in CNNs: Convolutional Layers Can Exploit Absolute Spatial Location. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), number class 1, pp. 14262-14273, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1556, 2014.
Sanjana Srivastava, Guy Ben- Yosef, and Xavier Boix. Minimal Images in Deep Neural Networks: Fragile Object Recognition in Natural Images. 7th International Conference on Learning Repre- sentations, ICLR 2019, 2 2019.
Ganesh Sundaramoorthi and Timothy Wang. Translation Insensitive CNNs. arXiv preprint arXiv: 1911. 2019.
Keiji Tanaka. Inferotemporal Cortex and Object Vision. Annual Review of Neuroscience, 19(1): 109-139, 3 1996.
Ivan I. Vankov and Jeffrey Bowers. Trainin Neural Networks to encode symbols enables com- binatorial generlization. Philosophical Transactions of the Royal Society B: Biological Sciences, 375(1791):20190309, 2 2020.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-MNIST: a Novel Image Dataset for Bench- marking Machine Learning Algorithms. arXiv preprint arXiv: 1708.07747, 2017.
Yichong Xu, Tianjun Xiao, Jiaxing Zhang, Kuiyuan Yang, and Zheng Zhang. Scale-Invariant Con- volutional Neural Networks. arXiv preprint arXiv: 1411.6369, 11 2014.
Richard Zhang. Making Convolutional Networks Shift-Invariant Again. 36th International Confer- ence on Machine Learning, ICML 2019, 2019-JJme:12711-11272. 2019.
Chengxu Zhuang, Siming Yan, Aran Nayebi, Martin Schrimpf, Michael C. Frank, James J. DiCarlo, and Daniel L. K. Yamins. Unsupervised Neural Network Models of the Ventral Visual Stream. bioRxiv, pp. 2020.06.12 1.....6..555 16.15556..... 2020.
11

---

Under review as a conference paper at ICLR 2021
A APPENDIX
A.1 BEYOND IMAGENET
In Section 3.3 we pretrained on the fully-translated version of each one of the the six datasets de- scribed in Section 3.1 We then fine-tuned the network each 1-location dataset, and tested it the same items, fully-translated, We repeated each training session 5 times. In Figure we show the average performance across each repetition. Here we also show the standard deviation for each one of the 36 conditions (Figure
Chance
100
sSS Pretraining:=ine-tumiii tuning
Accuracy (%)
80
Fine-tuned
and tested on:
EMNIST
FMNIST
40
KVNIST
MNIST
Leek10
Leek2
Pretrained on:
EMNIST
FMNIST
KMNIST
MNIST
Leek10
Leek2
Figure 6: Average results across each repetition for Experiment 1. Error lines indicate one standard deviation. Hatches indicate the condition in which fine-tuning had the same items as pretraining (corresponding to the diagonal in Figure 2
A.2
TRAINING ON LIMITED TRANSLATIONS: FURTHER TESTS
A.2.1 CONDITION I: INTER-CLASS ANALYSIS
We showed that CNN was only partially able to generalize untrained locations (Section 3.5 However, we noticed a certain degree inter-class variability: even though in most cases the results similar to the average shown in Figure 3 for few EMNIST letters the accuracy was high all across the canvas. We show in Figure 7 the accuracy across the canvas for each class, noticing that perfect translation invariance not clear why there was this difference across classes. Notice that
achieved for letter E and 7,, and several other classes reached an
almost perfect invariance. we could find much lower degree of inter-class variability for the Condition 2, and that analysis is therefore not shown here.
A.2.2 CONDITION AND 2: FINE-TUNING
In Conditions 1 and 2 in Section 3.5
trained network with EMNIST items by translating them everywhere across the canvas apart from the upper-right quadrant. We then tested the network on the same dataset it was trained (EMNIST), but fully translated. In this section, with a experimental design similar to that used in Section (3.3 that is, by training on a new 1-location dataset and testing on this fully-translated dataset.
show what happens
Condition 1, after having pretrained on the EMNIST dataset leaving empty the upper-right quadrant, we fine-tuned the network on l-location MNIST and tested it a fully-translated MNIST. As shown in Figure [8] left, obtained very similar results to those in Figure that is: the network showed translation invariance on locations all across the pretrained area, but only partial invariance on locations in which it had never been trained. We also analysed the inter-class variability finding that, like in Figure 7 the network showed perfect translation invariance for some of the classes in the MNIST dataset.
we filled the upper-right quadrant with subset of EMNIST classes, showing that the network learned to selectively recognise classes only on the trained location. this section, we show what happened when fine-tuning the network on shown in Figure right. We can see that the network had retained some kind of bias in that it would incorrectly classify items presented the upper-right quadrant, but the separation as before. As usual, we observed some inter-class variability regarding the network accuracy on the left-out quadrant, but in this case none of the classes reached perfect translation invariance.
In Condition 2 in Section 3.5
1-location dataset. This is
not clear
12

---

Under review as a conference paper at ICLR 2021
Condition 1
EMNIST
1860 Cccooae
W
Figure 7: Hcatmaps for each EMNIST class for the Condition 1 of Experiment 3. The network seem to be perfectly invariant to translation for some classes.
EHINST
Fin-Taning
NNIT
MNIST
ENNIST
Aini-uning
Tixt
All MINIST classes
100
Securase S (%)
Condition 1
Condition 2
Figure 8: Condition and from Experiment Here, instead of testing on the pretrain dataset, fine-tuned on a new dataset (MNIST) and tested on the fully-translated version of it. We also show the heatmap for each class, showing discrete degree of inter-class variability and perfect translation for some classes.
A.2.3 CONDITION 3: MORE DETAILS
In this section we provided more details about the Condition 3 briefly described in Section 3.5 We
used dataset structured in this way: subset of classes from EMNIST was placed anywhere the canvas (fully-translated)); the remaining classes were only placed on the right-upper quadrant. We varied the number of classes trained on the whole canvas the following way: for Condition 3.1, only the class corresponding to the letter A was trained on the whole canvas; for Condition 3.2, J; for Condition 3.3, the classes from A to T; from Condition 3.4, the classes from A-Y. Therefore the amount of classes trained on the whole canvas, for each sub-condition, 1, 10, 20, and 25. We then tested the network on the same EMNIST dataset, but this time all classes were fully translated. The relevant measure here is the accuracy for the classes presented only on the right-upper quadrant, when then tested on the whole canvas, and in particular whether the network would be able accurately classify those classes when presented the area of the canvas in which they were not trained on. The results are shown in Figure 9 We analysed separately the results from the classes. The interplay between these groups (fully-translated and partially-translane is
the classes from
13

---

Under review as a conference paper at ICLR 2021
non-trivial, with the Condition 3.1 showing almost perfect translation invariance for the letter A and the condition 3.2 showing a slightly degraded performance for the fully-translater letters A-J on the right-upper quadrant. However, across all sub-conditions, it is clear that the letters trained on the right-upper quadrant were never recognized when presented elsewhere
Condition
EMNIST
100
Ar
Accuracy (1)
3.3
U-2
Protrain
Figure 9: Condition 3 for Experiment 3. This condition was split in 4 sub-conditions, with increasing
number of letters trained the whole canvas. Heatmaps shown for each group.
A.3 COSINE SIMILARITY ANALYSIS
A.3.1 BEFORE AND AFTER FINE-TUNING FOR EACII PRETRAINING DATASET
We showed in Section 3.6 Figure the grouped results of the cosine similarity analysis before and after tine-tuning In Figure I0 we show the results for networks pretrained on each datasets. These results very consistent with the accuracy results shown in Section 3.3
A. A.I networks pretrained on complex dataset would mostly retain a high degree of cosine similarity after fine-tuning with most of the datasets, meaning that the translation invariance property was preserved. In other cases, and really strikingly with Leekl0 10 and Leek2, the cosine similarity dropped after fine-tuning with all the other complex datasets. Significantly, high degree of cosine similarity was observed before the fine-tuning, for all datasets, and for networks pretrained on any dataset. This implies that, for example, a network pretrained on fully-translater Leek2 dataset (an extremely simple and fast-to-learn dataset) would acquire certain degree of translation invariance even with more complex datasets, such as EMNIST. This property could be accessed if catastrophic forgetting were to be ameliorated during fine-tuning, by using method that would exploit the network internal representation itself.
Pretrained on EMNIST
Pretrained on FMNIST
Pretrained on KMNIST
EMNIST
FMNIST
Cosine Simil
KMNIST
Pretrained MNIST
Pretrained Leek10
Pretrained on Leek2
MNIST
Leek 10
Leek2
Before fine-tu tree-tuning
After fine-tuning
Before fine- fine-tuning
After fine-tuning
Before fine-tuning
After fine-tuning
Figure 10: Cosine similarity analysis before and after 1-location fine-tuning, for each pretraining fully-translated dataset, showing that in almost every condition the network acquired translation invariance, but it was some times disrupted by fine-tuning.
14

---

Under review as a conference paper at ICLR 2021
on ImageNet
Varilla VGG16
VGG16
Cos. Sim, MNIST
Cos. Sim. on MNIST
68
Dispi reglacoment- 7px
Casina Sinslaity
17px
SFpx
Displacoment
p7px
17px
3/v
27pM
60
Conv. Layers
Linear Layers
Conv. Layers
Uneer Layers
Figure 11: Cosine Similarity value across the whole network, for several displacements, for the vanilla network and network pretrained on ImageNet.
A.3.2 CONVOLUTION AND TRANSLATION INVARIANCE
We have argued the Discussion that the convolution operation is responsible for coding the property of translation invariance. Theoretically, this is due to convolution being equivariant rather than invariant. Practically, we can measure this by computing the cosine similarity across both the convolutional layers and the fully connected layers (whereas the cosine similarity in Section was was performed on penultimate layer, so a fully-connected one). We ran the analysis on a vanilla network and network pretrained on ImageNet (Figure 11.. This plot shows three things: 1) as expected, as displacement increases, the convolutional layers encode less and less of the similarity between the translated images. For the pretrained network, after 37 pixels, most of the similarity seems to be captured uniquely by the fully connected part networks. 2) As seen in Section 3.6 there seems to be very small amount of translation tolerance even for vanilla network. The similarity is high for translations until about 27 pixels. This may be due to some relationship with pooling, convolution, and the size of the items used (50 pixels). 3) Cosine Similarity on the
Train on ImageNet
Vanilla VGG16
Cos. Sim. for each Dataset after training on ImageNet
Cos. Sim. For each Dataset with no training
Dataset EMNIST
No training
Dalaset: FMNIST
Trained ImageNet
0.8
Dataset: KMNIST
Dataset: MNIST
Displacement-
Dataset Leek10
0.6
Casine Similarity
0.4
0.0
Conv. Layers
Linear Layers
Figure 12: Cosine Similarity values across the whole network, at the most extreme displacement, for the vanilla network and network pretrained on ImageNet.
15

---

Under review as a conference paper at ICLR 2021
convolutional layers seem much more pronounced the vanilla than the pretrained network. seems that pretraining makes the convolution less involved in translation invariance, pushing the encoding onto the fully-connected layer. Overall, noticed that the convolutional layers may be responsible for a limited degree of translation invariance. However, for larger displacement, the translation invariance
to be completely encoded into the fully connected layers, shown in Figure 12 with the maximum displacement of 199 pixels.
A.4 TRAINING A FULLY CONNECTED NETWORK
If, as argued in the Discussion and in Section A.3.2 convolutions are not responsible (or mini- mally responsible) for translation invariance, it should be possible train fully connected network to acquire translation invariance. We chose the architecture tested Cireşan et al. (2010), which proved able to obtain 0.35% test
the MNIST dataset. The architecture has 4 hidden layers of (2500, 2000, 1500, 1000, 500) units, and ReLu was applied after each layer. We pretrained the network with fully-translated dataset: used MNIST and Leck10 (without any data augmen- tation). We trained until convergence (corresponding to ~99% accuracy). We then fine-tuned the network I-location datasets and tested fully-translated version of that dataset. The network pretrained on MNIST was fine-tuned and tested on Leek 10 and Leek2. The network pretrained on Leck10 was fine-tuned and tested on MNIST and Leck2. We ran the experiment 5 times for each condition.
The results are shown in Figure 13 We observed that purely fully-connected network only showed translation invariance when pretrained on Leek10 10 and fine-tuned on Leek2. The lack of invariance when pretrained on Leek2 or 10 and fine-tuned on MNIST can be explained by catastrophic interfer- ence (Section 3.6) due to fine-tuning the lack of invariance when pretraining MNIST and fine-tuning on Leek10 puzzling. ble that fully-connected network is much more prone to catastrophic interference compared to CNN due to lack of weight sharing. Therefore, even though may be possible for a fully- fully-connected network learn translation invariance, and even though the convolutional layers not directly responsible for this ability, the convolution operation is clearly a major facilitator in transferring this ability new dataset.
more complex dataset than the pretrained one. However possi-
Fine-luned
100
Fire-laned
Accurecy
Rinaeluned
Fine-tuned MNET
Figure 13: Accuracy of a fully-connected network pretrained on fully-translated MNIST or Leck and fine-tuned on l-location MNIST, Leck Leck2. The network pretrained on Leek was able to exhibit translation invariance for Leek2, but the other combinations failed, probably due to catastrophic interference being more problematic for non-convolutional network.
16


Reviews:
[{"review_id": "bTn2iBdTLRs", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper addresses the problem of how convolutional neural networks (CNNs) achieve translation invariance, and the authors argue that this invariance es mostly learned from suitable datasets, rather than a result of the architecture. In particular, ImageNet-pretrained networks have learned to be invariant to translation, and fine tuned. The experiments are performed in MNIST-like datasets evaluating classification performance at different locations. The authors conclude that invariance is achieved when the CNN is trained with the different objects being presented at different locations across the canvas, and that the invariance can be forgotten after subsequent training.\n\nPros:\n- The experiments explore several settings and are convincing in clearly showing that translation invariance requires that the network observes the different objects translated across the canvas, in their particular setting (although I have some concerns about the setting).\n- Understanding how neural networks achieve certain properties, such as translation invariance, is very relevant.\n- The paper is well written and can be followed easily. I like particularly the ilustrations of the experiments.\n\n\nCons:\n- The main concern I have is that the insights are relatively incremental. The experimental setting replicates Blything et al 2020 and the main insight of CNNs can learn translation invariance from suitable large and diverse datasets such as ImageNet was already shown in that paper (note that, although available in arxiv in Sept 2020, the authors are aware of the work, since they state that they use Bything et al.'s dataset and replicate their experiments). The results in the submission are not showing significantly novel insights.\n- Results are shown with small datasets (MNIST-like), but not clear how they extrapolate to more complex one. It is also not clear to me that is possible to train a heavy model such as VGG16 with such small resolution datasets, even when they are translated to different locations. It probably results in very significant overfitting.\n- Only evaluated on VGG16. To be more convincing in the general claim, it is necessary to also evaluate other models. \n- The authors only analyze translation invariance of the whole network. It would be more interesting to analyze invariance of the different layers via intermediate representations. Experiment 3 for instance, encourages local translation invariance (within each quadrant), but not across the whole canvas. I would expect that higher layers still behave like Vanilla VGG16 which overfits to the location, while lower layers show higher level of invariance. \n- Fine tuning the whole network (I understand the authors train/fine tune all the layers) in this setting is probably leading to significant overfitting and therefore to catastrophic forgetting. The authors should consider the case where only the classifier is trained (and a variable number of layers in the top), and thus less prone to overfitting and avoid forgetting in lower layers, to further assess the invariance in different layers.\n\nQuestions\nPlease clarify cons.\n\nMinor comments\nSome figures seem to suggest that images have multiple objects in multiple locations. My understanding is that every image has only one object, and the location can change, so those figures may be misleading. Please clarify and modify the figure if necessary.", "extra_scores": {}, "extra_text": {}}, {"review_id": "UE_HQ5hvq2C", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper analysis and studies translation invariance in convolution neural networks. It argues that typically it is claimed that CNNs are translation invariant due to the convolution function, and that actually convolution are equivariant. While pooling is the actual function that gives local invariance (or global when the pooling is across all locations), this is not included in the description of invariance. One neural network, VGG-16, is used for the analysis in different scenarios: 1) pre-trained on Imagenet and fine-tuned to the new dataset on one location 2) trained from scratch using the new dataset in one location 3) trained from scratch using the new datasets in all locations of the canvas and test on the other datasets. The main conclusion in the paper is that CNNs are not invariant to translation by design of the architecture, but that when pre-trained on naturalistic images, they can be. \n\n\nPositive points:\n\n- the research question is important and this kind of analysis to understand CNNs are crucial to better understand network capabilities and being able to predict their behaviour.\n\n----\nCOMMENT AFTER REBUTTAL PERIOD:\nGiven that there was no rebuttal, I keep my initial rating.\n\n- study how using pre-trained networks affect the generalization in some factors, e.g. translation invariance in this paper, is interesting and novel.\n\n- The paper is well written and easy to follow.\n\n\n\nConcerns: \n\n- The main concern is about the experimental set-up and results presented in the paper. Only one network is used for the study, to validate if it is indeed the architectural design the gives translation invariance. It is a weak statement if other CNNs architectures with different configurations (number of layers, amount of local pooling, if global pooling is used, the effect of strides, etc ) analyzed, and the current observations might only apply to VGG-16.\n\n- The results in figure 2 B when comparing the translation invariance across the different datasets, it is mentioned that depending on the dataset that the network is initialized from it brings more or less translation invariance. A deeper analysis on the amount of data used, the similarity between the objects across datasets, and how this effect the final performance would be nice to include. Here talking about invariance is a bit confusing, and it might be that this robustness to position changes is more related the generalization properties to the other datasets due to the resemblance of the objects, and obtained by experience. \n \n- It would have been nice to see the behaviour in another transformation as well, since it would strengthen the claim that the invariance (I would say robustness) to transformations is not due to the architecture, but to previous exposure to naturalistic images, and if it is not, it would bring some light into why it is for translation and not for other transformations.\n\n\n\n \nMinor comments: \n \n* As a side comment, since there is some motivation in the introduction relating to human visual processing, humans have a loss of acuity (recognition performance) with distance to the focal point, and perceive high accuracy in the fovea a low-accuracy in the periphery, which is not captured by typical CNNs. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "2jXZSPp1_W", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper examines the source of translational invariance in CNNs and points out that the convolution operation is translationally equi-variant and not invariant. Authors thoroughly examine how training and architecture contribute to translational invariance in CNNs. My main concern about this paper is with respect to its novelty. While the experiments are done in a thorough way, the main point of this paper (that translation invariance is formed during training) is mostly known to the community. The equi-variance property of convolution operation is discussed in machine/deep learning textbooks (e.g. [2]) and the importance of image-augmentations like spatial jitter on network object recognition performance has repeatedly been demonstrated. Because of this reason I don’t think this paper would be suitable for publication at ICLR. \n\nOther comments: \n* The contribution of the paper is very unclear from the abstract, even getting past the first two sections I was still left wondering what I should be expecting to see in the rest of the paper. \n* The text needs more proofing reading, more than few typos and misuse of words. E.g. bases —> basis; human vision recognition —> human visual (object) recognition\n* LeCun et al. 1998 is cited as a biological model which is not a good example of a biological neural network. Although CNNs have some commonalities with biological neural networks they have many more differences. [1] might be a better reference to an early biologically inspired neural net\n* It is claimed that “CNNs achieve neither rotation nor scale invariance”. However this is not a binary property. What matters is the degree of invariance that could be measured. \n* section 3: by \"non-pretrained network” do you mean the untrained network?\n* It is unclear from the text under section 3 what “pretraining on the whole canvas” is\n* the 1-location dataset is not described anywhere in the paper and I had to go by a guess as to what this dataset contains. \n* in section 5, the use of the cosine similarity measure instead of the accuracy which was used in the previous 4 experiments is not well motivated. If this is a better measure why not using it in all experiments. \n\n\n[1] Fukushima, K. (1975). Cognitron: A self-organizing multilayered neural network. Biological cybernetics, 20(3-4), 121-136.\n\n[2] Goodfellow, I., Bengio, Y., Courville, A., & Bengio, Y. (2016). Deep learning (Vol. 1, p. 2). Cambridge: MIT press.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper receives 3 initial rejection ratings. No rebuttal was submitted by the authors. There is no basis for overturning the reviewers' decisions. This paper should be rejected.
Decision: Reject