Venue: NeurIPS 2020 Workshop SVRHM

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: CNNs are not fully invariant to translation, as is often assumed. Recent work has found that pretraining on ImageNet allows the network to learn translation invariant representations on a transfer task. In this paper, the authors ask: _What is it about pretraining on ImageNet that endows CNNs with translation invariance?_

The authors first demonstrate the ImageNet pretraining result. The authors either pretrained a vanilla VGG16 network on ImageNet or began with an untrained model. They then fine-tuned or trained the network on a dataset in which objects appear in only one location. At test, the objects could appear in unseen locations. The accuracy of the network for different object translations is visualized in a heatmap. The network pretrained on ImageNet performs nearly perfectly across all 6 datasets considered, whereas the untrained network completely fails to generalize.

The authors hypothesize that pre-training on ImageNet facilitates translation invariance because during images appear in all possible positions on the canvas. They extend this hypothesis to ask whether pretraining on simple fully-translated datasets would allow the network to learn translation invariant representations in the transfer task. The authors find that while some fully-translated datasets allow the network to generalize, many do not. In particular, the MNIST variants transfer reasonably well between each other, but the Leek datasets completely fail to transfer to the MNIST variants.

The authors suggest that this is due to catastrophic forgetting. The authors analyze the cosine similarity of the representations of translated versions of the same object and find that all networks have high invariance before fine-tuning, which degrades severely after fine tuning.

I find the catastrophic forgetting argument unconvincing. Catastrophic forgetting is not an explanation of the cause of the failure to generalize in the previous task. Rather, it is a term that simply describes the phenomenon observed. It provides little explanatory power. 

A more relevant hypothesis that is not considered in this work is that ImageNet is a rich dataset in which image features at many levels of abstraction appear in many positions on the canvas. Because of the rich structure in this dataset, the array of learned features include features that are relevant for solving other tasks, such as the MNIST variants. Because they appear in all locations, this gives the model an advantage at learning translation invariant representations. The structure in MNIST and Leek, on the other hand, is very narrow. It is unsurprising that the pre-training fails to generalize in some cases, as the models are fit to a very different set of features. 

In short, it seems the simplest hypothesis is that training on a rich dataset in which 1) the feature set is sufficiently rich to generalize and 2) these features appear at all locations on the canvas allows a model to transfer useful translation invariant features to new tasks. 

I would have liked to see this hypothesis considered and tested. In future work, it may also be fruitful to use feature visualization techniques to tease apart the learned structure of these models and pinpoint the causes of this phenomenon. 
review_title: Analyses need more development

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: There is a lot of misleading statements made in the literature on the supposed translation invariance/equivariance of CNNs, and as such this paper is very welcome. The experiments both on pretraining a CNN with ImageNet as well as simpler datasets and testing for translation invariance are thorough enough to be convincing of the basic thesis - that while CNNs are not intrinsically invariant, they can learn invariant representations.

Here are some points that would make the paper stronger:
- It would be interesting to see how the forgetting and lack of performance scales with the complexity – we see that Leek2 and Leek10 generalize very poorly, but maybe one could subsample the other datasets and get some better understanding of the dependence on the size of the dataset
- What effect does the CNN architecture have here? As mentioned in the intro, stride has impact, but it would be nice to explore this in more detail.
- Some info on how fine-tuning was achieved while changing the number of classes (EMNIST, MNIST, Leek2 each having different #classes)


review_title: Interesting study of translation invariance

Decision: Accept (Poster)