Grounding inductive biases in natural images: invariance stems from variations in data

1 Introduction . A dataset can be described in terms of natural factors of variation of the data : for example , images of objects can present those objects with different poses , illuminations , colors , etc . Prediction consistency of models with respect to changes in these factors is a desirable property for out-ofdomain generalization [ 5 , 25 ] . However , state-of-the-art Convolutional Neural Networks ( CNNs ) struggle when presented with “ unusual '' examples , e.g . a bus upside down [ 2 ] . Indeed , CNNs lack robustness not only to changes in pose , but even to simple geometric transformations such as small translations and rotations [ 4 , 15 ] . Invariance to factors of variation can be learned directly from data , built-in via architectural inductive biases , or encouraged via data augmentation ( Fig . 1A ) . Our goal with this work is to explore the relative impact of these three factors on the trained model ’ s learned invariances and performance . As of today , data augmentation is the predominant method for encouraging invariance to a set of transformations . Yet , even with data augmentation models fail to generalize to held out objects and to learn invariance . For example , [ 15 ] found that augmenting with rotation and translation does not lead to invariance to the very same transformations during testing . A complementary research direction ensures a model responds predictably to transformations , using group equivariance theory ( see Cohen ∗equal contribution , a coin was flipped 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . and Welling [ 9 ] , Cohen et al . [ 10 ] among others ) . Provably invariant models have limited large-scale applications as they require a priori knowledge of the underlying factors of variation [ 9 , 17 ] . Recent work tackles automatic discovery of symmetries in data [ 6 , 12 , 20 , 39 ] . However , these methods have mostly been applied to synthetic or artificially augmented datasets which are not directly transferable to real data settings , and can even hurt performance when transferred [ 12 ] . Thus , we aim to identify the factors of variation of a real image dataset and to understand whether such equi/invariant models would be a relevant choice . One way to characterize the consistency of a model ’ s response is by measuring its equivariance or invariance to changes of the input . A model , f , is equivariant to a transformation Tθ of an input x if the model ’ s output transforms in a corresponding manner via an output transformation T ′θ , i.e . T ′θ ( f ( x ) ) = f ( Tθ ( x ) ) for any x. Invariance is a special type of equivariance , where the model ’ s output is the same for all transformations , i.e . T ′θ = I . Throughout this work , we will use for f the penultimate layer of a Resnet18 [ 22 ] or a vision transformer [ 14 ] trained on ImageNet . To understand whether CNNs and other state-of-the-art models are invariant to changes in the data factors of variation , one needs explicit annotations about such factors . While this is trivial for synthetic datasets with known factors , identifying the factors of variations in real datasets is a complex task . As such , prior work turned to synthetic settings to show that knowing the underlying factors improves generalization [ 8 , 26 ] . Here , we take a first step towards understanding the links between data augmentations and factors of variation in natural images , in the context of image classification . We do so by carefully studying the role of data augmentation , architectural inductive biases , and the data itself in encouraging invariance to these factors . We primarily focus on ResNet18 trained ImageNet as a benchmark for large-scale vision models [ 13 , 22 ] , which we also compare to the recently proposed vision transformer ( ViT ) [ 14 ] . While previous works study the invariance properties of neural networks to a set of transformations [ 15 , 23 , 25 , 27 , 38 ] , we ground invariances in ImageNet factors of variations . We make the following contributions ( summarized in Fig . 1 ) : • What transformations do standard data augmentation correspond to ? In Sec . 2 , we demonstrate that the success of the popular random resized crop ( RRC ) augmentation amounts to a precise combination of translation and scaling . To tease out the relative role of these factors , we decomposed RRC into separate augmentations . While neither augmentation alone was sufficient to fully replace RRC , we observed that despite the approximate translation invariance built into CNNs , translation alone is sufficient to improve performance close to RRC , whereas the contribution of scale was comparatively minor . • What types of invariance are present in ImageNet trained models ? Do these invariances derive from the data augmentation , the architectural bias or the data itself ( Figure 1A ) ? In Sec . 3 , we demonstrate that when invariance is present , it is primarily learned from data independent of the augmentation strategy used with the notable exception of translation invariance which is enhanced by standard data augmentation . We also found that architectural bias has a minimal impact on invariance to the majority of transformations . • Which transformations account for intra-class variations in ImageNet ? How do they relate to the models ’ invariance properties discovered in Sec . 3 ? In Sec . 4 , we show that appearance transformations , often absent from standard augmentations , account for Method Top-1 ± SEM RandomResizedCrop ( RRC ) 70.05± 0.1 RandomSizeCenterCrop 67.84± 0.1 FixedSizeRandomCrop 67.93± 0.0 T. ( 30 % ) 69.14± 0.0 T. ( 30 % ) + RandomSizeCenterCrop 69.30± 0.0 T. ( 30 % ) + RandomSizeCenterCrop w/o a.r . 69.20± 0.1 FixedSizeCenterCrop ( no augmentation ) 63.49± 0.1 ( a ) Using different training augmentations used , w/o a.r . stands for without aspect ratio change , T. stands for Translation . β Top-1 ± SEM 0.1 69.32± 0.0 0.5 70.17± 0.1 1 ( ∼ RRC ) 70.11± 0.0 2 69.52± 0.1 3 68.79± 0.0 10 63.74± 0.1 ( b ) Varying the shape of the distribution over s. RRC stands for RandomResizedCrop . Our results demonstrate that the relationship among architectural inductive biases , the data itself , the augmentations used , and invariance is often more complicated than it may first appear , even when the relationship appears intuitive ( such as for convolution and translation invariance ) . Furthermore , invariance generally stems from the data itself , and aligns with the data factors of variations . By understanding both which invariances are desirable and how to best encourage these invariances , we hope to guide future research into building more robust and generalizable models for large scale , realistic applications . Code to reproduce our results is in supplementary material . 2 Decomposing the Random Resized Crop Augmentation . Data augmentation improves performance and generalization by transforming inputs to increase the amount of training data and its variations [ 28 ] . Transformations typically considered include taking a crop of random size and location ( random resized crop ) , horizontal flipping , and color jittering [ 22 , 32 , 33 ] . Here , we focus on random resized crop ( denoted RandomResizedCrop ; RRC ) that is commonly used for training ResNets 2 . For an image of width W and height H , RandomResizedCrop ( 1 ) samples a scale factor s from a uniform distribution , s ∼ U ( s− , s+ ) and an aspect ratio r ∼ U ( ln r− , ln r+ ) ( 2 ) takes a square crop of size √ sHWr × √ sHW/r in any part of the image ( 3 ) resizes the crop , typically to 224× 224 for ImageNet . Thus , the area of an object selected by the crop is randomly scaled proportional to 1/s , which encourages the model to be scale invariant . The crop is also taken in any location of the image within its boundaries , which is equivalent to applying a translation whose parameters depend on the percentage of the area chosen for the crop . However , the way translation and scale interact remains obscure . In this section , we contrast the role of translation and scale in RRC and analyze the impact of the parameters used to determine these augmentations . 2.1 The gain of RandomResizedCrop is largely driven by translation rather than scale To study the role of both the scaling and translation steps , we separate RandomResizedCrop into two component data augmentations : • RandomSizeCenterCrop takes a crop of random size , always at the center of the image . The distribution for scale and aspect ratio are the same as those used in RandomResizedCrop . This augmentation impacts scale , but not translation . • FixedSizeRandomCrop takes a crop of fixed size ( 224× 224 ) at any location of the image ( the image is first resized to 256 on the shorter dimension ) . This augmentation impacts translation , but not scale . 2We follow the procedure of RandomResizedCrop as implemented in the PyTorch library [ 29 ] https : //pytorch.org/vision/stable/_modules/torchvision/transforms/transforms.html . As with RRC , both of these transformations can remove information from the image ( for example , translation can shift a portion of the image out of the frame whereas zooming in will remove the edges of the image ) , but neither augmentation can fully reproduce the effect of RRC by itself . We train ResNet18 on ImageNet and report results on the validation set as in commonly done ( e.g . in Touvron et al . [ 33 ] ) since the labelled test set is not publicly available . FixedSizeCenterCrop corresponds to what is usually done for augmenting validation/test images , i.e . resize the image to 256 on the shorter dimension and take a center crop of size 224 . We apply FixedSizeCenterCrop during all evaluation steps and refer to it as “ no augmentation '' . Training details are in Appendix A. RCC combines scale and translation in a precise manner . Table 1a shows that RandomResizedCrop performs best , with 70.05 % Top-1 accuracy . Augmenting by taking a crop of random size ( RandomSizeCenterCrop ) , a proxy for scale-invariance of the center object , performs on par with FixedSizeRandomCrop , a proxy to invariance to translation , and both bring a substantial improvement compared to no augmentation ( 67.9 % vs. 63.5 % ) . However , neither is sufficient to fully recapture the performance of RRC . To further test that RRC amounts to translation and scale , we augment the image by translating it by at most 30 % in width and height followed by taking a center crop of random size ( denoted T. ( 30 % ) + RandomSizeCenterCrop ) . This method improves Top-1 accuracy over RandomSizeCenterCrop and FixedSizeRandomCrop , and almost match RRC but with a gap of 0.75 % , showing that scaling and translating interact in a precise manner in RRC that is difficult to reproduce with both transformations applied iteratively . RCC ’ s performance is driven by translation . FixedSizeRandomCrop impacts translation , but its behavior is contrived , for example an image in the top left corner will never be in the bottom left corner of a crop . Thus to further disambiguate the role of translation versus scale , we also experiment using T. ( 30 % ) only : we resize the image to 256 on the shorter dimension , apply the random translation of at most 30 % and take a center crop of size 224× 224 . The gain in performance from T. ( 30 % ) compared to no augmentation is surprising given the ( approximate ) translation invariance built in to convolutional architectures such as ResNets . Furthermore , T. ( 30 % ) performs almost as well as T. ( 30 % ) + RandomSizeCenterCrop , which has scale augmentation as well . Thus , adding the change in scale to the translation does not further improve performance , which was not the case when comparing FixedSizeRandomCrop and RandomSizeCenterCrop . We compare the validation images that become correctly classified ( compared to no augmentation ) when using FixedSizeRandomCrop , T. ( 30 % ) and T. ( 30 % ) + RandomSizeCenterCrop in Appendix B but no clear pattern emerged .

Summary: In this work, the authors explore to what extent learning, data augmentation, and architectural inductive bias (ResNet18 and ViT) impact invariance to different sets of transformations for ImageNet-trained classifiers. They measure this invariance as cosine distance between an image and its transformed counterpart. They further probe the transformations that explain variation in ImageNet by designing a metric to measure how transformations affect inter-image similarity (pairs of different base images where one is transformed). A higher score on this metric is said to indicate stronger correspondence between the transformation and factors of variation in the data.  The authors find in general that data augmentation and architectural inductive bias have measurable (and decomposable) but weak affects on the invariances exhibited by the models, and also that these invariances are only weakly aligned with the major variation in the data, leading them to argue that the main source of invariance must be the data.  There list of worthwhile component findings in the paper is large. Just a few interesting ones that stuck out to me were:  - ViT and ResNets have similar invariance properties, and ViT may even exhibit better invariance - Data augmentation is specific to each class and can even hurt performance for some when applied globally - Translation and cropping augmentations interact to produce better performance when applied together - Geometric transformations may be the least relevant to the primary factors of variation in ImageNet