Grounding inductive biases in natural images: invariance stems from variations in data
Authors: Diane Bouchacourt
Mark Ibrahim
Ari S. Morcos
Venue: NeurIPS 2021 Poster (2021)
Keywords: data augmentation
invariance
transformations
factors of variation

Abstract: To perform well on unseen and potentially out-of-distribution samples, it is desirable for machine learning models to have a predictable response with respect to transformations affecting the factors of variation of the input. Here, we study the relative importance of several types of inductive biases towards such predictable behavior: the choice of data, their augmentations, and model architectures. Invariance is commonly achieved through hand-engineered data augmentation, but do standard data augmentations address transformations that explain variations in real data? While prior work has focused on synthetic data, we attempt here to characterize the factors of variation in a real dataset, ImageNet, and study the invariance of both standard residual networks and the recently proposed vision transformer with respect to changes in these factors. We show standard augmentation relies on a precise combination of translation and scale, with translation recapturing most of the performance improvement---despite the (approximate) translation invariance built in to convolutional architectures, such as residual networks. In fact, we found that scale and translation invariance was similar across residual networks and vision transformer models despite their markedly different architectural inductive biases. We show the training data itself is the main source of invariance, and that data augmentation only further increases the learned invariances. Notably, the invariances learned during training align with the ImageNet factors of variation we found. Finally, we find that the main factors of variation in ImageNet mostly relate to appearance and are specific to each class.

Grounding inductive biases in natural images: invariance stems from variations in data
Diane Bouchacourt; Mark Ibrahim; Ari S. Morcos Facebook AI Research
{dianeb, marksibrahinn, effo com com
Abstract
To perform well on unseen and potentially ant-of-distributioo samples, is desir- able for machine learning models to have a predictable response with respect to transformations affecting the factors variation of the input. Here, we study the relative importance of several types of inductive biases towards such predictable be- havior: the choice of data, their augmentations, and model architectures. Invariance is commonly achieved through hand-engineered data augmentation, but do standard data augmentations address transformations that explain variations in real data? While prior work has focused synthetic data, attempt here to characterize the factors of variation real dataset, ImageNet, and study the invariance both standard residual networks and the recently proposed vision transformer with respect to changes in these factors. We show standard augmentation relies on a precise combination of translation and scale, with translation recapturing most of the performance improvement-ddespic the (approximate) translation invariance built in to convolutional architectures, such residual networks. In fact, found that scale and translation invariance was similar across residual networks and vi- sion transformer models despite their markedly different architectural inductive biases. We show the training data itself is the main source of invariance, and that data augmentation only further increases the learned invariances. Notably, the invariances learned during training align with the ImageNet factors of variation we found. Finally, we find that the main factors of variation in ImageNet mostly relate to appearance and are specific to each class.
Introduction
dataset can be described in terms of natural factors of variation of the data: for example, images of objects can present those objects with different poses, illuminations, colors, etc. Prediction consistency of models with respect to changes in these factors a desirable property for out-of- domain generalization [5] 225. However, ttate-of-the-art Convolutional Neural Networks (CNNs) struggle when presented with "unusual" examples, e.g. a bus upside down [2]. Indeed, CNNs lack robustness not only to changes in pose, but even to simple geometric transformations such as small
translations and rotations [ 15
Invariance to factors of variation can be learned directly from data, built-in via architectural inductive biases, or encouraged via data augmentation (Fig. [A]. Our goal with this work is explore the relative impact of these three factors on the trained model's learned invariances and performance. As of today, data augmentation the predominant method for encouraging invariance transformations. Yet, even with data augmentation models fail to generalize to held out objects and to learn invariance. For example, [15] found that augmenting with rotation and translation does not lead to invariance to the very same transformations during testing. A complementary research direction ensures a model responds predictably to transformations, using group equivariance theory (see Cohen
set of
"qqual contribution, a coin was flipped
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Possible Paths to Invariance
A
B
Alkis suriate
dracctioo Diar
Learned from arthitcctre
Lernedhonccss
TOAAA
FAG VARRAU ON.
X
V
rotation color
Learned diinity
POA BANCE
X
?
V
X
X
X
Shoar
X
X
X
X
sugrnentation
Paclase
X
X
Figure 1: Grounding invariance in factors of variation. In Fig. A. illustrate possible paths learning invariance: architectural inductive bias, directly from data, and through data augmentation. In Fig. B, we summarize our findings about invariance. A red arrow means invariance decreases, question mark means results do not allow for a clear answer.
and Welling [9]. Cohen et al. 10] among others). Provably invariant models have limited large-scale applications as they require a priori knowledge of the underlying factors of variation p 17 Recent work tackles automatic discovery of symmetries in data 66, 12 20 [3]. However, these methods have mostly been applied to synthetic to real data settings, and can even hurt performance when transferred [12). Thus, we aim to identify the factors of variation of real image dataset and understand whether such equi/invariant models would be a relevant choice.
artificially augmented datasets which are not directly transferable
One way to characterize the consistency of a model's response is by measuring its equivariance invariance to changes of the input. A model, f. is equivariant to a transformation T3 of an input if the model's output transforms in corresponding manner via output transformation Ty i.e. TT((()) = ((Ta(z)) for any x. Invariance is a special type of equivariance, where the model's output is the same for all transformations, i.e. Tg I. Throughout this work, we will use for f the penultimate layer of a Resnet18 [22] or a vision transformer [44 trained on ImageNet. To understand whether CNNs and other state-of-the-art models are invariant changes in the data factors of variation, one needs explicit annotations about such factors. While this is trivial for synthetic datasets with known factors, identifying the factors of variations in real datasets a complex task. As such, prior work turned to synthetic settings to show that knowing the underlying factors improves generalization 8 26).
Here, we take a first step towards understanding the links between data augmentations and factors of variation in natural images, in the context of image classification. We do by carefully studying the role of data augmentation, architectural inductive biases, and the data itself in encouraging invariance to these factors. We primarily focus ResNet18 trained ImageNet vision models 13 [22,, which we also compare to the recently proposed vision transformer (ViT) [14). While previous works study the invariance properties of neural networks to set of transformations 27 38) we ground invariances in ImageNet factors of variations. We make the following contributions (summarized in Fig. D
benchmark for large-scale
15 23 25
What transformations do standard data augmentation correspond to? In Sec. 2 we demonstrate that the success of the popular random resized crop (RRC) augmentation amounts to a precise combination of translation and scaling. To tease out the relative role of these factors, augmentation alone was sufficient to fully replace RRC, we observed that despite the approximate translation invariance built into CNNs, translation alone sufficient to improve performance close to RRC, whereas the contribution of scale was comparatively minor.
decomposed RRC into separate augmentations. While neither
What types of invariance are present in ImageNet trained models? Do these invariances derive from the data augmentation, the architectural bias the data itself (Figure I7A)? In Sec. we demonstrate that when invariance is present, it is primarily learned from data independent of the augmentation strategy used with the notable exception of translation in- variance which is enhanced by standard data augmentation. We also found that architectural bias has minimal impact on invariance the majority of transformations.
Which transformations account for intra-class variations in ImageNet? How do they relate to the models' invariance properties discovered in Sec. B In Sec. 4 we show that appearance transformations, often absent from standard augmentations, account for

---

Method
Top-1 SEM SEM
B
Top-1 SEM
RandomResiizedCrop (RRC)
70.05 - 0.1
0.1
69.32 1332:000 0.0
TandomSSizeGGenterrCrp
67.84
0.5
70.17 0.1
Pixed@izeehnndomCrrp
67.93 = 0.0
(~ RRC)
70.11 0.0
(30%)
69.14 0.0
69.52 0.1
T. (30%) + RandomSizacCenterCrrp
69.30 = 0.0
3
68.79 - 0.0
T. (30%) ZaadoonSSzeCenterCCrpp w/o a.r.
69.20 0.1
10
63.74 0.1
FixedSizeCenterCrrr (no augmentation)
63.49 + 0.1
(b) Varying the shape of the dis- tribution over s. RRC stands for RandomeesizedCrop.
(a) Using different training augmentations used, w/o stands for without aspect ratio change, stands for Translation.
Table 1: ImageNet validation Top-1 accuracy + SEM (standard error of the mean).
intra-class changes for factors of variation in ImageNet. We found training enhances model's natural invariance to transformations that account for ImageNet variations (including appearance transformations), and decreases invariance for transformations that do not seem to affect factors of variation. We also found factors of variation are unique per class, despite common data augmentations applying the same transformations across all classes.
Our results demonstrate that the relationship among architectural inductive biases, the data itself, the augmentations used, and invariance is often more complicated than it may first appear, even when the relationship appears intuitive (such as for convolution and translation invariance). Furthermore, invariance generally stems from the data itself, and aligns with the data factors of variations. By understanding both which invariances are desirable and how best encourage these invariances, we hope to guide future research into building more robust and generalizable models for large scale, realistic applications. Code to reproduce our results is in supplementary material.
Decomposing the Random Resized Crop Augmentation
Data augmentation improves performance and generalization by transforming inputs to increase the amount of training data and its variations (28). Transformations typically considered include taking a crop of random size and location (random resized crop), horizontal flipping, and color jittering 22 32 333. Here, we focus on random resized crop (denoted RandonResizedCrop RRC) that
commonly used for training ResNets
For image of width W and height H. RandomResizedCrop (1) samples scale factor from uniform distribution, & ~ (s_,8+) and an aspect ratio r ~ (mnr-, In rn)) (2) takes a square crop of size VrHWr x //HWr in any part of the image (3) resizes the crop, typically to 224 x 224 for ImageNet. Thus, the area of an object selected by the crop is randomly scaled proportional to 1/s, which encourages the model to be scale invariant. The crop is also taken in any location of the image within its boundaries, which equivalent to applying translation whose parameters depend on the percentage of the area chosen for the crop. However, the way translation and scale interact remains obscure. this section, we contrast the role of translation and scale RRC and analyze the impact of the parameters used to determine these augmentations.
2.1 The gain of Randomeessizddrrop is largely driven by translation rather than scale
To study the role of both the scaling and translation steps, we separate RandomResizedCrop into two component data augmentations:
zeCentrrz takes crop of random size, always at the center of the image. The distribution for scale and aspect ratio the same those used in BandomResizeddCrop. This augmentation impacts scale, but not translation.
FixedSizeffandomCrre takes crop of fixed size (224 x 224) any location of the image (the image is first resized to 256 on the shorter dimension). This augmentation impacts translation, but not scale.
Wwe follow the procedure of RandomRessizedCrog implemented in the PyTorch library 29 http://pwwwp.rttsoooooc//oooooos
html.

---

As with RRC, both of these transformations can remove information from the image (for example, translation can shift a portion of the image out of the frame whereas zooming in will remove the edges of the image), but neither augmentation can fully reproduce the effect of RRC by itself.
We train ResNet1 8 ImageNet and report results on the validation set in commonly done (e.g. in Touvron et al. [33]) since the labelled test set is not publicly available. FixedSizeCenterCrop corresponds to what usually done for augmenting validation/test images, i.e. resize the image to 256 on the shorter dimension and take a center crop of size 224. We apply FixedSizeCenter during all evaluation steps and refer to "no augmentation" Training details are in Appendix A
RCC combines scale and translation in a precise manner. Table RandomResizedCron performs best, with 70.05% Top-1 accuracy. Augmenting by taking a crop of random size [RandomSiizeCnteerrrre proxy invariance to translation, and both bring substantial
shows that
a proxy for scale-invariance of the center object, performs
on par with FixeddizeRandommGrrri improvement compared to no augmentation ( 67.9% vs. 63.5%). However, neither is sufficient to fully recapture the performance of RRC. To further test that RRC amounts translation and scale, we augment the image by translating it by at most 30% in width and height followed by taking a center crop of random size (denoted T. (30%) accuracy over RandomSi i www.oeerrreeeeererreerooo F FixedSizeRandomoee
RandomSSizeCennerrrro.. This method improves Top-1
and almost match RRC but with gap of 0.75%, showing that scaling and translating interact in a precise manner RRC that is difficult to reproduce with both transformations applied iteratively.
impacts translation, but its behavior is contrived, for example an image in the top left corner will never be in the bottom left comer of crop. Thus further disambiguate the role of translation versus scale, we also experiment using T. (30%) only: we resize the image to 256 on the shorter dimension, apply the random translation of most 30% and take a center crop of size 224 x 224. The gain in performance from T. (30%) compared to no augmentation is surprising given the (approximate) translation invariance built in convolutional architectures such as ResNets. Furthermore, T. (30%) performs almost as well as Т. (30%) + GandomSizeCenterOrie adding the change in scale to the translation does not further improve performance, which was not the case when comparing FixedSizeRandomooe the validation images that become correctly classified (compared to no augmentation) when using Fi ZeRaddorrooo T. (30%) and T. (30%) + RandonSizeCenterCCrg in Appendix B but no clear pattern emerged.
RCC's performance is driven by translation. FixedSizeRandommrree
which has scale augmentation as well. Thus,
and
We compare
2.2 Trade-off between variance and magnitude of augmentation
What role does the distribution over augmentations in RandonResizedCrrop play? Default values for the distribution over 8 are 8- 0.08, 8+ 1. Thus the scale factor can increase the size of an object in the crop up to to 1/s 8 1/0.08 x 12.5 times larger. Does only the range of augmentation magnitudes matter? What if we change the shape of this distribution, for the same range of values?
To test this, we modified RandomResiizedCrop to use a Beta distribution B(a, 3) over the standard interval for (00.08, 1)). Fixing a 1, we vary 8, which changes the distribution shape. Setting = I corresponds to a uniform distribution (0.08, 1), while smaller values of / lead to heavily sampling values of near (and vice versa; see Appendix Fig. Al for visualizations). Table Tb shows that just varying the shape of the distribution results in a 6% drop in performance. To explain this drop, we examine the discrepancy between average apparent object sizes during training and evaluation as in Touvron et al. 33 We note smaller values of 33 reduce the discrepancy between image sizes during training and evaluation (see Appendix for the mean of B(a, 3)). However, very small values of B3 (e.g.. 0.1) ) also decrease performance, as they do not encourage scale invariance by sampling s near 1 (no scale change) and 1/s has very little variance. Thus, we observe trade-off between variability to induce invariance and consistency between training and evaluation object sizes.
3 Invariance across architectures and augmentations
So far, we have measured the impact of decomposed augmentations on model performance, but how do these augmentations impact invariance to these and other transformations? To what extent do other elements, such as architectural inductive bias and learning contribute to these invariances? And finally, how do these invariances differ across categories of transformations? In this section,

---

address these questions by defining metric for invariance and evaluate this metric for number of transformations across combinations of architectures, augmentations, and training.
3.1 Measuring invariance
model f is considered invariant to a transformation with a specific magnitude To if applying To leaves the output unchanged. We choose to measure invariance by measuring the cosine distance, d between the embeddings of a sample a and its transformed version, relative to baseline, b:
InvT, (f(x)) b [(T((),
(1)
where f generates the embedding up from the penultimate layer of ResNet18 or a vision transformer trained on ImageNet [14] (36], see Appendix for training details. The baseline, b, is the embedding distance across two randomly selected samples, b = (ax), 1 (x;()) to account for the effect
different transformations (and magnitudes) may have on the distribution of embeddings. The closer InvT, to 1, the more invariance to Tg. We report the distribution of Invr, across pairs.
To measure how invariance changes as transformations intensify, we report invariance as a function of transformation magnitude, with magnitude 0 meaning no transformation and magnitude corre- sponding to large transformation (see Appendix Fig. A2 for examples). We expect invariance to decrease as transformation magnitude increases for the majority of settings.
3.2 Invariance to geometric and appearance transformations
To understand the extent and source of invariance, we measured invariance to common appearance and geometric transformations for both 8 and ViT models. Appearance transformations, such as changes in brightness, alter color or illumination, while geometric transformations such as scaling, translation, and rotation alter the spatial arrangement of pixels.
First, we found that models indeed featured invariance to number of common transformations, including translation and scale (Fig-2). 2) Examining the impact of architecture, we observed that for translation, both ResNet18 and ViT models Icarned to be invariant, with ViT models consistently achieving slightly higher translation invariance than R8sNet18. Surprisingly, untrained VIT models also featured stronger invariance to translation when compared to untrained ResNet18 (Fig. 2 b, compare orange with light blue), suggesting that despite the convolutional inductive bias present in ResNet18, translation invariance is more prominent in ViT models.
We next examined the impact of training on invariance. While training consistently increased invariance to translation and to appearance based transformations (Fig. Pb.b and additional figures in Appendix Fig. A3), training resulted in no change in average invariance to scale (zooming-in) but reduced the variability this invariance (Fig. 2}. In contrast, training reduced invariance to rotation and shear (Fig. 24 and A3). Finally, examined the role of augmentation in learning invariance. Consistent with our finding in Sec. [.I] we found standard augmentation improves translation invariance (albeit only slightly; compare dark and medium blue in Fig. 2b..). Surprisingly, it barely increases scale invariance, and had equivocal effects on other transformations. Together, these results suggest that the data itself is the major factor influencing learned invariance rather than the choice of architecture or the specific augmentations used.
We have shown that ResNets present less invariance to translation than vision transformers, despite the inductive bias the convolutional architecture. As such, what role does the architectural inductive bias in ResNets play? To answer this, we measure equivariance to assess whether models encode predictable responses to transformations. As described in Section [ property than invariance: Invariance is specific case of cquivariance and equivariance does not necessarily imply invariance (though invariance does imply equivariance). We evaluate equivariance by examining the alignment of embedding responses to transformations. We find while an untrained R is not invariant to translation, it is equivariant to translation, highlighting the architectural inductive bias of CNNs to translation (see Appendix D for detailed results).
equivariance is a more general
Nevertheless, we observed that trained ViT and ResNet18 are both able to learn invariance to both geometric and appearance transformations, regardless of their particular inductive biases. However, it remains unclear how these learned invariances relate to the factors of variation present in ImageNet. In the next section, attempt to answer this question.
5

---

Translation (x-axis)
Invarance
Trained VIT (L/16)
Trained ResNet18
Untrained ResNet18
Trainad Aug ResNet18
magnitude
Translation (y-axis)
Scale (zoom in)
invariance
Rotation
Posterize (color)
Invarance
-05
magnitude
magnitude
Figure 2: Comparing sources of invariance. We compare invariance across trained and untrained ResNet18 and VIT isolate the effect of architectural bias, training, and data augmentation invariance. Standard augmentations used for R and ViT are RRC+horizontal flips. Values are training pairs, though we found similar trends on validation pairs.
Characterizing factors of variation with similarity search
In the previous section, assessed the extent and source of invariance transformations for ResNet18 and VIT. Among learning, data augmentation, and architectural inductive biases, we identified learning from data the predominant source of invariance. Consequently, here investigate which aspects of the data's variation drive models' learned invariances. We characterize the variation ImageNet in terms of transformations and relate the variation to the models' learned invariances. To do so, we design a metric comparing how well each transformation allows us to travel from one image to another image with the same label, thus capturing factor of variation.
In the previous section, we assessed whether ResNet18 and ViT are invariant to a set transfor- mations, and explored to what extent learning, data augmentation and architectural inductive biases impact this invariance. But do learned invariances correspond to the transformations that actually affect factors of variation in the data? What the transformations that explain variations in Ima- geNet? In this section, we design metric to answer these questions by comparing how well each transformation allows us travel from one image another image with the same label, thus capturing a factor of variation. Finally, we relate our findings to models' invariances from the previous section.
Characterizing the factors of variation present in natural images is challenging since don't have access to generative model of these images. Here, we introduce a metric to determine these transformations based simple idea: factors of variation in the data should be able to explain the differences between images of the same class. For example, suppose one of the primary factors of variation in dataset of animals is pose. In this case, by modifying the pose of one image of dog, we should able to match another image of dog with a different pose. Concretely, we measure the change in similarity transformation brings to image pairs. For image pair (æ1, x22 from the same class and transformation Tp we measure the percent similarity change as
SimChanger, (x (To(x2))) sim(s(x1), ((x2)
sim (f f(2))
where sim measures cosine similarity (see Appendix E To control for any effect from data aug- mentation, take f to be ResNet18 model up to the penultimate layer trained without data

---

Across classes
Per class
Top transfomattinn invariance
wio Aug
Standaad Aug
Average 16 siriiiiity change
Average simikrriy chernge
Invetanne triinnng stancird ausi
Class
Figure 3: Factors of variation are class-specific and have higher invariance after training. Fig A. Com- pares the average percent change similarity of pairs across all classes. No single geometric or appearance transformation increases pair similarity across all classes. In contrast, Fig. B shows per class transformations consistently increase similarity. Fig C. shows training increases invariance for transformations more likely to appear among the top 5 transformations per class.
augmentation and report values on training image pairs. A higher similarity SimChangey, among pairs implies a stronger correspondence between the transformation and factors of variation. We select relevant pool of transformations using Augmmnu,,nt, procedure across several image datasets [T]. The set of transformations encompasses geometric and appearance transformations of varying magnitude For each Ta we report SimChanger, averaged on image pairs.
automated augmentation search
This metric has both advantages and disadvantages. Its primary advantage is that it can be measured on realistic image datasets such as ImageNet without requiring access to a ground-truti generative model, is often used in synthetic datasets. It also uses pool of possible transformations which are realizable with standard data augmentation techniques, and encompasses both geometric and appearance based transformations. However, because this metric does not exploit information about the generative process, it has limited ability to capture realistic transformations that occur in the abstract space describing semantic image content. Furthermore, it is difficult to make conclusions based on absolute metrics; as such, we use relative comparisons.
4.1
Can image pairs across ImageNet be described by consistent set of transformations?
Underlying the common practice of data augmentation the assumption that images ought to vary consistently across a dataset. Throughout training, samples are augmented with the same set of transformations, albeit with differing magnitudes. To test this assumption we ask whether we can explain the variation among image pairs with the same set of transformations. If so, we expect to find a set of transformations which consistently increases the similarity of pairs. In Fig. 3/ we show the distribution of average similarity changes across each transformation measured by Equation 2 We observe that no single transformation consistently increases average similarity of image pairs across all classes, including geometric and appearance transformations. We find the same pattern holds whether a ResNet18 is trained with or without standard augmentations (RRC + horizontal flips). This result suggests that although standard approaches to data augmentation apply the same transformation distribution to all classes, no single transformation consistently improves similarity (see Appendix E). Could instead consistently increase similarity among image pairs by a combination of transformations?
Combinations of transformations do not consistently increase similarity. We repeat our anal- ysis of transformations' effect image pair similarity using sub-policies, which combine two transformations of varying magnitudes. We found that while sub-policies can help or hurt by larger margin as they apply multiple transformations, no sub-policy increases average similarity across all classes (Fig. A4
Does using combinations of transformations increase similarity? To answer this, repeat the analysis using sub-polien sub-policies, which combine two transformations of varying magnitudes. We found that while sub-policies can help hurt by larger margin as they apply multiple transformations, no sub-policy increases average similarity across all classes (Fig. A4).
Note that f fully invariant to To then SimChanger, will zero. Thus we also chose trained w/o augmentation to reduce translation invariance and confirmed full invariance not achieved in Sec, "See Appendix A2 for full list of possible transformations.
7

---

Counter to the common practice of data augmentation, this result suggests the same set of transforma- tions does not consistently explain the variation among image pairs. What effect then, any, does augmenting with the same set of standard transformations have on image similarity?
Training augmentations dampen pair similarity changes. Similar to our earlier results, we observed that training with augmentations induce more invariance relative to training without aug- mentations. Though all transformations consistently decrease similarity for models trained with and without augmentation, models trained with augmentation exhibited both a smaller decrease in similarity and lower variance (Fig. [AA. Training with augmentation also reduced the decrease in similarity for transformations beyond simply scale and translation (Appendix Fig. A5), suggesting that these augmentations impact the response even transformations not used during training.
4.2 Are factors of variation specific to each class?
In Sec. 4. we showed no transformation (or sub-policy) consistently increased similarity across classes. However, the factors of variation and consequently, the optimal transformation, may be different for different classes, especially those which are highly different. Can we instead consistently increase similarity if allow flexibility for transformations to be class specific?
To test this hypothesis, we examined the top transformation for each class. In contrast to the global result, we found the top transformation per class consistently increased the average similarity for all classes (Fig. B). Notably, the top transformation per class increased similarity by +.3%% (mean + SEM) compared effectively change (0.020% + 0.021%) for the top transformation across all classes. Similar to Hauberg et al. (21] which learn per class transformations for data augmentation that boost classification performance MNIST [24], applied per class data augmentation variants ImageNet, but observed no significant classification performance boost. We leave further applications of per class augmentation future work.
Data augmentation not beneficial for all classes, Since the optimal transformation differs across classes, do standard augmentations benefit all classes only some classes? To test this, examined the impact of RRC on the performance of individual ImageNet classes. Interestingly, we found that on average] 12.3 + 0.21% % have lower performance when using RRC. Critically, some classes are consistently hurt by the use of RRC with a difference in top-1 accuracy as high as 22%. We systematically study these classes in Appendix D.I but no clear pattern emerged.
4.3 Appearance transformations are more prevalent
Data augmentation and most of the literature on invariant models often rely only on geometric transformations such as translations, rotations, and scaling. However, it is not clear whether or not the factors of variation in natural images are primarily geometric. this the case, we would expect the top transformations from our similarity search to be geometric rather than appearance-based. In contrast, we found that appearance transformations accounted for more variation in ImageNet than geometric transformations, consistent with recent work [II]. Of the top transforms, 78% were appearance based compared to only 22% geometric. We confirmed this difference is not due to sampling bias by ensuring approximately transformations, In fact, we isolate geometric transformations, we find for 64% of classes the top transformation is the identity, suggesting geometric transformations are worse when applied to an entire class than no transformation at all. We find a similar pattern among the top transforms per class for R8 trained with standard augmentations: for more than 98.4% of classes the top transformation alters appearance not geometry. In Appendix E.I we also examine local variation in foregrounds to translation and scale.
split between geometric and appearance
4.4 Training increases invariance for factors of variation
In Sec. 3 we showed that training increases invariance to number of transformations, independent of architectural bias and augmentation and in Sec. 4.1 we used similarity search to characterize the transformations present in natural images. However, do the invariances learned over training correspond to the factors of variation present in natural images?
'Computed over 25 pairwise comparisons of 5 runs with both augmentations.

---

Transformaiion rank similarity
(Spoarman nark correlatioo)
0.9
0.333
56967
0.233 0.000
Wordnet pair similarity
Fncsssiigg 1g simiarky
(Wu-Palmer score)
Figure 4: Similar classes share factors of variation. Wordnet class pair similarity versus transformation ranking. Violin plots show the distribution of transformation Spearman's rank correlation as class pair similarity increases. We see similar classes rank transformations similarly.
To test this, we asked whether the transformations to which networks learn to be invariance correspond the same transformations which are highly ranked in similarity search. We found transformations that exhibit increased invariance over training were substantially more likely to be ranked in the top transformations per class compared transformations which exhibited decreased or minimal change in invariance over training (Fig. CC) This result demonstrates training increases invariance to factors of variation present in real data, regardless whether there an inductive bias or data augmentation is specifically designed to encourage invariance.
4.5 Characterizing factors of variation across classes
We have shown that there exist factors of variation which consistently increase similarity for given class, but it remains unclear why a particular factor might impact a particular class. Here, we investigate whether related classes feature related factors of variation.
One prominent pattern which emerged was among textile-like classes such as velvet, wool, hand- kerchief, and envelope. When considering single transformations, rotation is the top transformation or rotation plus an appearance transformations (such as color or posterize) for sub-policies (see Appendix E.2 for full list). The relationship between rotation and textiles makes intuitive sense: fabrics generally ly don't It have a canonical orientation and can appear in many different colors.
To test this systematically, it to the similarity between the top transforms for each class. We computed class similarities using the most specific common ancestor in the Wordnet against the Spearman rank of transformation types. We found that while dissimilar classes often have similar transformations, similar classes consistently exhibit more similar transformations (Fig. Appendix E.3
measured the pairwise class similarity using Wordnet [16] and compared
5 Related work
Data augmentation approaches. Standard data augmentations often amount to taking crop of random size and location, horizontal flipping and color jittering [22 32 33 In self-supervised learning, (7] boost performance by using multiple crops of the same image. Hauberg et al. 21] learn augmentueeeeeee per class augmentation and improve performance on the MNIST dataset 24. Auto.Aumentt is
reinforcement learning- based technique that discovers data augmentations that most aid downstream performance [IT]. Antoniou al. [3] train Generative Adversarial Networks (GANs [8] based model to generate new training samples. van der Wilk et al. (34] follow a Bayesian approach and integrate invariance into the prior. Recent works aim to automatically discover symmetries
dataset (20 and enforce equivariance or invariance to these 66] 12 39]. While these methods are promising, they have mostly been applied to synthetic datasets or augmented versions of real datasets. Their application to a real dataset such as ImageNet is not straightforward:
tried applying the Augerino model to ImageNet, we found was struggling to discover effective augmentations composed of multiple transformations (see Appendix
Consistency neural architectures. Zhang 38 show that invariance to translation is in neural networks, and propose solution based anti-aliasing by low-pass filtering. Kauderer-

---

Abrams 23 studies the source of CNNs translation invariance of CNNs on a translated MNIST maps based on Euclidean distance of embeddings, and find that data augmentation has bigger effect on translation invariance than architectural choices. Very recently, Myburgh et al. [27] replace Fuclidean distance with cosine similarity, and find that fully connected layers contribute more to translation invariance than convolutional ones. Lenc and Vedaldi [25] study the equivariance, invariance and equivalence different convolutional architectures with respect to geometric transformations. Here, we study invariance a larger set of transformations with varying magnitudes and compare ResNet18 and ViT. Engstrom et al. [15] Touvron et al. [3]] explore the specificities of standard data augmentations, and Engstrom et al. [15] found that model augmented at training with rotations and translations still fails on augmented test images. We differ from these works by grounding invariance into the natural factors of variation of the data, which we try to characterize. To the best of our knowledge, the links between invariance and the data factors of variation has yet not been studied on a large-scale real images dataset.
dataset 24 using cunslation-sensitivve
Discussion
In this work, we explored the source of invariance in both convolutional and vision transformer architectures trained for classification on ImageNet, and how these invariances relate to the factors of variation of ImageNet. We compared the impact of data augmentation, architectural bias and the data itself on the models' invariances. We observed that LandomResizedCrop relies on a precise combination of translation and scale that is difficult to reproduce and that, surprisingly, augmenting with translation recaptures most of the improvement despite the (approximate) invariance to translation built in to convolutional architectures. By analyzing the source of invariance in ResNet18 18 and ViT, we demonstrated that invariance generally stems from the data itself rather than from architectural bias, and is only slightly increased by data augmentation.
Finally, we connected the models' learned invariance to the factors of variation present in the data. We characterized variation in ImageNet by examining pair similarity in response to transformations, finding that transformations which explain variation in ImageNet are class-specific, more appearance- based, and align with the invariances' learned during training.
Limitations and future work We provide an analysis of the invariant properties of models using specific set of metrics based on model performance and similarities of input embeddings. Using these, some of our conclusions are shared with existing works, but a different set metrics could potentially bring more insights our conclusions. Additionally, we only experiment on ImageNet, but it would be interesting to perform the same type of analysis on a wider range of datasets and data types. Does handful of transformations describe the variations of most standard image datasets? Our study sheds light on ImageNet factors of variation but some conclusions remain obscure, such as the role of scaling. This emphasizes the difficulty of performing systematic study of real datasets.
Finally, our findings spark further exploration. Could tailoring augmentations per class or introducing appearance- based augmentations improve performance?
Potential negative societal impacts. While our work is concerned with robustness of vision models which can have various societal impacts, our work is primarily analytical and we do not propose new model. Hence, we do not foresee any potential negative societal impacts of our findings. Our study does emphasize the importance of dataset construction, the predominant source of invariance, relative to other modelling considerations. Consequently, our work encourages researchers to also consider dataset construction as an important aspect of vision models' socictal impact.
We would like to thank David Lopez-Paz, Armand Joulin, Yann Olivier, Kamalika Chaudhuri, Aaron Aadock, Pascal Vincent, Michael Alcom, and Nicolas Ballas for helpful discussions and feedback.
Acknowledgements
References
[1] Sumukh K Aithal, Dhruva Kashyap, and Natarajan Subramanyam. Robustness to augmentations as a generalization metric. CoRR, abs/2101 066599, 2021. httpssttts::// //arxii org/ /bs/ 2101. 06459.
10

---

[2] A. Michael Alcorn Qi Li, Zhitao Gong, Chengfei Wang, Long Mai, Wei-Shinn Ku, and Anh Nguyen. Strike (with) a pose: Neural networks are easily fooled by strange poses of familiar objects. CVPR, pages 4854-4854, 2019.
[3] Antreas Antoniou, Amos Storkey, and Harrison Edwards. Data augmentation generative adversarial networks, 2018.
[4] Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize poorly to small image transformations? Journal of Machine Learning Research, 20(184):1-25, 2019. URL http::ttp:::mlleeee org/ http://appr///eaeeeee / 1220/19555 html
[5] Yoshua Bengio, Aaron Courville, and Pascal Vincent. Representation learning: A review and new perspectives. IEEE transactions on pattern analysis and machine intelligence, 35(8): 1798-1828, 2013. Publisher: IEEE.
[6] Gregory Benton, Mare Finzi, Pavel Izmailov, and Andrew Gordon Wilson. Learning invariances in neural networks. arXiv preprint arXiv:2010. 1882, 2020.
[7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Ar- mand Joulin. Unsupervised learning of visual features by contrasting cluster assign- ments. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Ad- vances in Neural Information Processing Systems, volume 33, pages 9912-9924. Curran Associates, Inc.. 2020. URL https::/ pdf
neurips wtwppappp//022/f11///
[8] Xiuyuan Cheng, Qiang Qiu, Robert Calderbank, and Guillermo Sapiro. Rotdef: Decompo- sition of convolutional filters for cotatin-equivarre deep networks. International Confer- ence on Learning Representations 2019 (ICLR' 19). URL https: //par nsf gov/bbblio/ 10099150]
[9] Taco Cohen and Max Welling. Group equivariant convolutional networks. In Maria Florina Balcan and Kilian Q. Weinberger, editors, Proceedings of The 33rd International Conference on Machine Learning, volume 48 of Proceedings of Machine Learning Research, pages 2990- 2999, New York, New York, USA, 20 22 Jun 2016. PMLR. JRLhhtp://proceddnng mlr
[10] Taco Cohen, Mario Geiger, and Maurice Weiler. A General Theory of Equivariant CNNs on Homogeneous Spaces. arXiv: 1//11.00001 [cs, stat), January 2020. URL http::/arax/ org/
abs/1811.0200
arXiv: 1811.02017.
[11] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc Le. Autoaugment: Learning augmentation strategies from data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 113-123, 2019.
[12] Nima Dehmamy, Yanchen Liu, Robin Walters, and Rose Yu. Lie algebra convolutional neural networks with automatic symmetry extraction. 2021. URL / net/ ForumPiddccThn2PIIoon
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pages
Ieee, 2009.
[14] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2020.
[15] Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring the landscape of spatial robustness. Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 1802-1811. PMIR, 09-15 Jun 2019. URI attp://proceeding mlr. wwwpps/vs9////// wwwpeennnnggaaa.i.eeeeee html html
11

---

[16] Christiane Fellbaum. WordNet: An Electronic Lexical Database. Bradford Books, 1998.
[17] M. Finzi, Stanton, P. Izmailov, and A. G. Wilson. Generalizing Convolutional Networks for Equivariance to Lie Groups on Arbitrary Continuous Data. In Proceedings of the International Conference on Machine Vision and Machine Learning, 2020.
[18] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. I_ Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K. Q. Weinberger, editors, Advances in Neural Information Processing Systems, volume 27. Curran Associates, Inc., 2014. htttt:::ss::p/eeeeee/ee pdf
neur
[19] B. Hall. Lie Groups, Lie Algebras, and Representations: An Elementary Introduction. Graduate Texts in Mathematics. Springer International Publishing, 2015, ISBN 9783319134673. URL
google. com
[20]
Tatsunori B Hashimoto, Percy $ Liang, and John Duchi. Unsupervised transformation learning via convex relaxations. In Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30, pages 6875-6883. Curran Associates, Inc., 2017. URL htt.ttts:://occeedaaaaias Paper. pdf
neur http://.c.pppeooo//////////////a
[21] Søren Hauberg, Oren Freifeld, Anders Boesen Lindbo Larsen, John Fisher, and Lars Hansen. Dreaming more data: Class-dependent distributions over diffeommrphisms for learned data augmentation. In Arthur Gretton and Christian C. Robert, editors, Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pages 342 342-350, 350, Cadiz, Spain, 09-11 May 2016. PMLR. URL http://prooceedings mlr. wwwpppewsssll...pp haaass//11hhabbrg1eee html
[22] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. arXiv preprint arXiv:1512.003385 2015.
[23] Eric Kauderer-Abrams, Quantifying translltioon-invaiinnee in convolutional neural networks. CoRR, abs/1800.011450 2018. URLittpp///rraiv..g//bbbb//// 01450
[24] Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.d http://ynm..ccmm.com/// lecum.com/exxb//niiss 2, 2010.
[25] Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. International Journal of Computer Vision (IJV), 127(5), 2019.
(26) Francesco Locatello, Ben Poole, Gunnar Rätsch, Bernhard Schölkopf, Olivier Bachem, and Michael Tschannen. Weakly- Disentanglement Without Compromises. arXiv:2002.c 02886 [cs, stat], June 2020. URL 2002.02886.
arXiv:
[27] Johannes C. Myburgh, Coenraad Mouton, and Marelie H. Davel. Tracking translation invariance in cnns. CoRR, abs/2 hiss2104405997,, 2021. URL
05997
[28] P. Niyogi, F. Girosi, and T. Poggio. Incorporating prior information in machine learning by creating virtual examples. In Proceedings of the IEEE, pages 2196-2209, 1998.
[29] Adam Paszke. Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang. Zachary Veto Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high- performance deep learning library. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché- Buc, E. Fox, and Garnett, editors, Advances in Neural Information Processing Systems 32, pages 8024-8035. Curran Associates, Inc., 2019. URL http: http://pppperr...eipppppp. pttpuriipscccpppprr//. an-
library
pdf
12

---

[30] Ning Qian. On the momentum term in gradient descent learning algorithms. Neural Networks, 12(1):145-151, 1999. URL httt:://bllp trrerrrrrr.. www.:rrrrr..edeedeee httpi/reeee/b///oaaaaaaaa/////aa html#
Qian99
[31] Xuebin Qin, Zichen Zhang, Chenyang Huang, Masood Dehghan, Osmar R. Zaiane, and Martin Jagersand. U2-net: Going deeper with nested u-structure for salient object detection. Pattern Recognition, 106:107444, Oct 2020. ISSN 0031-3205 doi: 10. pttp://.atcog.2220...o
URL http:///// doi. org/ 10. 1016/j patcog, 2020. 107404
[32] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
[33] Hugo Touvron, Andrea Vedaldi, Matthijs Douze, and Herve Jegou. Fixing the train-test resolution discrepancy. In H. Wallach, H. Larochelle, A. Beygelzimer, F. dd'Alcé-Buue E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https: httpps://proceeddingg hegggss.euiip... cc/ cc/paper/2019/ pdf
[34] Mark van der Wilk, Matthias Bauer, ST John, and James Hensman. Learning invariances using the marginal likelihood. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https: htttps://proceeddngs. hetpsssnurrppp.. cc/ ac/paper/2018/ pdf
[35] Lijun Wang, Huchuan Lu, Yifan Wang, Mengyang Feng, Dong Wang, Baocai Yin, and Xiang Ruan. Learning to detect salient objects with image-level supervision. In CVPR, 2017.
[36] Ross Wightman. Pytorch image models.
http:://gitaab..ommrruttttatt
pytorch- models 2019.
[38] Richard Zhang. Making convolutional networks http:://www....orrrrmllpaa.aa.aa
[37]
WolframlAlpha. 2021 (accessed June 2021). URL
again. In ICML, 2019.
[39] Allan Zhou, Tom Knowles, and Chelsea Finn. Meta-learning symmetries by reparameterizzation arXiv preprint arXiv:2007.02933, 2020.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the Discussion section.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See the Discussion section.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A] We did not include theoretical results.
(b) Did you include complete proof' of all theoretical results? [N/A] We did not include theoretical results.
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as URL)? [Yes] Code to reproduce the main experiments is available at htttps://github. com/ hagbooorreaarr//gooonnnnniiieiee
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)?
[Yes] See details in each experiment section, well Appendix A
G.1
G.3 D
(c) Did you report error bars (e.g., with respect to the random seed after running exper- iments multiple times)? [Yes] We report standard error of the mean (SEM) for all
experiments.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Appendix A
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] We cite all assets and existing code / pre-trained models we used (ImageNet dataset, Wordnet database, U2Net, pre-trained ViT, Augerino library, AutoAugment).
(b) Did you mention the license of the assets? [N/A] We used existing assets which licenses can be found in the references we cite. While ImageNet does not own the copyright of the images, they allow non-commereial uses of the data under the terms of access written in https: / /t///mmaaeenneee
org/ dw//loadddd php.
(c) Did you include any new assets either in the supplemental material as URL? [Yes] While we don't provide any new assets such as datasets, we provide code to reproduce main results in the Supplementary.
(d) Did you discuss whether and how consent obtained from people whose data you' 're using/curating? [N/A] We use the ImageNet dataset as and reference to it. (e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A] We use the ImageNet dataset is and reference to it.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A] We did not conduct research with human subjects or used crowd- sourcing.
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A] We did not conduct research with human subjects or used crowdsourcing.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A] We did not conduct research with human subjects or used crowdsourcing.
14


Reviews:
[{"review_id": "qk-pVqUoUqv", "reviewer": "Reviewer_WyYA", "summary": "This paper provides an analysis of the invariance of neural networks (ResNet) trained on image classification (ImageNet) to image transformations, and of the role of data augmentation in encouraging invariance. The paper provides different types of results. First, in Section 2, the authors analyse what components of the data augmentation scheme _random resize crop_ (RRC) mainly drive the gains in performance, and conclude that translation is the main transformation. Second, in Section 3, the paper discusses the invariance distribution (measured as the relative cosine distance between augmented and original images) of models to several transformations. The paper presents results of untrained models, trained with and without augmentation, in order to isolate the sources of invariance. The authors conclude that untrained models already exhibit certain level of invariance, training improved the invariance to some (but not all) transformations, and data augmentation further improved invariance, but only slightly and not always. Finally, in Section 4, the authors aim to characterise the factors of variation within ImageNet comparing the similarity of samples under some pre-defined transformations. This section presents multiple results, the ones highlighted by the authors being that factors of variation differ across classes and that training a model increases the invariance to the transformations that are found to explain most variation in ImageNet.", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "_Edit, post-rebuttal: I have upgraded my numerical evaluation by one point_ \n\n---\n\n## Strengths or contributions\n\nI highlight that the paper addresses a topic that has received increasing attention in the last couple of years, namely data augmentation and invariance to image transformations, and therefore it has the potential of being of interest to the community. The paper offers a wide set of results and some of them are interesting and insightful. For example, the most relevant results to me are those presented in Section 3, which analyse the invariance of models separately to various transformations.\n\n## Weaknesses and justification of overall score\n\nI begin by summarising the justification of my overall evaluation in terms of the four main dimensions of the review: in terms of *quality*, I think the paper is technically correct, the methods are justified and sufficient details are generally provided (with some exceptions that I mention below). The paper is also *clearly written* for the most part and is easy to follow. As an exception, I note that Section 4 is a little confusing, perhaps because many results are tried to be presented in a limited space. As a weaker aspect, while the paper provides some interesting results, I would argue that the overall *significance* is limited. In my humble opinion, there are no major findings that offer new perspectives or insights about data augmentation and invariance, but rather a collection of results that either confirm findings from previous work or offer limited new insights. In terms of *originality*, the methods used are mostly standard or slightly different from other variants in the literature. In the remainder of my review I elaborate on these weaker aspects.\n\nThe main part of the paper is divided into three sections (2, 3 and 4) that present significantly diverse analyses and results. As a general comment, I believe that such breadth of points of analyses may dilute the main messages and compromise the depth of the most interesting ones. For example, in my opinion, the most relevant results are those in Section 3, but the conclusions could be much stronger with an analyses of additional architectures and data sets. I will comment on each Section separately.\n\n### Section 2, analysis of `RandomResizedCrop`\n\nI think the significance and relevance of the results in this section are quite limited. The section refers, in the title and in the text, to an analysis of the composition of standard data augmentation *techniques* (in plural), but in fact it is an analysis of random resized crop (`RandomResizedCrop`, RRC). While an in-depth analysis of multiple transformations could be of interest, the results of one transformation on one architecture and one data set are, in my opinion, of little relevance.\n\nFurthermore, the conclusion, namely that translation augmentations plays a major role in improving the performance, should provide little insights to those familiar with the computer vision literature. While RRC may have gained popularity in recent years, a very common and simple augmentation scheme and widely known to provide very good results has been the combination of horizontal and vertical translations and horizontal flips. See for example [1, 2]. The results of using a $\\beta$ distribution instead of uniform sampling for the augmentation parameters are also expected and, in fact, previous work has reported results of training with data augmentation with different sampling than uniform with slight improvements.\n\n### Section 3, invariance of architectures and augmentations\n\nAs mentioned before, I find this section the most interesting and relevant. In particular, I am not aware of other articles that have analysed the role of individual transformations on model invariance. However, the main conclusions highlighted by the authors are more or less known from the literature. For example, that convolutional neural networks do not exhibit sufficient invariance to translations, despite the widely spread claim, has been noted by multiple authors before, perhaps most clearly by [3] (as this paper cites in the related work section). Nonetheless, in Section 3 this observation is qualified as \"surprising\". \n\nThe other main observation of the section is that data augmentation barely increases the invariance of the models, also highlighted as surprising. However, this has also been reported before, for example by [4], with a related but different invariance score.\n\nFinally, I have technical question regarding the invariance metric: it is not entirely clear what the baseline $b$ is in Equation (1). What are $x_i$ and $x_j$ in its definition (line 170)? Two random samples? This is a crucial aspect that I have not fully understood from the paper, although I have assumed the high-level intuition is clear.\n\n### Section 4, factors of variation in ImageNet\n\nThis section contains multiple results, some of them of interest, but the amount of different experiments and results seems to compromise the readability of the section. For example, some conclusions are to be interpreted from figures in the main paper, some from figures in the appendix and some from results that are not displayed in figures but only reported textually in a paragraph. As I mentioned above, the main messages may be diluted among the variety of minor results and conclusions.\n\nOn a more technical level, it is not surprising that transformations do not measurably increase the average similarity between pairs, since the factors of variations in natural images are much more complex. In other words, the goal of trying to disentangle the factors of variation in ImageNet is of interest, but it is perhaps too ambitious to be tackled as a section of a paper, by measuring the similarity explained by a limited set of transformations.\n\nFinally, like in other sections discussed above, the main conclusions do not provide substantial insights. For example, that training improves the invariance to the transformations that drive the most similarity is to be expected. In contrast, the conclusion about class-specificity of some transformations is interesting and could have practical impact, but the analysis provided in this paper is not deep enough and is left as future work.\n\n## Other questions\n\nHere are some other comments, questions or suggestions that are more minor or disconnected from the main arguments of my review. These had a nearly negligible impact on my overall assessment and are included here for completeness:\n\n* In the first paragraph of Section 3.2, it is not clear that the conclusions refer to the \"Untrained\" box plots of Figure 2. Furthermore, it is particularly confusing that the text reads \"models **learned** to be invariant\", while we are talking about _untrained_ models.\n* The notion of invariance in neural networks changes across papers. For example, some consider invariance as measured by the classification output of a model (which should probably refer to as robustness), some measure invariance at the softmax output, and more generally, some measure invariance at a high-level embedding. It would be positive to define the notion of invariance that the authors use early enough in the introduction.\n* In Section 4, why is AutoAugment used, instead of the set of transformations analysed in Section 3?\n* The last paragraph of Section 3.2 is confusing. Is it a summary of what has been said before? If so, why do the authors talk here about _equivariance_ instead of _invariance_? Is it a discussion of new results beyond Figure 2? If so, where are these results?\n* Do the authors have an explanation for the higher invariance to rotations of untrained models?\n\n## Typos \n\nBelow I list some potential typos or language errors:\n\n* Line 169: [13] [31] -> [13, 31]\n* Line 211: access to generative model -> access to a generative model\n* Line 218: effect data augmentation -> effect of data augmentation\n\n## References\n\n* [1] Goodfellow et al. Maxout networks. ICML. 2013.\n* [2] Springenberg et al. Striving for simplicity: The all convolutional net. ICLR. 2014.\n* [3] Zhang. Making convolutional networks shift-invariant again. 2019. ICML\n* [4] Hernandez-Garcia et al. Learning robust visual representationsusing data augmentation invariance. 2019. https://openreview.net/forum?id=B1elqkrKPH", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do mention at the end of the paper some of the limitations of their work, as well as state that they do not see potential negative societal impacts.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6"}, "extra_text": {"ethical_concerns": "I see no relevant ethical concerns specifically derived from this work, but it would be worth mentioning, in my opinion, the concerns associated with _any_ work that makes use of ImageNet. See, for instance, [Prabhu and Birhane (2020)](https://arxiv.org/abs/2006.16923).\n\n_Note: I have updated this note after the original review was submitted._"}}, {"review_id": "m0Tiq4jdfc", "reviewer": "Reviewer_dbox", "summary": "In this work, the authors explore to what extent learning, data augmentation, and architectural inductive bias (ResNet18 and ViT) impact invariance to different sets of transformations for ImageNet-trained classifiers. They measure this invariance as cosine distance between an image and its transformed counterpart. They further probe the transformations that explain variation in ImageNet by designing a metric to measure how transformations affect inter-image similarity (pairs of different base images where one is transformed). A higher score on this metric is said to indicate stronger correspondence between the transformation and factors of variation in the data.\n\nThe authors find in general that data augmentation and architectural inductive bias have measurable (and decomposable) but weak affects on the invariances exhibited by the models, and also that these invariances are only weakly aligned with the major variation in the data, leading them to argue that the main source of invariance must be the data.\n\nThere list of worthwhile component findings in the paper is large. Just a few interesting ones that stuck out to me were:\n\n- ViT and ResNets have similar invariance properties, and ViT may even exhibit better invariance\n- Data augmentation is specific to each class and can even hurt performance for some when applied globally\n- Translation and cropping augmentations interact to produce better performance when applied together\n- Geometric transformations may be the least relevant to the primary factors of variation in ImageNet", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I found this paper quite interesting and packed with meaningful results. I have some doubts about whether the metrics proposed won't end up misleading in the end, but I also understand that such metrics are difficult to validate. There have been number of recent works indicating that the inductive bias of CNNs may not be as essential or well-understood as previously thought, and this may be one of the clearest analyses of this observation to date. My comments and criticisms are mostly minor:\n\n- I read the following two sentences from the same section as contradictory: (1) \"Surprisingly, untrained ViT models also featured stronger invariance to translation when compared to untrained ResNet18 (Fig. 2a, b, compare orange with light blue), suggesting that despite the convolutional inductive bias present in ResNet18, translation invariance is more prominent in ViT models.\" (2) \"We find equivariance to translation for untrained ResNet18 that is absent for ViT, highlighting the architectural inductive bias of CNNs to translation.\" This should be cleared up as it's an important finding in the paper.\n\n- The authors argue that ResNets/ViTs have similar invariances despite \"markedly different inductive biases\", but is this true? Don't the findings provide evidence that architectural differences are misleading as biases if their generalization properties turn out to be quite similar?\n\n- If class-level augmentation doesn't help performance in the end, what should we make of the relevant results in the paper? Could they be spurious? Does it help with a minority of difficult validation images?\n\n- typo: \"prose\" --> \"pose\"", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors should discuss the potential impact of the claim that data is more responsible than the model design in determining invariance, because if true it would imply that researchers may have a stronger responsibility to attend to dataset construction than to modeling when considering potential negative outcomes of a model. This isn't so much a negative impact of this particular work as it is informative about how researchers should assess negative impact of their own work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "MsOsFsIzdl", "reviewer": "Reviewer_b4MW", "summary": "The paper explores the links between popular data augmentation techniques and the natural factors of variation present in data, such as translation, rotation, scale, and color. The paper is structured into three main sections, which each describe a set of experiments designed to answer a high level research question. Section 2 is focused on understanding how the popular data augmentation technique Random Resized Crop (RRC) can be decomposed into simpler factors of variation such as translation and scale.  Section 3 proposes a measurement of invariance and studies how the choice of architecture and data augmentation strategy impact invariance to specific transformations.  Section 4 proposes a technique to identify which factors of variation are most prevalent in the data, and finds that, in ImageNet, 1) the factors of variation are class specific, and 2) appearance transformations (like contrast, posterization, brightness changes) account for more variation in ImageNet compared to geometric transformations (translate, scale, shear). \n\nThe experiments presented suggest that tailoring augmentation to each class of using more appearance-based transformations might help improve robustness and accuracy.\n", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Major Issues**\n\n1. *Lack of testing for the prescriptions*.  The authors' experiments reveal some interesting directions for improving data augmentation strategies (such as tailoring augmentations per class or using more appearance based augmentations), but the authors don’t test these strategies themselves.  The experiments and experimental techniques used by the authors would be more valuable if the authors could prove that they lead to beneficial prescriptions. \n\n2. *Unclear how decomposition of RRC is helpful*.  The authors show that RRC can be decomposed into a scale and translation, but find that neither augmentation alone can replace RRC.  So, similar to the previous issue, it is not clear why knowing which factors of variation a data augmentation technique corresponds to if we cannot use this knowledge to improve performance.\n\n3. *Role of the scale transformation*. In Section 2, it's surprising that the contribution of scale was relatively minor in the performance of RRC. Possibly, the center crop might miss the object of interest so we end up scaling a piece of the image that doesn't correspond to the class at all.  In contrast, for the translation, the fixed size random crop selects a random location of the image, so it has some chance of randomly selecting the part of the image that is relevant for the class.  The authors could look at whether scale helps certain classes where the object is less likely to be cut out of the image (such as the textile or scenery classes). Or, using a dataset with bounding boxes available like the ILSVRC2013 detection dataset (https://image-net.org/challenges/LSVRC/2013/index.php), the authors could ensure that some fraction of the bounding box overlaps with the crop before scaling.\n\n4. *Synthetic factors of variation*. The set of transformations that the authors explore are limited in that they can all be written as a coded function that can be applied to the images. The robustness community has started to refer to this type of transformation as “synthetic”. It has been shown that synthetic augmentations often do not help on more realistic or natural distribution shifts [1]. It would be interesting if the authors tried to explore some way to capture more natural factors of variation, which could be discovered through human labeling.\n\n**Minor Issues**\n\n5. Not clear where the results are for the experiments described in lines 195-201\n6. Distinction between appearance and geometric transformations should be more clearly described or illustrated in main text\n7. Difficult to read figures – text is quite small.\n\n**Originality**: The work has novel experimental insights.\n\n**Quality**: See comments above.\n\n**Clarity**: The writing is sometimes hard to follow, but the main points are still understandable.  There are too many rhetorical questions sprinkled throughout.\n\n**Significance**: The experimental findings would be more significant if the authors could show they resulted in prescriptions that could improve model performance and accuracy.\n\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes, the authors discussed both in Section 6.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "CgJTrvlyLSr", "reviewer": "Reviewer_Ybab", "summary": "The authors characterize invariances of standard models on ImageNet through the lens of commonly used data augmentation strategies and their relative effects.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The authors provide a thorough and convincing study disproving the existence of many properties that I, and probably many other readers, would have taken for granted. Many very important and surprising things are learned from this paper:\n\t-- The depth of analysis of the effect of RCC, and of derived augmentations related to it, was very insightful and a rare glimpse into something so ubiquitously used but not often studied.\n\t-- Viewing the invariances per-class helpfully identify the heterogeneity of the problem and will hopefully encourage per-class solutions rather than the traditional augmentations which are one-size-fits-all vis-a-vis classes.\n\t-- The discussion of appearance versus geometric invariance is crucial, and this work will hopefully inspire future focus on geometric augmentations that better reflect the underlying non-Euclidean-pixel-space source of these images.\n\t-- I would have liked to see more about the evaluation of training on one augmentation and then it's impact on other transformations, as for example the appendix figure suggests these effects could be substantial.\n\t-- It is disappointing that no organizing thoughts could be garnered from the classes where RRC significantly hurt accuracy, as ideally the types of analyses used in this paper could be used to generate better augmentations, but merely identifying them here can be used as a step in that process.\n\nWhile this study was limited to one kind of model/task on one dataset, and while I think it would be interesting to explore these problems in other contexts that use the same augmentations (e.g. generative models), I view that is definitely beyond the scope of one paper, and thus see no problem with its current scope.\n\nOverall, I found many insightful points that were drawn from highly relevant contexts and think this would be a valuable contribution.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper brought about very divergent opinions from the reviewers — they all agreed that the analyses across the three sections are technically solid, and the topic is of broad interest to the community, but the main points of disagreement were over the novelty of the results, the paper being hard to read since it is packed with results, and whether the paper provides any actionable insights for future modeling approaches. I don’t necessarily view actionable insights as 100% necessary (although they would certainly add value to the paper), but upon reading the paper myself, I agree with reviewer WyYA’s assessment that Sections 3 and 4 are probably the more interesting ones and I wish the authors had dived deeper into them and perhaps had a more cohesive messaging for the paper (it is indeed packed with many, many results which makes it hard to digest). Overall however, I do think some of the results presented in this paper are surprising and will inform ongoing and future research in this field.
Decision: Accept (Poster)