Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: In this work, the authors explore to what extent learning, data augmentation, and architectural inductive bias (ResNet18 and ViT) impact invariance to different sets of transformations for ImageNet-trained classifiers. They measure this invariance as cosine distance between an image and its transformed counterpart. They further probe the transformations that explain variation in ImageNet by designing a metric to measure how transformations affect inter-image similarity (pairs of different base images where one is transformed). A higher score on this metric is said to indicate stronger correspondence between the transformation and factors of variation in the data.

The authors find in general that data augmentation and architectural inductive bias have measurable (and decomposable) but weak affects on the invariances exhibited by the models, and also that these invariances are only weakly aligned with the major variation in the data, leading them to argue that the main source of invariance must be the data.

There list of worthwhile component findings in the paper is large. Just a few interesting ones that stuck out to me were:

- ViT and ResNets have similar invariance properties, and ViT may even exhibit better invariance
- Data augmentation is specific to each class and can even hurt performance for some when applied globally
- Translation and cropping augmentations interact to produce better performance when applied together
- Geometric transformations may be the least relevant to the primary factors of variation in ImageNet
main_review: I found this paper quite interesting and packed with meaningful results. I have some doubts about whether the metrics proposed won't end up misleading in the end, but I also understand that such metrics are difficult to validate. There have been number of recent works indicating that the inductive bias of CNNs may not be as essential or well-understood as previously thought, and this may be one of the clearest analyses of this observation to date. My comments and criticisms are mostly minor:

- I read the following two sentences from the same section as contradictory: (1) "Surprisingly, untrained ViT models also featured stronger invariance to translation when compared to untrained ResNet18 (Fig. 2a, b, compare orange with light blue), suggesting that despite the convolutional inductive bias present in ResNet18, translation invariance is more prominent in ViT models." (2) "We find equivariance to translation for untrained ResNet18 that is absent for ViT, highlighting the architectural inductive bias of CNNs to translation." This should be cleared up as it's an important finding in the paper.

- The authors argue that ResNets/ViTs have similar invariances despite "markedly different inductive biases", but is this true? Don't the findings provide evidence that architectural differences are misleading as biases if their generalization properties turn out to be quite similar?

- If class-level augmentation doesn't help performance in the end, what should we make of the relevant results in the paper? Could they be spurious? Does it help with a minority of difficult validation images?

- typo: "prose" --> "pose"
limitations_and_societal_impact: The authors should discuss the potential impact of the claim that data is more responsible than the model design in determining invariance, because if true it would imply that researchers may have a stronger responsibility to attend to dataset construction than to modeling when considering potential negative outcomes of a model. This isn't so much a negative impact of this particular work as it is informative about how researchers should assess negative impact of their own work.
review_title: null

final_score: rating: 5: Marginally below the acceptance threshold
review_content: summary: This paper provides an analysis of the invariance of neural networks (ResNet) trained on image classification (ImageNet) to image transformations, and of the role of data augmentation in encouraging invariance. The paper provides different types of results. First, in Section 2, the authors analyse what components of the data augmentation scheme _random resize crop_ (RRC) mainly drive the gains in performance, and conclude that translation is the main transformation. Second, in Section 3, the paper discusses the invariance distribution (measured as the relative cosine distance between augmented and original images) of models to several transformations. The paper presents results of untrained models, trained with and without augmentation, in order to isolate the sources of invariance. The authors conclude that untrained models already exhibit certain level of invariance, training improved the invariance to some (but not all) transformations, and data augmentation further improved invariance, but only slightly and not always. Finally, in Section 4, the authors aim to characterise the factors of variation within ImageNet comparing the similarity of samples under some pre-defined transformations. This section presents multiple results, the ones highlighted by the authors being that factors of variation differ across classes and that training a model increases the invariance to the transformations that are found to explain most variation in ImageNet.
main_review: _Edit, post-rebuttal: I have upgraded my numerical evaluation by one point_ 

---

## Strengths or contributions

I highlight that the paper addresses a topic that has received increasing attention in the last couple of years, namely data augmentation and invariance to image transformations, and therefore it has the potential of being of interest to the community. The paper offers a wide set of results and some of them are interesting and insightful. For example, the most relevant results to me are those presented in Section 3, which analyse the invariance of models separately to various transformations.

## Weaknesses and justification of overall score

I begin by summarising the justification of my overall evaluation in terms of the four main dimensions of the review: in terms of *quality*, I think the paper is technically correct, the methods are justified and sufficient details are generally provided (with some exceptions that I mention below). The paper is also *clearly written* for the most part and is easy to follow. As an exception, I note that Section 4 is a little confusing, perhaps because many results are tried to be presented in a limited space. As a weaker aspect, while the paper provides some interesting results, I would argue that the overall *significance* is limited. In my humble opinion, there are no major findings that offer new perspectives or insights about data augmentation and invariance, but rather a collection of results that either confirm findings from previous work or offer limited new insights. In terms of *originality*, the methods used are mostly standard or slightly different from other variants in the literature. In the remainder of my review I elaborate on these weaker aspects.

The main part of the paper is divided into three sections (2, 3 and 4) that present significantly diverse analyses and results. As a general comment, I believe that such breadth of points of analyses may dilute the main messages and compromise the depth of the most interesting ones. For example, in my opinion, the most relevant results are those in Section 3, but the conclusions could be much stronger with an analyses of additional architectures and data sets. I will comment on each Section separately.

### Section 2, analysis of `RandomResizedCrop`

I think the significance and relevance of the results in this section are quite limited. The section refers, in the title and in the text, to an analysis of the composition of standard data augmentation *techniques* (in plural), but in fact it is an analysis of random resized crop (`RandomResizedCrop`, RRC). While an in-depth analysis of multiple transformations could be of interest, the results of one transformation on one architecture and one data set are, in my opinion, of little relevance.

Furthermore, the conclusion, namely that translation augmentations plays a major role in improving the performance, should provide little insights to those familiar with the computer vision literature. While RRC may have gained popularity in recent years, a very common and simple augmentation scheme and widely known to provide very good results has been the combination of horizontal and vertical translations and horizontal flips. See for example [1, 2]. The results of using a $\beta$ distribution instead of uniform sampling for the augmentation parameters are also expected and, in fact, previous work has reported results of training with data augmentation with different sampling than uniform with slight improvements.

### Section 3, invariance of architectures and augmentations

As mentioned before, I find this section the most interesting and relevant. In particular, I am not aware of other articles that have analysed the role of individual transformations on model invariance. However, the main conclusions highlighted by the authors are more or less known from the literature. For example, that convolutional neural networks do not exhibit sufficient invariance to translations, despite the widely spread claim, has been noted by multiple authors before, perhaps most clearly by [3] (as this paper cites in the related work section). Nonetheless, in Section 3 this observation is qualified as "surprising". 

The other main observation of the section is that data augmentation barely increases the invariance of the models, also highlighted as surprising. However, this has also been reported before, for example by [4], with a related but different invariance score.

Finally, I have technical question regarding the invariance metric: it is not entirely clear what the baseline $b$ is in Equation (1). What are $x_i$ and $x_j$ in its definition (line 170)? Two random samples? This is a crucial aspect that I have not fully understood from the paper, although I have assumed the high-level intuition is clear.

### Section 4, factors of variation in ImageNet

This section contains multiple results, some of them of interest, but the amount of different experiments and results seems to compromise the readability of the section. For example, some conclusions are to be interpreted from figures in the main paper, some from figures in the appendix and some from results that are not displayed in figures but only reported textually in a paragraph. As I mentioned above, the main messages may be diluted among the variety of minor results and conclusions.

On a more technical level, it is not surprising that transformations do not measurably increase the average similarity between pairs, since the factors of variations in natural images are much more complex. In other words, the goal of trying to disentangle the factors of variation in ImageNet is of interest, but it is perhaps too ambitious to be tackled as a section of a paper, by measuring the similarity explained by a limited set of transformations.

Finally, like in other sections discussed above, the main conclusions do not provide substantial insights. For example, that training improves the invariance to the transformations that drive the most similarity is to be expected. In contrast, the conclusion about class-specificity of some transformations is interesting and could have practical impact, but the analysis provided in this paper is not deep enough and is left as future work.

## Other questions

Here are some other comments, questions or suggestions that are more minor or disconnected from the main arguments of my review. These had a nearly negligible impact on my overall assessment and are included here for completeness:

* In the first paragraph of Section 3.2, it is not clear that the conclusions refer to the "Untrained" box plots of Figure 2. Furthermore, it is particularly confusing that the text reads "models **learned** to be invariant", while we are talking about _untrained_ models.
* The notion of invariance in neural networks changes across papers. For example, some consider invariance as measured by the classification output of a model (which should probably refer to as robustness), some measure invariance at the softmax output, and more generally, some measure invariance at a high-level embedding. It would be positive to define the notion of invariance that the authors use early enough in the introduction.
* In Section 4, why is AutoAugment used, instead of the set of transformations analysed in Section 3?
* The last paragraph of Section 3.2 is confusing. Is it a summary of what has been said before? If so, why do the authors talk here about _equivariance_ instead of _invariance_? Is it a discussion of new results beyond Figure 2? If so, where are these results?
* Do the authors have an explanation for the higher invariance to rotations of untrained models?

## Typos 

Below I list some potential typos or language errors:

* Line 169: [13] [31] -> [13, 31]
* Line 211: access to generative model -> access to a generative model
* Line 218: effect data augmentation -> effect of data augmentation

## References

* [1] Goodfellow et al. Maxout networks. ICML. 2013.
* [2] Springenberg et al. Striving for simplicity: The all convolutional net. ICLR. 2014.
* [3] Zhang. Making convolutional networks shift-invariant again. 2019. ICML
* [4] Hernandez-Garcia et al. Learning robust visual representationsusing data augmentation invariance. 2019. https://openreview.net/forum?id=B1elqkrKPH
limitations_and_societal_impact: The authors do mention at the end of the paper some of the limitations of their work, as well as state that they do not see potential negative societal impacts.
review_title: null

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper explores the links between popular data augmentation techniques and the natural factors of variation present in data, such as translation, rotation, scale, and color. The paper is structured into three main sections, which each describe a set of experiments designed to answer a high level research question. Section 2 is focused on understanding how the popular data augmentation technique Random Resized Crop (RRC) can be decomposed into simpler factors of variation such as translation and scale.  Section 3 proposes a measurement of invariance and studies how the choice of architecture and data augmentation strategy impact invariance to specific transformations.  Section 4 proposes a technique to identify which factors of variation are most prevalent in the data, and finds that, in ImageNet, 1) the factors of variation are class specific, and 2) appearance transformations (like contrast, posterization, brightness changes) account for more variation in ImageNet compared to geometric transformations (translate, scale, shear). 

The experiments presented suggest that tailoring augmentation to each class of using more appearance-based transformations might help improve robustness and accuracy.

main_review: **Major Issues**

1. *Lack of testing for the prescriptions*.  The authors' experiments reveal some interesting directions for improving data augmentation strategies (such as tailoring augmentations per class or using more appearance based augmentations), but the authors don’t test these strategies themselves.  The experiments and experimental techniques used by the authors would be more valuable if the authors could prove that they lead to beneficial prescriptions. 

2. *Unclear how decomposition of RRC is helpful*.  The authors show that RRC can be decomposed into a scale and translation, but find that neither augmentation alone can replace RRC.  So, similar to the previous issue, it is not clear why knowing which factors of variation a data augmentation technique corresponds to if we cannot use this knowledge to improve performance.

3. *Role of the scale transformation*. In Section 2, it's surprising that the contribution of scale was relatively minor in the performance of RRC. Possibly, the center crop might miss the object of interest so we end up scaling a piece of the image that doesn't correspond to the class at all.  In contrast, for the translation, the fixed size random crop selects a random location of the image, so it has some chance of randomly selecting the part of the image that is relevant for the class.  The authors could look at whether scale helps certain classes where the object is less likely to be cut out of the image (such as the textile or scenery classes). Or, using a dataset with bounding boxes available like the ILSVRC2013 detection dataset (https://image-net.org/challenges/LSVRC/2013/index.php), the authors could ensure that some fraction of the bounding box overlaps with the crop before scaling.

4. *Synthetic factors of variation*. The set of transformations that the authors explore are limited in that they can all be written as a coded function that can be applied to the images. The robustness community has started to refer to this type of transformation as “synthetic”. It has been shown that synthetic augmentations often do not help on more realistic or natural distribution shifts [1]. It would be interesting if the authors tried to explore some way to capture more natural factors of variation, which could be discovered through human labeling.

**Minor Issues**

5. Not clear where the results are for the experiments described in lines 195-201
6. Distinction between appearance and geometric transformations should be more clearly described or illustrated in main text
7. Difficult to read figures – text is quite small.

**Originality**: The work has novel experimental insights.

**Quality**: See comments above.

**Clarity**: The writing is sometimes hard to follow, but the main points are still understandable.  There are too many rhetorical questions sprinkled throughout.

**Significance**: The experimental findings would be more significant if the authors could show they resulted in prescriptions that could improve model performance and accuracy.



limitations_and_societal_impact: Yes, the authors discussed both in Section 6.
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: The authors characterize invariances of standard models on ImageNet through the lens of commonly used data augmentation strategies and their relative effects.
main_review: The authors provide a thorough and convincing study disproving the existence of many properties that I, and probably many other readers, would have taken for granted. Many very important and surprising things are learned from this paper:
	-- The depth of analysis of the effect of RCC, and of derived augmentations related to it, was very insightful and a rare glimpse into something so ubiquitously used but not often studied.
	-- Viewing the invariances per-class helpfully identify the heterogeneity of the problem and will hopefully encourage per-class solutions rather than the traditional augmentations which are one-size-fits-all vis-a-vis classes.
	-- The discussion of appearance versus geometric invariance is crucial, and this work will hopefully inspire future focus on geometric augmentations that better reflect the underlying non-Euclidean-pixel-space source of these images.
	-- I would have liked to see more about the evaluation of training on one augmentation and then it's impact on other transformations, as for example the appendix figure suggests these effects could be substantial.
	-- It is disappointing that no organizing thoughts could be garnered from the classes where RRC significantly hurt accuracy, as ideally the types of analyses used in this paper could be used to generate better augmentations, but merely identifying them here can be used as a step in that process.

While this study was limited to one kind of model/task on one dataset, and while I think it would be interesting to explore these problems in other contexts that use the same augmentations (e.g. generative models), I view that is definitely beyond the scope of one paper, and thus see no problem with its current scope.

Overall, I found many insightful points that were drawn from highly relevant contexts and think this would be a valuable contribution.

limitations_and_societal_impact: Yes
review_title: null

Meta-review: This paper brought about very divergent opinions from the reviewers — they all agreed that the analyses across the three sections are technically solid, and the topic is of broad interest to the community, but the main points of disagreement were over the novelty of the results, the paper being hard to read since it is packed with results, and whether the paper provides any actionable insights for future modeling approaches. I don’t necessarily view actionable insights as 100% necessary (although they would certainly add value to the paper), but upon reading the paper myself, I agree with reviewer WyYA’s assessment that Sections 3 and 4 are probably the more interesting ones and I wish the authors had dived deeper into them and perhaps had a more cohesive messaging for the paper (it is indeed packed with many, many results which makes it hard to digest). Overall however, I do think some of the results presented in this paper are surprising and will inform ongoing and future research in this field.
Decision: Accept (Poster)