Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 2: Strong Reject: For instance, a paper with major technical flaws, and/or poor evaluation, limited impact, poor reproducibility and mostly unaddressed ethical considerations.
review_content: summary: The paper starts with three claims in existing works that motivate their work---(1) the relationship between generalization and invariance are largely qualitative in existing work, (2) they are restricted to in-distribution datasets, and that (3) existing quantitative methodologies to quantify invariance do not take into account confidence of predictions. To address these, the authors propose a metric for invariance (EI), which uses both consistency and confidence of model predictions. Then, they conduct an evaluation of how generalization and invariance are correlated across a variety of architectures.
strengths_and_weaknesses: Strengths:

1. The problem is extremely important. Understanding the mechanisms driving generalization is a fundamental problem in ML. Invariance has been proposed as one potential mechanism generalization could be achieved. And so, studying their relationship in deep models is important.

2. Scale of the experiments conducted is large.

Weaknesses:

1. Severely incorrect claims: Unfortunately, there are serious flaws at in the claims made in this work. Below I list them with reasoning/citations:

a) Existing works are qualitative: There is a very long list of existing works which study this relationship quantitatively, and have reported it before [1-8]
b) Existing works are in-distribution: In [7,8,2] authors have studied specifically the problem of out-of-distribution and invariance. In fact, [2] presents new, complex datasets to study specifically this particular problem.
c) Not taking into account confidence: Invariance has been long measured in neuroscience with metrics that take into account confidence. Adaptations of this metric can be seen in [1,2,3,4]. Theoretical investigations [7,8] have also presented much more articulate representations of invariance measurement.
d) No strong quantitative correlation shown on OOD data: Both [1,2] show quantitatively that generalization positively correlates with invariance. This has also been discussed at length in previous works including [7,8].
 
2. Completely missing a large portion of literature on invariance and generalization: There is a large body of literature on invariance and generalization (beyond the ones cited below), which this paper has completely ignored. It is not placed in the context of these works, and that may be partly why the paper has made incorrect claims due to being unaware of existing works. 

3. No novel contribution: The formulation of EI is very similar to those proposed in [1], and the specialization score presented in [2]. It is what EI adds beyond existing work, as the problems mentioned in this paper as motivations are not true, and have been already addressed by past work which the paper has not referred to.

In summary: This paper seems unaware of a large subfield of existing literature in neuroscience and AI, which has led to incorrect claims about existing works. It is unclear how EI adds beyond existing works. All in all, the paper needs a major rethinking and reevaluation of the contributions in light of a thorough literature review on generalization and invariance.

References

1. https://www.nature.com/articles/s41593-018-0310-2
2. https://www.nature.com/articles/s42256-021-00437-5
3. https://papers.nips.cc/paper/2009/file/428fca9bc1921c25c5121f9da7815cde-Paper.pdf
4. https://papers.nips.cc/paper/1997/file/792c7b5aae4a79e78aaeda80516ae2ac-Paper.pdf
5. https://arxiv.org/pdf/1706.01350.pdf
6. https://papers.nips.cc/paper/1996/hash/2812e5cf6d8f21d69c91dddeefb792a7-Abstract.html
7. Poggio, T.A. and Anselmi, F., 2016. Visual cortex and deep networks: learning invariant representations. MIT Press.
8. https://academic.oup.com/imaiai/article-abstract/5/2/134/2363483?redirectedFrom=fulltext

questions: N/A
limitations: Yes.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work proposes a very simple to implement, yet effective, metric (effective invariance or EI) to assess the invariance of a model with respect to some input transformation. The main novelty of the proposed method is that it does not rely on the true label, but rather on the agreement between the predictions given an image and its transformed version.

The main findings of the paper are about the role of invariance in multi-class classification problems. The authors find that invariance (as measured by EI) is strongly correlated with accuracy for a large set of models and datasets. Moreover, it is empirically proven that EI is much better than Jensen-Shannon Divergence at capturing invariance for OOD datasets, showing that EI is a more robust metric for invariance.

An additional (and important) observation included in the paper is that training with additional data is more effective (in terms of invariance) than any of the augmentation strategies tried. 

The experimental setup is very solid, with a large breadth of models and datasets being evaluated. Moreover, the paper is well written and the language used is clear and concise.
strengths_and_weaknesses: **Strengths:**

* The proposed measure for invariance is extremely simple to implement, and well backed up by examples and experimental results.
* The experimental setup is very solid, with a large amount of state of the art models being analyzed. The distinction between fully-supervised, semi-supervised and models trained with more data is smart, and allowed the authors to reach some conclusions in terms of the use of more data in training wrt. augmentations.
The comparison with other measures for invariance in Fig. 6, Table 1 and Appendix A is very interesting, and also clearly favorable for EI.
* The paper's language is clear, I enjoyed the reading. The experimental setup is very well described, with links to all models used and datasets. I believe one of the major strengths of this paper is the experimental setup itself.

**Weaknesses:**

* Although the metric proposed is evaluated using 3 rotations of 90 degrees per image, I believe the method could benefit from using multiple transformed versions of $x$ (eg. randomly sampled rotations), and then measuring the agreement similarly to Eq. (1). However, I believe the proposed 3-way rotation is enough to prove the validity of EI in this work.

* The EI proposed in Eq. (1) is always 0 when there is a disagreement between the prediction of the unaugmented $x$ and the augmented $x^\prime$. It is reasonable to discard all cases where both predictions disagree, but many times they might mildly disagree, even within the expected standard deviation. I think that the "otherwise" case in Eq. (1) could be further refined accounting for those cases where it is not clear whether the model is invariant or not.

* All the OOD datasets analyzed are modifications of Imagenet, which was used for training. I really appreciated the breadth of OOD datasets tested, but I missed at least one dataset that is strongly OOD, for example train on Imagenet and finetune for a dataset that is completely unrelated to Imagenet. I expect EI to perform well in those cases too, so please take this as a suggestion to further improve the paper.
questions: * DNNs are known to be very sensitive to specific translations [1] even if they are "hard-wired" to be translation invariant. This sensitiveness is much stronger for other transformations for which the DNN has learnt the invariance. Could you comment on the use of three 90 degree rotations per sample to compute EI. Could this be missing many cases where the model is invariant (or not)? I believe EI could be used to "map" the invariance of a CNN in the full transformation domain, which could be extremely important to guide machine learning practitioners to improve their models.

* I would suggest including randomly initialized networks (not trained) in the study. These networks could also be invariant, but do they follow the same trend shown in the plots? Or training till convergence is a condition for EI to work? The study of randomly initialized networks could also shed some light on the invariance induced by the architectural design alone.

* What is the correlation between EI(grayscale) and EI(rotation). Intuitively, they are strongly correlated; if so, have the authors thought about understanding which of the two is the most important source of invariance for generalization?


[1] Biscione and Bowers, Convolutional Neural Networks Are Not Invariant to Translation, but They Can Learn to Be, JMLR 2021.
limitations: I believe the authors have addressed the limitations of their work correctly, pointing out:
* The scope of the paper is on multi-class classification. The study (and maybe new formulations of EI) could be extended to regression, detection, etc.
* The inherent invariances in some architectures (eg. hard-wired rotation invariance), which are not included in this study.
* The impact of *geographic shift*, not included in this work.

Additionally, the authors comment on the societal impact, which is limited since this work uses already pre-trained models. The major impact is inherited from the impact of each of these models themselves.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work considers generalization and invariance, two key properties of machine learning models. Specifically, this work conducts a large-scale quantitative correlation study between generalization and invariance on ImageNet and CIFAR setups. To help the correlation study, this work introduces the Effective Invariance Score (EI). It is more suitable for measuring model invariance than other measures such as KL and L2. Two key observations are reported: 1) generalization and invariance exhibit strong linear relationships on both in-distribution and out-of-distribution test sets; 2) when tested on different test sets, the accuracy of the model is linearly related to its invariance ability.
strengths_and_weaknesses: [Strengths]
[1] Overall, the work is well written. Motivation, focus, experimental setup, and key observations are clearly presented. Related work is well organized.

[2] The proposed Invariance Score (EI) is reasonable and well-discussed. The authors give an intuitive illustration in Figure 1. Furthermore, comparisons with other measures in the Supplementary file help to understand why EI is more suitable for correlation studies.

[3] Existing work mainly gives qualitative hints on the relationship between invariance and generalization. The main contribution of this work is it provides quantitative analysis. Furthermore, observations are presented on both in-distribution and out-of-distribution test sets. This is another contribution compared with the existing works. The results on the ImageNet testbed are comprehensive and systematic.

[4] Another interesting observation is that model invariance and accuracy also exhibit strong linear correlations on various test sets (Section 5.7). The results are reported on ImageNet-C with several models, which are helpful.

[5] In addition to the two main observations, this work investigates other points, including how the correlation on the hard test set varies (Section 5.3), and why JS is not suitable for correlation studies (Section 5.5 and in Supplementary Material). These points help to understand the relationship between model invariance and generalization.

[Weaknesses]
[1] This work quantitatively investigates the relationship between model invariance and generalization. This work points out that previous work [11-13, 15, 28] gave some qualitative hints about the positive correlation between them (Lines 23-29, 68-75). However, some recent works such as [a] and [b] are missing. They should be included and discussed.

[a] Zhu Sicheng, Bang An, Huang Furong. "Understanding the generalization advantage of model invariance from a data perspective." Advances in Neural Information Processing Systems 34 (2021): 4328-4341.
[b] Sannei, Akiyoshi, Masaki Imaizumi, and Makoto Kawano. "Improving generalization bounds for group-invariant/equivariant deep networks through quotient feature spaces." Uncertainty in Artificial Intelligence. PMLR, 2021.

[2] Continuing the above point, existing works mainly focus on the in-distribution generalization. In contrast, this work investigates both in- and out-of-distribution test sets. Differences or new contributions should be further highlighted in Section 2. The current version (lines 68-75) is not very clear.

[3] This work conducts correlation studies on the ImageNet testbed, which includes several test sets. These test sets are diverse and come from different distributions (Lines 154-157). It is good and convincing. Another test set is ObjectNet, where object backgrounds, rotations, and viewpoints are diverse. It would be better if this work could include this test set in the correlation study.

[4] Section 5.7 should be highlighted as it presents another key observation. However, it is not heightened in the current version. In addition, the results in the Supplementary Materials (Figure A-3) should be included in the main paper, which helps illustrate the observations.
questions: The following questions and suggestions are expected to be addressed in the rebuttal.

[1] First, lines 97-105 explain why KL is not used for correlation studies. I understand why KL and other measures are not suitable. However, using "Drawback of existing invariance measures" to summarize the paragraph is not very coherent and is a bit confusing.

[2] Also, in semi-supervised learning, other additional techniques such as confidence [48] and sharpness [49] are used when KL [50, 51] or L2 are used for maintaining consistency. These techniques may avoid the problems illustrated in case (c) and case (d) in Figure 1. It can also be mentioned on lines 97-105 to help illustrate why KL and L2 are not suitable for correlation studies at work.

[3] As illustrated in the weaknesses, a correlation study on ObjectNet would be interesting.

[4] In addition, related works (eg, [a,b]) should be included in this work. Section 5.7 should be highlighted (see Weaknesses above).

limitations: In Section 6, the authors clearly discuss potential limitations and propose possible solutions and potential research directions. Potential negative social impacts are also discussed.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The authors design an empirical measure of model invariance based on prediction consistency and invariance. With this measure they evaluate the correlation between model invariance (to rotation and grayscale) and generalization on several in-distribution and out-of-distribution ImageNet variants. They show that the model’s accuracy and invariance are linearly correlated, and that such correlation is better captured by their designed measure than other existing measures.
strengths_and_weaknesses: ### Strengths:
1. The linear relationship between model generalization and the invariance captured by the introduced measure is interesting, especially considering that the measure is handcrafted to have a specific form. 

2. The evaluation covers the most recent model architectures, including several vision transformers.

### Weaknesses:
1. Current evaluation datasets are not comprehensive enough to draw convincing conclusions about the introduced measure. Most of the results are based on ImageNet variants, whereas for CIFAR-10 the authors do not do any comparison of measures. For such a handcrafted measure, readers may need similar observations on more diverse datasets, including some existing OOD benchmarks.

2. To make fair comparisons among different invariance measures, Spearman’s rank correlation is arguably a better choice than Pearson’s correlation, since different measures have different algebraic forms so a simple linear correlation may be too restrictive. I’m glad to see the authors use both correlation measures for evaluation. Nevertheless, some results are missing, including Spearman’s rank correlation for JS-divergence on ImageNet-S and ImageNet-A. Also, if we focus on Spearman’s rank correlation, the introduced measure is no longer superior in predicting generalization among other measures like the prediction consistency.

3. The motivation for designing the EI measure is subjective — why does a model making confident mistakes have the same invariance as a model making unconfident mistakes (Figure 1 (c) and (d))? If we look at the invariance from the loss perspective, the former model usually incurs a higher consistency loss than the latter one, hence less invariant. I recommend the authors to elaborate more on this motivation.

questions: Could you also evaluate the Spearman’s rank correlation for JS-divergence on ImageNet-S and ImageNet-A? This could make the comparison fairer.
limitations: Yes, the authors addressed the limitations and potential negative societal impact of their work.
review_title: null

Meta-review: metareview: This work proposes a very simple to implement, yet effective, metric (effective invariance or EI) to assess the invariance of a model with respect to some input transformation. The main novelty of the proposed method is that it does not rely on the true label, but rather on the agreement between the predictions given an image and its transformed version.

The committee appreciates the comprehensiveness of the empirical evaluation conducted in the paper. Although theoretical analysis on how invariances improves generalization exists, as pointed out by the reviewers (missed in the paper's initial version), this paper performs a large-scale quantitative correlation study using various models and different test sets and empirically reports that model invariance and generalization exhibit a strong linear correlation on both in- and out-of-distribution test sets. 

There is a heated discussion about the novelty of this work, as one reviewer pointed out that the authors missed a large subfield of existing literature in neuroscience and AI. The rest of the committee, however, does recognize the differences and believes that the efforts of the large scale experimental evaluations outweigh. The authors, however, are strongly suggested to provide a detailed comparison of the mentioned works in the revised paper. 
confidence: Less certain
Decision: Accept