On the Strong Correlation Between Model Invariance and Generalization
Authors: Weijian Deng
Stephen Gould
Liang Zheng
Venue: NeurIPS 2022 Accept (2022)
Keywords: Model Invariance
Model Generalization
Correlation Study
Unsupervised Model Selection

Abstract: Generalization and invariance are two essential properties of  machine learning models. Generalization captures a model's ability to classify unseen data while invariance measures consistency of model predictions on transformations of the data. Existing research suggests a positive relationship: a model generalizing well should be invariant to certain visual factors. Building on this qualitative implication we make two contributions. First, we introduce effective invariance (EI), a simple and reasonable measure of model invariance which does not rely on image labels. Given predictions on a test image and its transformed version, EI measures how well the predictions agree and with what level of confidence. Second, using invariance scores computed by EI, we perform large-scale quantitative correlation studies between generalization and invariance, focusing on rotation and grayscale transformations. From a model-centric view, we observe generalization and invariance of different models exhibit a strong linear relationship, on both in-distribution and out-of-distribution datasets. From a dataset-centric view, we find a certain model's accuracy and invariance linearly correlated on different test sets. Apart from these major findings, other minor but interesting insights are also discussed.

On the Strong Correlation Between Model Invariance and Generalization
Weijian Deng Stephen Gould Liang Zheng
Australian National University
(firstname lastname) edu.
Abstract
Generalization and invariance are two essential properties of machine learning models. Generalization captures model's ability to classify unseen data while invariance measures the consistency of model predictions Existing research suggests a positive relationship: a model generalizing well should be invariant to certain visual factors. Building this qualitative implication we make two contributions. First, we introduce effective invariance (EI), simple and reasonable measure of model invariance which does not rely image labels. Given predictions on a test image and its transformed version, EI measures how well the predictions agree and with what level of confidence. Second, using invariance scores computed by EI, we perform large-scale quantitative correlation studies between generalization and invariance, focusing rotation and grayscale transfor- mations. From a model-centric view, we observe generalization and invariance different models exhibit a strong linear relationship, on both in-distribution and out-of cutoof-distribution datasets. From dataset-centric view, we find a certain model's s accuracy and invariance linearly correlated different test sets. Apart from these major findings, other minor but interesting insights are also discussed.
transformed data.
1 Introduction
Generalization and invariance are two important model properties machine learning. The former characterizes how well a model performs when encountering in-distribution or ofttoo-diisributtin (OOD) test data [1-5]. The latter assesses how consistent model predictions are on transformed test data [6-12]. Therefore, understanding how the two properties are related would benefit model decision analysis under dynamic environments.
The importance of model invariance on generalization has been qualitatively explored [9, 12-15]. For example, adding rotation invariance to the model improves its in-distribution (ID) classification accuracy [14, 16, 17]; a
model is robust to perturbation [9]. In addition, some works provide a neuroscientific perspective investigate the importance of invariant representations for pattern recognition [13, 15, 18-22]. Furthermore, theoretical investigations suggest that learning invariant features benefits model generalization [12, 22]. However, most existing research limited to few ID) datasets and model architectures. As such, the relationship of interest remains unknown in many other scenarios, such as OOD and large-scale test datasets, and other types of models.
Before performing the quantitative study, it is necessary to first quantify generalization and invariance. For the former, the deep models we consider are well trained, so simply use the accuracy on the test set, as in many previous works [3, 2, 23]. In comparison, quantifying invariance is not as straightforward. Some works use model accuracy drop when the test set undergoes transformations to indicate invariance ability [24, 7, 17]. While this strategy is useful for single model, its effectiveness limited when comparing the invariance of multiple models. Others resort to consistency, i.e.,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

models should have the same decision [9, 6]. but this method neglects prediction confidence, which we find critical for describing invariance (see discussions in Section 3).
We make two contributions to the community. First, we propose a new method to measure model invariance, named effective invariance (EI), which considers both the consistency and confidence of predictions. Given a test image and its transformed counterpart, if the model predicts the same class with high confidence, the EI value invariance strength is high. Otherwise, if the model makes different class predictions or the confidence is low, the El score will be low. We show this new measure solves invariance valuation in canonical cases where the commonly used metrics (e.g.. Jensen-Shannon divergence) may fail. Second, we conduct broad correlation study to quantitatively understand the relationship between model generalization and invariance. Specifically, we use test sets with various distribution types, such as the in in-distribution ImageNet validation set [25], and out- of-distribution CrageNet-RReditiee from traditional convolution neural networks (VGGs [26]) to the very recent vision transformers (e.g.- BEiT [27D. Below we list two key observations and example insights.
with style shift [4]. We evaluate 150 ImageNet models ranging
For various models, there is a strong correlation between their accuracy and invariance on both in-distribution and out-of-distribution datasets (Sections 5.1 and 5.2). This finding can be useful for unsupervised model selection because El does not require test ground truths.
On various out-of-distribue lated (Section 6). This observation can be used to predict model accuracy on out-of-distribution datasets without access to ground truths.
datasets, a model's accuracy and EI scores are also strongly corre-
Compared with data augmentation, training with more data seems more effective to improve invariance and generalization (Section 5.6).
2 Related Work
Predicting generalization gap: a model-centric view. This task aims to predict the generalization gap of machine learning models on in-distribution data, i.e., the difference between training and test accuracy. Most existing works focus on developing complexity measure of trained network parameters and training data [28-36], such as persistent topology [31] and the product of norms of the weights across layers [33]. These methods, assuming the training and test distributions are the same, do not consider the characteristics of the test distribution. Moreover, they only study limited types of neural networks. In comparison, we conduct much more comprehensive study on both nn-distribution and out-of-distribus that invariance, under our definition, serves as a strong indicator of model generalization ability or accuracy on both in-distribution and -uttof-distribution test data.
test sets, using various network architectures. We show
Closely related to our work, two recent methods [37, 24] predict ID generalization gap based on how network performs perturbed data points. Specifically, Kashyap al. [37] use confidence drop to represent invariance, which is less effective for invariance measurement under OOD data. Schiff er al. [24] uses accuracy drop to measure invariance, which needs test labels, while El does not require test labels and is more reasonable than accuracy under OOD data. In addition, drawing response curve [24] is computationally heavy, while our method relatively efficient. Further, both studies are limited in their scope: they mainly study ID generalization, has few types of networks and lack large-scale test sets, while our work is much more comprehensive.
Predicting generalization gap: a dataset-centric view. The overall goal of this task is to predict the performance of a given model on various unlabeled test sets [38-43]. Many methods take into account the statistics of the test set for accuracy prediction [38, 39, 44, 40, 45], such as distribution shift [38], average Softmax score on each test set [39]. We contribute new solution: using the model's invariance on the OOD dataset to predict its accuracy. This is supported by our new observation of a strong linear correlation between certain model's accuracy and invariance on various test sets.
Improving robustness with data augmentation. Data augmentation transforms training data to increase its diversity, which helps learn more robust models [46-52]. For example, Mixup [53, 54] and AutoAuugment [48] are shown to improve model performance under distribution changes [55, 51].
Instead of using common transformations, adversarial training [56-58] augments training images with an adversarially learned noise distribution. While these works aim to improve corruption robustness with data augmentation, we instead use the latter to analyze model invariance (and generalization).

---

9=
g 9,
g 9
15-0.076
J5-0.019
IS-0 665
15-0.029
f(x')
0.5
EI-0.63
0.5
B1-.1112
0.5
0.5
(b)
(c)
(d)
An illustrative comparison of El and censen-Shannon divergence (JS) as invariance measures. Four representative cases are shown, where for each class (cl-c4) show the Softmax output of the original image f(x) and the transformed image fæx'). On the one hand, the model exhibits higher invariance in (a) than (b), because it makes the same class predictions (ú (t) and has higher confidence in (a). This is correctly reflected by EI (0.630 vs. 0.112; higher is better), but IS incorrectly decides the opposite way (0.076 vs. 0.019; lower is better), because IS does consider confidence explicitly. In cases (c) and (d), the model makes different class predictions ( # út), its invariance should be very low. This is again correctly captured by EI (0 value for both cases), but IS erroneously gives high invariance to (d), due to the fact that JS merely looks at the global shape of the Softmax vectors without explicitly considering class prediction consistency.
Figure I:
3 Proposed Effective Invariance (EI)
Notations. Considering an K K-way classification task, define input space X' Rd and label space y = [1,..., KJ. Given a sample (æ, y) drawn from an unknown distribution и on X' x V, neural network classifier f _R + AK produces a probability distribution for æ on K classes, where Ak denotes the K- dimensional unit simplex. Specifically, f.(x) denotes the i-th element of the Softmax output vector produced by f. Then, 9 =: arg max; (x) is the predicted class, and D =: max; (( the associated confidence score. Image transformation is defined as T IRd Rd Then, the transformed image is x' Tax), and its predicted class is ÿt =: arg max; fi(w'" with confidence score P.: =: max; f((x").
Drawback of existing invariance measures. A commonly seen strategy is to directly use the distance between the Softmax vectors of two predictions as an invariance measure: lower distance means higher invariance, and vice versa. Examples of the similarity metrics are Lensen-Shannor divergence (JS) [59, 46] and & distance [60, 61] and Lelbeack-Leibler divergence [62, 63]. However, they only leverage the global similarity between two Softmax vectors without explicit consideration of prediction class consistency and confidence. We illustrate this drawback by taking JS divergence as an example in Fig. 1. In cases (a) and (b) where the predicted classes are both consistent, JS decides classifier f in (b) has higher, which ignores the low confidence in (b). cases (d) where predicted classes are different, JS still gives high invariance (small JS score), indicating clear error. Moreover, works [46, 60, 61] cases (b) and (d) are discarded when computing the consistency loss.
Definition of EI. Unlike neuroscience works that study the invariance of an individual neuron of the network [13, 15, 18, 19], we measure the invariance the network level. Specifically, given an image and its transformed sample, model with high invariance should give the same predicted class, and vice versa [6, 9]. our definition of EI, we further use prediction confidence. Our motivation as follows. When model predicts the same class for the two images, if either of the two predictions of low confidence, we should not consider have a high invariance if and only if it is highly confident in predicting the same class, Based these considerations, El defined as:
highly invariant but give a penalty. model should
Vp.
po
gf = 9: otherwise.
El T(x),
To better understand the soundness of EI, we depict four representative cases in Fig. 1. In cases (a) and (b), classifier f gives the same predicted class (j = ÿi) on original and transformed images. Under EI, Classifier f has higher invariance ability in (a) because it has high confidence scores. In case (c), the predicted class are different (ú + ūt), and in (d), the predictions are low in confidence (and give different classes), so define invariance in both cases to be O.

---

Computation of EI in practice. Given a test image, we generate a transformed image using certain transformation. Then, we compute the El score based on their Softmax vectors (Eq. 3). We obtain the model invariance by averaging the El scores over all the test. images. In this work, we mainly investigate the rotation transformation and grayscale transformation. For the former, to avoid interpolation that would introduce artifact, we only use three transformation angles (90", 180", 270%). For each rotation angle, we compute an overall invariance score by comparing with the predictions of the original data. By averaging the three EI scores, we obtain rotation invariance on the test set. For the grayscale transformation, we remove color information and keep only luminous intensity information. Then, we compare the predictions of grayscale and original data and compute the overall grayscale invariance on each test set.
4 Experimental Setup
4.1 Models to Be Evaluated
We consider both very recent and classic image classification models with different architectures, including Convolutional Neural Networks (e.g., standard VGGs [26], ResNets [64], and modern ConvNeXt [65], Vision Transformers (e.g., ViTs [66], Swin [67], and BEiT [27]), and all-MLP architectures [68, 69] (i.e, MLP-Mixer [69]).
In addition different architectures, we also cover models with various training and regularization strategies (e.g., learning rate schedule [70], label smooth [71] and data augmentation [46, 47, 49, 48]), scaling strategies model dimension (width, depth, and resolution) [72-74], and learning manners (supervised learning, semi- -umissupee learning [75] and knowledge distillation [76, 77J). In total, we have 150 models provided by TIMM [78]. They are either trained or fine-tuned on the ImageNet-Ik training set [25]. The selected models can be roughly divided into the following three categories:
Standard neural networks. This category includes 100 models only trained ImageNet training set. These networks cover various architectures ranging from VGGs [26] to EfficientNet [73].
Semi-supervised learning. We include 15 models trained in a semi-supervised learning manner. They leverage a large collection of unlabelled images of YFCC100N [79] or Instagram 900M [80] to improve the performance. We use models trained based SWst-ResNet [75] and SSL-ResNet [75]). Models trained with self- teff-training methods on unlabeled JFT-300M [81] (e.g. EfficientNet-L2-NSS [82]) are also included.
ceacher-studeny paradigm (e.g.,
Pretraining on more data. We use another 35 models that are pre-trained on significantly larger datasets than the standard ImageNet training set. Specifically, we consider three pre-training methods: a) weakly supervised pretraining on IG-3.6B (i.e., RegNetY [83] and ResNeX:101-Wo [80]);
b) supervised pre-training ImageNet-21K [25] (e.g., BiT [84] and Swin [67]); (c) supervised pretraining on JFT-300M [81] (e.g., ViT L/16 [66]).
4.2 Test Sets
We use both in-distribution (ID) and Specifically, the ImageNet validation set (ImageNet- Val) is used as ID test set. For OOD test sets, we use seven datasets, each with a different distribution from standard ImageNet. Their distribution shift can be divided into the following five types.
(OOD) datasets for the correlation study.
Dataset reproduction shift. ImageNet-V2 [23] is a recollected version of ImageNet- Val. It contains three versions resulting from different data sampling strategies: Batched-FFequency (A), Threshold- 0.7 (B), and Top-Images (C). Each version has 10, 000 images from 1, 000 classes.
Natural adversarial shift. ImageNet-Ade -dverrrrriaaaaaaiiaaa [85] is adversarially selected to be misclassified by ResNet-50. Its natural adversarial examples are unmodified real-world images and have been shown to be hard for other models [85, 3]. It has 7, 500 samples from 200 ImageNet classes.
Sketch shift. ImageNet-S(kette categories and scale. It contains 50, 000 images and shares the same 1, 000 classes as ImageNet.
[86] consists of sketch-like images and matches ImageNet- Val in
Blur shift. We use ImageNet- with the highest severity provided by [87]. This dataset is synthesized by blurring ImageNet- Val images with Gaussian function.
Style shift. and video game) of ImageNet classes. ImageNet-R has 30, 000 images of 200 ImageNet classes.
[4] contains various abstract visual renditions (e.g., art, paintings,

---

Standard madels
Trained with semi semi supervised learning
Pre trained on more data
Lincar fit
ImageNet-Val
ImagoNat-VV-8
yno 929
y=0.538
y=0.521
accuracy (5)
72
80
70
60
0.4
0.5
0.7
0.2
6.3
0.4
6.5
677
9.3
0.4
0.5
66
0.7
rotation invariance score
rotation invariance score
rotation Invariance score
ImageNet V2-C
ImageNec-5
ImageNet-R
y=0.527
y-1.915
y-0.925
p-0
p=0.807
p.0.846
accurscy
80
70
40
50
75
rotation invariance
rotation invariance
rotation
Figure 2: Correlation between accuracy (%) and rotation invariance (ET) for 150 models. Each figure is obtained from testing a different ImageNet test set. In each figure, each dot denotes model, and straight lines are fit by robust linear [90]. The shaded region in each figure a 95% confidence region for the linear fit from 1, 000 bootstrap samples. We clearly observe strong linear relationship (Pearson's ' Correlation r > 0.915 and Spearman's Correlation p > 0.875).
4.3 Correlation Measures
We Pearson Correlation coefficient (r) [88] and Spearman's 's Rank Correlation coefficient (p) [89] to measure the linearity and monotonicity between invariance and generalization, respectively. Both coefficients range from [-1, 1]. value closer -1 indicates strong negative positive correlation, respectively, and implies no correlation [88]. To precisely show the correlation, we use logit axis scaling that maps the accuracy range from [0, 1] [-o0, +oo], following [3, 2]. Unless noted otherwise, the correlation coefficients are calculated using invariance score and non-linearly scaled accuracy numbers.
5
Experimental Observations
The experiment is from model-centric perspective, where we investigate how different models' accuracy and invariance correlate, and a series of observations are made (Section 5.1 Section 5.6).
5.1 Strong Correlation Between Model's Rotation Invariance and Accuracy
In Fig. 2, we show the correlation results of rotation invariance and generalization. We have two observations. First, we find that for different models, their rotation El scores have a linear relationship with their classification accuracy. The correlation holds for both ID test and OOD test sets, various architectures, and training strategies. Specifically, both correlation metrics À and are higher than 0.840. This indicates models with higher accuracy numbers most likely to have stronger rotation invariance (measured by El), and vice versa. To our knowledge, is a very early observation of the quantitative relationship between generalization and invariance (to certain factor).
Second, training with more data benefits rotation invariance and generalization. Large datasets contain images with various geometric variations. When (pre)trained with large datasets, models (blue and orange dots in Fig. 2) adapt the rotation variations and gains stronger invariance, which, according to study, likely means higher generalization accuracy on ID and OOD test sets.
5

---

Standard models
Trained with semi-supervised learning
Pre-trained on more data
Linear fit
VnaaeNottVaal Val
ImageNic-V2-A
ImageNet V2-B
ymb..668
y-0.300
y-0.8ES
p.0.831
prd.124
accuracy
70
20
70
20
60
50
40
60
0.5
0.8
0.6
2.8
0.7
0.3
0.5
6.7
grayscale inveriance
grayscale invoriance
grayscale invariance
ImageNNet-S
ImageNettR
y=q 888
y=0.942
y-0.940
clesification accurary
p=0 p=0.856
p-0.875
p=0.900
10
3ea
10
50
20
grayscale invariance
grayscale invariance
grayscalo invariance
Figure 3: Correlation between accuracy (%) and grayscale invariance (El) of 150 models. Similar to Fig. 2, each dot denotes model, where different colors denote different training strategies (see Section 4). The subfigures correspond to three ImageNet test sets, respectively. We observe the correlation is also strong: Pearson's Correlation 0.840, and Spearman's 's Correlation p > 0.820.
Standard models
Trained with semi supervssee learning
Pre-troined on more data
Imagahizz
Imagehen-Wiir
y-0.910
y-0.902
7-0.514
Top- accuraoy
708
p-0.858
35
02
64 K5
0621 1.5
0175
0425
$115
cotation inariance score
grayscale invariance soore
rotation invriince score
grayscale invoriance score
Figure 4: Correlation study very hard test sets. We the very hard ImageNet-A and Image- Blur for testing, where the accuracy of 83 models is lower than 20%. Both rotation invariance and grayscale invariance are evaluated. Overall we observe relatively solid correlation in all four cases. Looking more closely, most standard models (gray dots) are scattered in the low-accuracy region, while models trained with more data (blue and orange ones) move away from this region and exhibit linear trends. Thus, the overall linear correlation high (r > 0.880), and the overall rank correlation slightly less consistent (p ranges from 0.703 0.858) but still has clear trends.
5.2 Strong Correlation Between Model's Grayscale Invariance and Accuracy
We now focus on grayscale invariance and report the correlation results in Fig. 3. We have the following conclusions. First, among the 150 models, there a strong linear correlation between accuracy and grayscale invariance measured by EI. Specifically, both correlation coefficients and are higher than 0.820 on all test sets. Second, we find that pretrained or semi-supervised models tend to have higher grayscale invariance and accuracy than standard models, which again indicates the usefulness of large training sets. Third and interestingly, the correlation is stronger on ImageNet-R than ImageNet- Val and ImageNet-V2--. V2--A (0.940 vs. 0.900 vs. 0.868). In fact, ImageNet-R is featured by style shift during its collection [4], so for this test set being invariant to color changes is an important property for stronger generalization ability.
5.3 Correlation Exists on Very Hard Test Sets
We now study the correlation under very hard test sets and use ImageNet-A and ImageNet-GGaussBll for testing, which the accuracy of 83 models is lower than 20%. We study rotation invariance and

---

closifiiatioo aicuuaayy (%)
CIAM--50 Text 7-0 sia
CIFAMI0 7+0.856
15
yau 917
p-o 934
p-d 527
80
75
0.20
5:25
0.10
2.40
0.225
0.325
0.373
0.34
rotation inveriance score
rotation invariance score
rotation invariance score
10
20
80
classification accuracy (9)
CIV 10
y-1 loXs
A-
975
70
00
0.60
0.75
0.35
0.00
5.05
0.85
0.00
grayscale invariance score
grayscale invariance score
grayscale invariance score
Figure 5: Correlation study under the CIFAR-10 setup, Each data point is CIFAR model. We report the correlation results with rotation invariance (top) and grayscale invariance (bottom). We observe that strong correlation exists between accuracy and invariance on all three test sets.
supervised learning Unear
Pre-trained more data
Test set
JS
EI
Val y
Img.-
-0.861 /-0.905
+0.929 70.927
100 decuooo 189
0.934
Img.
-0.173//0.172
+0.928 70.846
Img.-S
+0.131 /40.148
+0.915
Imagzhet-
7--23722
Img. A
-0.297 /-0.257
A0.778
56
3.10
5.35
2.2>
rotation inveriance score measured by IS
Table 1: Comparing El and JS. Un- der Pearson's 's Correlation r Spearman's 's Rank Correlation p. JS does not show a strong correlation on most datasets, while El does on all four datasets.
Figure 6: Correlation between accuracy (%) and invariance (JS). Despite of the good correlation on Vall the correlation on ImageNet-R, harder test set, weak. More results and analysis are provided in the supplementary materials.
grayscale invariance. In Fig. 4, we find strong linear correlation in the four cases (r < 0.88). The rank correlation is less consistent but still indicates clear trends. Moreover, most standard models have low accuracy, while models (pre)traines) with more data tend to have high accuracy and invariance. We also notice standard models are scattered differently in the fow-accuracy regime of ImageNet-A and ImageNet-Bile -lur, possibly due to their dataset bias introduced during dataset construction [91-93].
5.4 Correlation Holds Under the CIFAR-10 Setup
We conduct correlation study the CIFAR-10 setup. We collect 90 CIFAR models ranging from
LeNet to EfficientNet. We use the ID CIFAR-I0 test set and two OOD test sets. 1) CIFAR-I0.1 [94] is reproduction of the CIFAR-10 test set but with distribution shift arising from changes in data collection. It contains 2, 000 test images sampled from Tinylmages [95]. 2) CINIC-10 10 test set [96] is an extended alternative for CIFAR-10. It has 90, 000 images sampled from ImageNet.
We show the relationship between model generalization and invariance Figure 5. On all three test sets, we observe strong correlation between model accuracy numbers and rotation invariance scores, where both r and p are greater than 0.90. Moreover, when testing grayscale invariance, strong correlation still exists (both and p are greater than 0.83).

---

Trained with diverse sugmentation
Trained on more data
20
90
Imagehiit-A
ImageNett-R
ImageNett-all
Top accuuaacy (66)
FMG So
Farti Sedtenna
08
35
2004 Rio lot
Baokee
Resken-s0
13L WX..oMM--S 50
40
Staddrd Barlen--
0.25 0.20
0.30
0.33 2.40 0.45
rotation invariance
rotation invariance score
rotation invoriance score
Figure Comparing data augmentation and (pre)training with more data. We use 14 ResNet-50 models (red dots) trained with various types of augmentation methods, and 3 ResNet-50 models (green dots) trained on more data (real-world). We also mark ResNet-50 trained with the standard learning strategies. On ImageNet- Val and ImageNet-R, we observe models trained with diverse augmentation follow linear trend. However, they deviate from the trend on ImageNet-A. In comparison, training with more data allows models to achieve relatively high accuracy and invariance on three test sets.
5.5 El Gives Stronger Correlation Than JS
In the representative cases (Figure 1), EI shows superiority to JS in measuring model invariance by explicitly considering Softmax consistency and confidence. Now we compare the invariance scores measured by EI and JS w.r. their correlation strength with accuracy in Figure and Table We find that JS provides a good correlation on the ID ImageNet-Vai test set, but much weaker correlation on the more difficult OOD tests than EI. As discussed in Section 3, the advantage of El is that only follows the definition of invariance but also integrates confidence to strengthen it, while JS only compares the overall Softmax vector. fact, the drawback of JS is primarily reflected in hard test sets, where the Softmax vector usually has flat shape. This explains why JS does not give a strong correlation on the OOD test sets (see Section 5.3 for El's performance on very hard test sets). We refer readers to the supplementary material for comparisons with other measures.
5.6 Comparing Two Training Manners of Their Generalization and Invariance
Existing works report that using more diverse training data artificially (i.e,, data augmentation) or naturally (i.e., more real-world data) improves model invariance [4, 46-51, 97]. In this study, compare the two strategies of their generalization and invariance abilities. We use 14 ResNet-50 models trained with strong data augmentation such as PixMix [52] and AutoAugment [48]. For comparison, we employ another 3 ResNet-50 models that learn invariance using more training samples: SWSL-ResNNt-500, SSL-ResNet-500, and ResNet-50- 50---MM..... In Fig. 7, we observe that models with heavy and strong augmentation exhibit linear trends on ImageNet- Val and ImageNet-R, but deviate on ImageNet-A. In comparison, models (pre)drained with more data follow the linear trend on three test sets. The latter seems more effective in improving invariance and accuracy compared with data augmentation, especially on ImageNet-A. To thoroughly understand and extend this initial observation, we will conduct more comprehensive experiments on this specific point in the future.
Correlation of A Model's Invariance and Generalization on OOD Test Sets
From a datasef-centric perspective, we study the relationship between generalization and invariance of given model different OOD test sets. Given model, we calculate its accuracy every test set of ImageNet-C [87] and compute its rotation and grayscale invariance scores.
We evaluate ResNet-152 152, ViT-Base-16 16, and DenseNet-121 classifiers for the correlation study. As shown in Fig. 8, there is a very strong correlation between classifier accuracy and invariance on various datasets (r > 0.93 and 0.95). The results indicate that a classifier tends to have high accuracy on the test set where it has a high score. The above analysis indicates that it feasible to use El to access the out-of-distribution error of model.
8

---

(96)
ynd BSA
prd s00
70
ynd
Reshet-152
.0.0
70
p-0.558
2-0536
70
20
10
20
20
10
0.2
2:00
6:25 rotation invariance score
0.05
0:30
0.25
0.25
0.30
rotation inveriance score
rotation invariance score
Top- accuracy (9)
80
yon
VIT Seee Paten
70
yob
70
DenseNett-12
70
50
20
20
10
10
0.2
3.1
0.3
grayscale invariance score
grayscale Invariance score
grayscale Invariance score
Figure& 8: Correlation between model's invariance and accuracy various OOD test sets. In each figure, data point corresponds to a test set from ImageNet-C [87]. The straight lines are fit with robust linear regression [90]. We test rotation invariance (top) and grayscale invariance (bottom). In all subfigures, we observe a strong negative correlation (Pearson's 's Correlation r and Spearman's s Rank Correlation p are greater than 0.930) between invariance and accuracy.
Conclusion
This work considers critical properties of machine learning model: invariance and generalization. To study their relationship. we first introduce effective invariance (EI) to more reasonably measure invariance and then provide in-depth and comprehensive correlation experiment. From model- centric perspective, we report accuracy and El various models have a strong linear relationship on both ID and OOD datasets, which validated on many scenarios such as large-scale test sets, CIFAR-10, 10, and very hard test sets. From a dataset-centric perspective, we show the accuracy and El of model have strong linear relationship on various OOD datasets.
Limitations and potential directions. First, some networks with specially designed modules can be highly invariant to some transformations [7-9, 16, 98], such as rotation invariance networks [14, 99, 17]. The rotation invariance scores of these models may present very different correlations from observations. That said, their color invariance scores may still exhibit similar correlation with this work. In future works, would be interesting to understand how these models deviate from others. Second, ImageNet test sets do not include the geographic shift where images are captured from various locations [4]. Recent works show this shift is also a key factor influencing model accuracy [100]. We leave this question future study. Moreover, we focus on classification tasks, where models are supervised by image-level annotations. In other computer vision tasks, models may be trained with different levels of supervision, such instance-Ieve bounding box annotations [101], pixel-level labels [102] and temporal context supervision [103]. Different types of supervision may lead to invariance ability to various factors, which may exhibit different correlation profiles. Lastly, in designing EI. we mainly study four representative cases and explain the limitations of some commonly used measures (e.g., JS and 1.2). Considering more special cases would be beneficial.
Potential negative social impact. We study fundamental model properties with public classification datasets, which might be misused in certain applications with ethical concerns.
Acknowledgments and Disclosure of Funding
We thank all anonymous reviewers for their constructive comments in improving this paper. This work was in part supported by the ARC Discovery Early Career Researcher Award (DE200101283 and the ARC Discovery Project (DP210102801). Stephen Gould is the recipient of an ARC Future Fellowship (project number FT200100421) funded by the Australian Government. Weijian Deng is recipient of the Australian Government Research Training Program (RTP) Scholarship.

---

References
[1] Shai Ben-David. John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Vaughan. theory of learning from different domains. Machine Learning, 79:151-175, 2010.
[2] John Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Perey Liang. Yair Carmon, and Ludwig Schmidt. Accuracy on the line: on the strong correlation between out-of-distribution and in-distribution generalization. In Internationnl Conference on Machine Learning, pages 7721-7735, 2021. 1,5
[3] Rohan Taori, Achal Dave, Vaishaal Shankar, Nicholas Carlini, Benjamin Recht, and Ludwig Schmidt. Measuring robustness natural distribution shifts image classification. Advances in Neural Information Processing Systems, 33:18583- 18599, 2020. 1, 4,
[4] Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang. Evan Dorundo, Rahul Desai, Tyler 7hu, Samyak Parajuli, Mike Guo, et al. The many faces of rohustness: A critical analysis of out-of-distribation generalization. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 8340-8349, 2021, 2, 4,4, 6, 8,9
[5] Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understanding deep learning (still) requires rethinking generalization. Communications the ACM, 64(3): 115, 2021.
[6] Aharon Azulay and Yair Weiss. Why do deep convolutional networks generalize poorly to small image transformations? arXiv preprint arxiv: 1805. /2177, 2018. 2,
[7] Logan Engstrom, Brandon Tran, Dimitris Tsipras, Ludwig Schmidt, and Aleksander Madry. Exploring the landscape of spatial robustness. International Conference on Machine Learning, pages 1802-1811, 2019. 1,
[8] Can Kanbak, Seyed- Doosavi-Dezfooli, and Pascal Frossard. Geometric robustness of deep networks: analysis and improvement. I Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pages 4441 14449, 2018
[9] Richard Zhang. Making convolutional networks shift-invariant again. In International conference on machine learning, pages 7324-7334, 2019. 2,
[10] Alessandro Achille and Stefano Soatto. Emergence of invariance and disentanglement in deep representa- tions. The Journal of Machine Learning Rexearch, 19(1):9947-188, 2018.
[11] Osman Semih Kayhan and van Gemert. On translation invariance in cnns: Convolutional layers can exploit absolute spatial location. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14274-14285, 2020.
[12] Sicheng Zhu, Bang An, and Furong Huang. Understanding the generalization benefit of model invariance from data perspective. In Advances in Neural Information Processing Systems, volume 34, pages
4328 4328-4341, 2021. 1
[13] Guangyu Robert Yang, Madhura Joglekar, H Francis Song, William T Newsome, and Xiao-Jing Wang. Task representations in neural networks trained to perform many cognitive tasks. Nature neuroscience, 22 (2):297-306, 2019. 1,3
[14) Yanzhao Zhou, Qixiang Yc, Qiang Qiu, and Jianbin Jiao. Oriented response networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 519-528, 2017. 1,9
[15] Maximilian Riesenhuber and Tomaso Poggio. Just one view: Invariances in inferotemporal cell tuning.
1997. 1,3
[16] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and koray kavukcuoglu. Spatial transformer networks. In Advances in neural information processing systems, 2015. 1,
[17] Valentin Delchevalerie, Adrien Bibal, Benoît Frénay, and Alexandre Mayer. Achieving rotational invariance with hessel-convolutional neural networks. Advances in Neural Information Processing Systems, 2021.
[18] Spandan Madan, Timothy Henry, Jamell Dozier, Helen Ho, Nishchal Bhandari, Tomotake Sasaki, Frédo Durand, Hanspeter Plister, and Xavier Boix. When and how convolutional neural networks generalize to category viewooint combinations. Nature Machine Intelligence, 4(2):146 (000:146-13,,,,0 2022.
3.3
10

---

[19] Ian Goodfellow, Honglak Lee, Quoe Le, Andrew Saxe, and Andrew Ng. Measuring invariances in deep networks. In Advances in neural information processing systems, 2009. 3
[20] Emanuela Bricolo, Tomaso Poggio, and Nikos K Logothetis. 3d object recognition: A model of view- tuned neurons. 1996.
Tomaso A Poggio and Fabio Anselmi. Visual cortex and deep networks: learning invariant representations. [21]
MIT Press, 2016.
[22] Fabio Anselmi, Lorenzo Rosasco, and Tomaso Poggio. On invariance and selectivity in representation learning. Information and Inference: A Journal of the IMA, 5(2):134-158, 2016.
[23] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do imagenet classifiers generalize to imagenet? In International Conference on Machine Learning, pages 5389-5400. PMLR, 2019. 1,4
[24] Yair Schiff, Brian Quanz, Payel Das, and Pin-Yu Chen. Predicting deep neural network generalization with perturbation response curves. In Advances in Neural Information Processing Systems, 2021. 1,2
[25] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattenn Recognition, pages 248-255, 2009. 2,4
[26] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409.1556, 2014, 2,
[27] Hangho Bao, Li Dong, and Furu Wei. Beit: Bert pre-training of image transformers. arXiv preprint
2021. 2, 4
[28] Gabriel Bilertsen, Daniel Jönsson, Timo Ropinski, Jonas Unger, and Anders Ynnerman. Classifying the classifier: dissecting weight space of neural networks. arXiv preprint arXiv:2002.05688 2020.
[29] Thomas Unterthiner, Daniel Keysers, Sylvain Gelly, Olivier Bousquet, and Ilya Tolstikhin. Predicting neural network accuracy from weights. In International Conference on Learning Representations, 2020.
[30] Sanjeev Arora, Rong Ge, Behnam Neyshabur, and Yi Zhang. Stronger generalization bounds for deep nets via compression approach. arXiv preprint Xrvi: 802.05296, 2018.
[31] Ciprian A Corneanu, Sergio Escalera, and Aleix M Martinez. Computing the testing error without a testing set. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2677-2685, 2020. 2
[32] Yiding Jiang, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Predicting the generalization gap in deep networks with margin distributions. In International Conference Learning Representations, 2019.
[33] Behnam Neyshabur, Srinadh Bhojanapalli, David McAllester, and Nati Srebro. Exploring generalization in deep learning. In Advances in neural information processing systems, pages 5947-5956, 2017. 2
[34] Yiding Jiang, Behnam Neyshabur, Hossein Mobahi, Dilip Krishnan, and Samy Bengio. Fantastic generalization measures and where to find them. In International Conference on Learning Representations, 2019.
[35] Saurabh Garg, Sivaraman Balakrishnan, Zaco Kolter, and Zachary Lipton. Ratt: data to guarantee generalization. In International Conference on Machine Learning, pages 3598-3609, 2021.
Leveraging unlaheled
[36] Yiding Jiang, Vaishnavh Nagarajan, Christina Back, and J Zico Kolter. Assessing generalization of sgd via disagrcement. arXiv preprint arXiv:2106.13780 2021. 2
[37] Sumukh Aithal, Dhruva Kashyap, and Natarajan Subramanyam. Robustness to augmentations as a generalization metrie. arXiv preprint arXiv:2101 06459, 2021. 2
[38] Weijian Deng and Liang Zheng. Are labels always necessary for classifier accuracy evaluation? In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 15069- 15078, 2021. 2
[39] Devin Guillory, Vaishaal Shankar, Sayna Ebrahimi, Trevor Darrell, and Ludwig Schmidt. Predicting with confidence on unseen distributions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1134-1144 1144, 2021. 2
11

---

[40] Saurabh Garg, Sivaraman Balakrishnan, Zachary C Lipton, Behnam Neyshabur, and Hanie Sedghi. Leveraging unlabeled data predict nuttof-distribution performance. In International Conference on Learning Representations, 2022. 2
[41] Christina Back, Yiding Jiang. Aditi Raghunathan, and Zico Kolter. Agreement-onntth-inge Predicting the performance of neural networks under distribution shift. arXiv preprint arXiv:2206. 2022.
[42] Nathan Ng. Kyunghyun Cho, Neha Hulkund, and Marzych Ghassemi. Predicting out-of-domain general- ization with local manifold smoothness. arXiv preprint arXiv:2207.02099 2022.
[43] Yaodong Yu. Zitong Yang, Alexander Wei, Yi Ma, and Jacob Steinhardt. Predicting -ft-t-iiiititttiinn error with the projection norm. In Advances in Neural Information Processing Systems, 2022. 2
[44] Weijian Deng. Stephen Gould, and Liang Zheng. What does rotation prediction tell about classifier accuracy under varying testing environments? International conference on machine learning, 2021.
[45] Jiefeng Chen, Frederick Liu, Besim Avei, Xi Wu, Yingyu Liang, and Somesh Jha. Detecting errors and estimating accuracy on unlabeled data with self self-training ensembles. Advances Neural Information Processing Systems, 34, 2021. 2
[46] Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint urXiv:1912.02781, 2019. 2, 3,
[47] Sangdoo Yun, Dongyoon Han, Seong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032, 2019. 4
[48] Ekin D Cubuk, Barret Zoph, Dandelion Mane, Vijay Vasudevan, and Quoc V Le. Autoaugment: Learning augmentation policies from data, arXiv preprint arXiv:1805.09500 2018. 2, 8
[49] Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with reduced search space. In Proceedings of the IEEE/CVF Conference Computer
Vision and Puttern Recognition Workshops, pages 702-703, 2020. 4
[50] Terrance DeVries and Graham W Taylor. Improved regularization of convolutional neural networks with cutout. arXiv preprint 13Kiv:1708.04555 2017.
[51] Eric Mintun, Alexander Kirillov, and Saining Xie, On interaction between augmentations and corruptions in natural corruption robustness. In Advances in Neural Information Processing Systems, 2021. 2, 8
[52] Dan Hendrycks, Andy Zou, Mantas Mazeika. Leonard Tang. Dawn Song, and Jacob Steinhardt. Pixmix: Dreamlike pictures comprehensively improve safety measures. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2022. 2,8
[53] Hongyi Zhang, Moustapha Cisse. Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.0944 2017. 2
[54] Yuji Tokozume, Yoshitaka Ushiku, and Tatsuya Harada. Between-elat learning for image classification. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1486-5494, 2018.
[55] Dong Yin, Raphael Gontijo Lopes, Jon Shlens, Ekin Dogus Cubuk, and Justin Gilmer. A fourier perspective on model robustness in computer vision. In Advances in Neural Information Processing Systems, 2019. 2
[56] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06000 2017.
[57] Hadi Salman, Andrew Ilyas, Logan Engstrom, Ashish Kapoor, and Aleksander Madry. Do adversarially robust imagenet models transfer better? In Advances in Neural Information Processing Systems, pages 3533-33 3545, 2020.
[58] Evgenia Rusak, Lukas Schott, Roland S Zimmermann, Julian Bitterwolf, Oliver Bringmann, Matthias Bethge, and Wieland Brendel. A simple way to make neural networks robust against diverse image corruptions. In European Conference on Computer Vision, pages 53-69, 2020. 2
[59] Bent Fuglede and Flemming Topsoe. Jensen-shannon divergence and hilbert space embedding. In International Symposium onInformation Theory, 2004. ISIT 2004, Proceedings. page 31, 2004. 3
12

---

[60] Kihyuk Sohn, David Berthelot, Nicholas Carlini, Zizhao Zhang, Han Zhang, Colin A Raffel, Ekin Dogus Cubuk, Alexey Kurakin, and Chun-Liang Li. Fixmatch: Simplifying seni-supervised learning with consistency and confidence. In Advances Neural Information Processing Systems, pages 596-608, 2020.
[61] David Berthelot, Nicholas Carlini, Ian Goodfellow, Nicolas Papernot, Avital Oliver, and Colin A Raf- fel. Mixmatch: A holistic approach to semi-supervised learning. In Advances in Neural Information Processing Systems, 2019. 3
[62] Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathematical statistics, 22(1))79--8, 1951. 3
[63] Jiachen Sun, Akshay Mehra, Bhavya Kailkhura, Pin- Yu Chen, Dan Hendrycks, Jihun Hamm, and Z Morley Mao. Certified adversarial defenses meet distr--dittributton comuptiins: Benchmarking robustness and simple baselines. arXiv preprint arXiv:2 1t.p2//2..00662 020669 2021. 3
arX:v:21
[64] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
4
[65] Zhuang Liu, Hanzi Mao, Chao- Yuan Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. A convnet for the 2020s. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, 2022. 4
[66] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010. 11929, 2020.
[67] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 10012-10022, 2021. 4
[68] Xiaohan Ding, Chunlong Xia, Xiangyu Zhang, Xiaojie Chu, Jungong Han, and Guiguang Ding. Repmlp: Re-parameterizing convolutions into fully-connected layers for image recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, 2022. 4
[69] Ilya O Tolstikhin, Neil Houlsby, Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Thomas Unterthiner, Jessica Yung, Andreas Steiner, Daniel Keysers, Jakob Uszkoreit, et al. Mlp-mixer: An all-mip architecture for vision. Advances Neural Information Processing Systems, 2021. 4
[70] Priya Goyal, Piotr Dollár, Ross Girshick, Picter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jía, and Kaiming He, Accurate, large minibatch sgd: Training imagenet hour.
arXiv preprint arXiv: 706.02267, 2017. 4
[71] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojua. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2888800 08-222, 2016. 4
[72] Irwan Bello, William Fedus, Xianzhi Du, Ekin Dogus Cubuk, Aravind Srinivas, Tsung- Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. Advances in Neural Information Processing Systems, 34, 2021.
[73] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. International conference on machine learning, pages 6105-6114, 14, 2019.
[74] Mingxing Tan and Quoe Le. Efficientnetv2: Smaller models and faster training. International Conference on Machine Learning, pages 10096-10106, 2021. 4
[75] IZeki Yalniz, Hervé Jégou, Kan Chen, Manohar Paluri, and Dhruv Mahajan. Billion-scale semi-supervised learning for image classification. arXiv preprint arXiv:1905 9055600666. 2019.
[76] Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint arXiv: 1503.0233, 2(7), 2015. 4
[77] Byeongho Heo, Sangdoo Yun, Dongyoon Han, Sanghyuk Chun, Junsuk Choe, and Seong Joon Oh. Rethinking spatial dimensions of vision transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 11936-11945.. 2021. 4
13

---

[78] Ross Wightman.
Pytorch
image
models.
fttpas///gttuub.coo/rvvittt
pytorch- pytorch-image image-modela, modela, 2019. 4
[79] Bart Thomee, David A Shamma. Gerald Friedland, Benjamin Elizalde. Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. The new data and new challenges in multimedia research. arXiv preprint
/OXiv:/50..018/7, 2015.
[80] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming Hc, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pages 181-196, 2018. 4
[81] Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unreasonable effective- ness of data deep learning era. Proceedings of the IEEE international conference on computer vision, pages 843-852, 2017. 4
[82] Qizhe Xie, Minh-Thang
Thang Luong, Eduard Hovy, and Quoc Le. Self Belf-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and puitern recognition, pages 10687-10698, 2020. 4
[83] Mannat Singh, Laura Gustafson, Aaron Adcock, Vinicius de Freitas Reis, Bugra Gedik, Raj Prateek Kosaraju, Dhruv Mahajan, Ross Girshick, Piotr Dollár, and Laurens van der Maaten, Revisiting weakly supervised pre-training of visual perception models. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition, 2022.
[84] Alexander Kolesnikov, Lucas Beyer, Xiaohua Zhai, Joan Puigcerver, Jessica Yung, Sylvain Gelly, and Neil Houlsby. Big transfer (bit): General visual representation learning. In European conference on computer vision, pages 491 507, 2020.
[85] Dan Hendryeks, Kevin Zhao, Steven Basart, Jacob Steinhardt, and Dawn Song. Natural adversarial examples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15262-15221 2021, 4
[86] Haohan Wang, Songwei Ge, Zachary Lipton, and Fric P Xing. Learning robust global representations by penalizing local predictive power. In Advances in Neural Information Processing Systems, pages 10506-10518, 2019. 4
[87] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In Proceedings of the International Conference on Learning Representations, 2019. 4, 8, 9
[88] Jacob Benesty, Jingdong Chen, Yiteng Huang, and Isracl Cohen. Pearson correlation coefficient. In Noise reduction in speech processing, pages 1-4. Springer, 2009. 5
[89] Maurice George Kendall. Rank correlation methods. 1948. 5
[90] Peter Huber. Robust statistics. In International encyclopedia of statistical science, pages 1248-1251. Springer, 2011. 5, 9
[91] Kristof Meding, Luca M Schulze Buschoff, Robert Geirhos, and Felix A Wichmann. Trivial impossible- dichotomous data difficulty masks model differences (on imagenet and beyond). Proceedings of the International Conference on Learning Representations, 2022.
[92] Guy Hacohen, Leshem Choshen, and Daphna Weinshall. Let's agree to agree: Neural networks share classification order on real datasets. In International Conference on Machine Learning, pages 3950-3960, 2020.
[93] Kaiyu Yang. Klint Qinami, Li Fei-Fei, Jia Deng, and Olga Russakovsky, Towards fairer datasets: Filtering and balancing the distribution of the people subtree in the imagenet hierarchy. In Proceedings of the 2020 Conference on Fairness, Accountability and Transparency, pages 547-558, 2020. 7
[94] Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar- 10 classifiers generalize to cifar- 10? arXiv preprint 1rXiv:1806.00000 00450 2018. 7
[95] Antonio Torralba, Rob Fergus, and William T Freeman. 80 million tiny images: A large data set for nouparametric object and scene recognition. IEEE transactions on pattern analysis and machine intelligence. 1918))1998-9700 2008.
[96] Patryk Chrabaszez, 1lya Loshchilov, and Frank Hutter. A downsampled variant of imagenet as an alternative to the cifar datasets. arXiv preprint arXiv: 708898 2017.
14

---

[97] Julius Von Kügelgen, Yash Sharma, Luigi Gresele, Wieland Brendel. Bernhard Schölkopf, Michel Besserve, and Francesco Locatello. Self-supervised learning with data augmentations provably isolates content from style. In Proceedings of the International Conference on Learning Representations. 2021.
[98] Maurice Weiler and Gabriele Cesa. General equeiivrar steerable cans. In Advances in Neural Information Processing Systems, 2019.
[99] Taco S Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. arXiv preprint
arXiv:1801.111 2018.
[100] Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Marvin Zhang, Akshay Balsubramani. Weihua Hu, Michihiro Yasunaga. Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Stavness, Wei Guo, Berton A. Earnshaw, Imran S. Haque, Sara Beery, Jure Leskovec, Anshul Kundaje, Emma Pierson, Sergey Levine, Chelsca Finn, and Percy Liang. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML), 2021. 9
[101] Shaoqing Ren, Kaiming He, Ross Girshick. and Jian Sun. Faster r-cmn: Towards real-time ti-e object detection with region proposal networks. In Advances in neural information processing systems, 2015.
[102] Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic seg- mentation. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 3431 1441-344,2200 2015.
[103] Xiaolong Wang. Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 7794-7803, 2018.
15

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] See Section
(b) Did you describe the limitations of your work? [Yes] Please see Section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] Please see Section 7.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as URL)? [Yes] We clearly illustrated the publicly available datasets and models in Section We also provide details about the calculation of EI in Section 3.
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [N/A] This work does not require any training process. Both datasets and models are fixed following the standard protocol.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [N/A] In our work, the trained models are provided by PyTorch Image Models (timm), the correlation study does not require retraining the model, and the datasets are also commonly used standard benchmarks. Therefore, our results do not have any randomness.
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, cloud provider)? [Yes] We illustrate the computational resources in Supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/reelusing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] Section 4, we cite PyTorch Image Models (timm) where trained models are provided. The datasets we used are also clearly cited and introduced.
(b) Did you mention the license the assets? [Yes] We use publicly released datasets and timm models where the licenses provided.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A] This work does not involve any new assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
16


Reviews:
[{"review_id": "ucMAUxUaK2d", "reviewer": "Reviewer_qWbK", "summary": "This work considers generalization and invariance, two key properties of machine learning models. Specifically, this work conducts a large-scale quantitative correlation study between generalization and invariance on ImageNet and CIFAR setups. To help the correlation study, this work introduces the Effective Invariance Score (EI). It is more suitable for measuring model invariance than other measures such as KL and L2. Two key observations are reported: 1) generalization and invariance exhibit strong linear relationships on both in-distribution and out-of-distribution test sets; 2) when tested on different test sets, the accuracy of the model is linearly related to its invariance ability.", "questions": "The following questions and suggestions are expected to be addressed in the rebuttal.\n\n[1] First, lines 97-105 explain why KL is not used for correlation studies. I understand why KL and other measures are not suitable. However, using \"Drawback of existing invariance measures\" to summarize the paragraph is not very coherent and is a bit confusing.\n\n[2] Also, in semi-supervised learning, other additional techniques such as confidence [48] and sharpness [49] are used when KL [50, 51] or L2 are used for maintaining consistency. These techniques may avoid the problems illustrated in case (c) and case (d) in Figure 1. It can also be mentioned on lines 97-105 to help illustrate why KL and L2 are not suitable for correlation studies at work.\n\n[3] As illustrated in the weaknesses, a correlation study on ObjectNet would be interesting.\n\n[4] In addition, related works (eg, [a,b]) should be included in this work. Section 5.7 should be highlighted (see Weaknesses above).\n", "limitations": "In Section 6, the authors clearly discuss potential limitations and propose possible solutions and potential research directions. Potential negative social impacts are also discussed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "[Strengths]\n[1] Overall, the work is well written. Motivation, focus, experimental setup, and key observations are clearly presented. Related work is well organized.\n\n[2] The proposed Invariance Score (EI) is reasonable and well-discussed. The authors give an intuitive illustration in Figure 1. Furthermore, comparisons with other measures in the Supplementary file help to understand why EI is more suitable for correlation studies.\n\n[3] Existing work mainly gives qualitative hints on the relationship between invariance and generalization. The main contribution of this work is it provides quantitative analysis. Furthermore, observations are presented on both in-distribution and out-of-distribution test sets. This is another contribution compared with the existing works. The results on the ImageNet testbed are comprehensive and systematic.\n\n[4] Another interesting observation is that model invariance and accuracy also exhibit strong linear correlations on various test sets (Section 5.7). The results are reported on ImageNet-C with several models, which are helpful.\n\n[5] In addition to the two main observations, this work investigates other points, including how the correlation on the hard test set varies (Section 5.3), and why JS is not suitable for correlation studies (Section 5.5 and in Supplementary Material). These points help to understand the relationship between model invariance and generalization.\n\n[Weaknesses]\n[1] This work quantitatively investigates the relationship between model invariance and generalization. This work points out that previous work [11-13, 15, 28] gave some qualitative hints about the positive correlation between them (Lines 23-29, 68-75). However, some recent works such as [a] and [b] are missing. They should be included and discussed.\n\n[a] Zhu Sicheng, Bang An, Huang Furong. \"Understanding the generalization advantage of model invariance from a data perspective.\" Advances in Neural Information Processing Systems 34 (2021): 4328-4341.\n[b] Sannei, Akiyoshi, Masaki Imaizumi, and Makoto Kawano. \"Improving generalization bounds for group-invariant/equivariant deep networks through quotient feature spaces.\" Uncertainty in Artificial Intelligence. PMLR, 2021.\n\n[2] Continuing the above point, existing works mainly focus on the in-distribution generalization. In contrast, this work investigates both in- and out-of-distribution test sets. Differences or new contributions should be further highlighted in Section 2. The current version (lines 68-75) is not very clear.\n\n[3] This work conducts correlation studies on the ImageNet testbed, which includes several test sets. These test sets are diverse and come from different distributions (Lines 154-157). It is good and convincing. Another test set is ObjectNet, where object backgrounds, rotations, and viewpoints are diverse. It would be better if this work could include this test set in the correlation study.\n\n[4] Section 5.7 should be highlighted as it presents another key observation. However, it is not heightened in the current version. In addition, the results in the Supplementary Materials (Figure A-3) should be included in the main paper, which helps illustrate the observations.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "X5qdsAAacNE", "reviewer": "Reviewer_G2Vw", "summary": "This work proposes a very simple to implement, yet effective, metric (effective invariance or EI) to assess the invariance of a model with respect to some input transformation. The main novelty of the proposed method is that it does not rely on the true label, but rather on the agreement between the predictions given an image and its transformed version.\n\nThe main findings of the paper are about the role of invariance in multi-class classification problems. The authors find that invariance (as measured by EI) is strongly correlated with accuracy for a large set of models and datasets. Moreover, it is empirically proven that EI is much better than Jensen-Shannon Divergence at capturing invariance for OOD datasets, showing that EI is a more robust metric for invariance.\n\nAn additional (and important) observation included in the paper is that training with additional data is more effective (in terms of invariance) than any of the augmentation strategies tried. \n\nThe experimental setup is very solid, with a large breadth of models and datasets being evaluated. Moreover, the paper is well written and the language used is clear and concise.", "questions": "* DNNs are known to be very sensitive to specific translations [1] even if they are \"hard-wired\" to be translation invariant. This sensitiveness is much stronger for other transformations for which the DNN has learnt the invariance. Could you comment on the use of three 90 degree rotations per sample to compute EI. Could this be missing many cases where the model is invariant (or not)? I believe EI could be used to \"map\" the invariance of a CNN in the full transformation domain, which could be extremely important to guide machine learning practitioners to improve their models.\n\n* I would suggest including randomly initialized networks (not trained) in the study. These networks could also be invariant, but do they follow the same trend shown in the plots? Or training till convergence is a condition for EI to work? The study of randomly initialized networks could also shed some light on the invariance induced by the architectural design alone.\n\n* What is the correlation between EI(grayscale) and EI(rotation). Intuitively, they are strongly correlated; if so, have the authors thought about understanding which of the two is the most important source of invariance for generalization?\n\n\n[1] Biscione and Bowers, Convolutional Neural Networks Are Not Invariant to Translation, but They Can Learn to Be, JMLR 2021.", "limitations": "I believe the authors have addressed the limitations of their work correctly, pointing out:\n* The scope of the paper is on multi-class classification. The study (and maybe new formulations of EI) could be extended to regression, detection, etc.\n* The inherent invariances in some architectures (eg. hard-wired rotation invariance), which are not included in this study.\n* The impact of *geographic shift*, not included in this work.\n\nAdditionally, the authors comment on the societal impact, which is limited since this work uses already pre-trained models. The major impact is inherited from the impact of each of these models themselves.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n\n* The proposed measure for invariance is extremely simple to implement, and well backed up by examples and experimental results.\n* The experimental setup is very solid, with a large amount of state of the art models being analyzed. The distinction between fully-supervised, semi-supervised and models trained with more data is smart, and allowed the authors to reach some conclusions in terms of the use of more data in training wrt. augmentations.\nThe comparison with other measures for invariance in Fig. 6, Table 1 and Appendix A is very interesting, and also clearly favorable for EI.\n* The paper's language is clear, I enjoyed the reading. The experimental setup is very well described, with links to all models used and datasets. I believe one of the major strengths of this paper is the experimental setup itself.\n\n**Weaknesses:**\n\n* Although the metric proposed is evaluated using 3 rotations of 90 degrees per image, I believe the method could benefit from using multiple transformed versions of $x$ (eg. randomly sampled rotations), and then measuring the agreement similarly to Eq. (1). However, I believe the proposed 3-way rotation is enough to prove the validity of EI in this work.\n\n* The EI proposed in Eq. (1) is always 0 when there is a disagreement between the prediction of the unaugmented $x$ and the augmented $x^\\prime$. It is reasonable to discard all cases where both predictions disagree, but many times they might mildly disagree, even within the expected standard deviation. I think that the \"otherwise\" case in Eq. (1) could be further refined accounting for those cases where it is not clear whether the model is invariant or not.\n\n* All the OOD datasets analyzed are modifications of Imagenet, which was used for training. I really appreciated the breadth of OOD datasets tested, but I missed at least one dataset that is strongly OOD, for example train on Imagenet and finetune for a dataset that is completely unrelated to Imagenet. I expect EI to perform well in those cases too, so please take this as a suggestion to further improve the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GG1KR1m9ylv", "reviewer": "Reviewer_NZba", "summary": "The authors design an empirical measure of model invariance based on prediction consistency and invariance. With this measure they evaluate the correlation between model invariance (to rotation and grayscale) and generalization on several in-distribution and out-of-distribution ImageNet variants. They show that the model’s accuracy and invariance are linearly correlated, and that such correlation is better captured by their designed measure than other existing measures.", "questions": "Could you also evaluate the Spearman’s rank correlation for JS-divergence on ImageNet-S and ImageNet-A? This could make the comparison fairer.", "limitations": "Yes, the authors addressed the limitations and potential negative societal impact of their work.", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\n1. The linear relationship between model generalization and the invariance captured by the introduced measure is interesting, especially considering that the measure is handcrafted to have a specific form. \n\n2. The evaluation covers the most recent model architectures, including several vision transformers.\n\n### Weaknesses:\n1. Current evaluation datasets are not comprehensive enough to draw convincing conclusions about the introduced measure. Most of the results are based on ImageNet variants, whereas for CIFAR-10 the authors do not do any comparison of measures. For such a handcrafted measure, readers may need similar observations on more diverse datasets, including some existing OOD benchmarks.\n\n2. To make fair comparisons among different invariance measures, Spearman’s rank correlation is arguably a better choice than Pearson’s correlation, since different measures have different algebraic forms so a simple linear correlation may be too restrictive. I’m glad to see the authors use both correlation measures for evaluation. Nevertheless, some results are missing, including Spearman’s rank correlation for JS-divergence on ImageNet-S and ImageNet-A. Also, if we focus on Spearman’s rank correlation, the introduced measure is no longer superior in predicting generalization among other measures like the prediction consistency.\n\n3. The motivation for designing the EI measure is subjective — why does a model making confident mistakes have the same invariance as a model making unconfident mistakes (Figure 1 (c) and (d))? If we look at the invariance from the loss perspective, the former model usually incurs a higher consistency loss than the latter one, hence less invariant. I recommend the authors to elaborate more on this motivation.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "CVnvpNtK0an", "reviewer": "Reviewer_h3Wy", "summary": "The paper starts with three claims in existing works that motivate their work---(1) the relationship between generalization and invariance are largely qualitative in existing work, (2) they are restricted to in-distribution datasets, and that (3) existing quantitative methodologies to quantify invariance do not take into account confidence of predictions. To address these, the authors propose a metric for invariance (EI), which uses both consistency and confidence of model predictions. Then, they conduct an evaluation of how generalization and invariance are correlated across a variety of architectures.", "questions": "N/A", "limitations": "Yes.", "rating": 2, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The problem is extremely important. Understanding the mechanisms driving generalization is a fundamental problem in ML. Invariance has been proposed as one potential mechanism generalization could be achieved. And so, studying their relationship in deep models is important.\n\n2. Scale of the experiments conducted is large.\n\nWeaknesses:\n\n1. Severely incorrect claims: Unfortunately, there are serious flaws at in the claims made in this work. Below I list them with reasoning/citations:\n\na) Existing works are qualitative: There is a very long list of existing works which study this relationship quantitatively, and have reported it before [1-8]\nb) Existing works are in-distribution: In [7,8,2] authors have studied specifically the problem of out-of-distribution and invariance. In fact, [2] presents new, complex datasets to study specifically this particular problem.\nc) Not taking into account confidence: Invariance has been long measured in neuroscience with metrics that take into account confidence. Adaptations of this metric can be seen in [1,2,3,4]. Theoretical investigations [7,8] have also presented much more articulate representations of invariance measurement.\nd) No strong quantitative correlation shown on OOD data: Both [1,2] show quantitatively that generalization positively correlates with invariance. This has also been discussed at length in previous works including [7,8].\n \n2. Completely missing a large portion of literature on invariance and generalization: There is a large body of literature on invariance and generalization (beyond the ones cited below), which this paper has completely ignored. It is not placed in the context of these works, and that may be partly why the paper has made incorrect claims due to being unaware of existing works. \n\n3. No novel contribution: The formulation of EI is very similar to those proposed in [1], and the specialization score presented in [2]. It is what EI adds beyond existing work, as the problems mentioned in this paper as motivations are not true, and have been already addressed by past work which the paper has not referred to.\n\nIn summary: This paper seems unaware of a large subfield of existing literature in neuroscience and AI, which has led to incorrect claims about existing works. It is unclear how EI adds beyond existing works. All in all, the paper needs a major rethinking and reevaluation of the contributions in light of a thorough literature review on generalization and invariance.\n\nReferences\n\n1. https://www.nature.com/articles/s41593-018-0310-2\n2. https://www.nature.com/articles/s42256-021-00437-5\n3. https://papers.nips.cc/paper/2009/file/428fca9bc1921c25c5121f9da7815cde-Paper.pdf\n4. https://papers.nips.cc/paper/1997/file/792c7b5aae4a79e78aaeda80516ae2ac-Paper.pdf\n5. https://arxiv.org/pdf/1706.01350.pdf\n6. https://papers.nips.cc/paper/1996/hash/2812e5cf6d8f21d69c91dddeefb792a7-Abstract.html\n7. Poggio, T.A. and Anselmi, F., 2016. Visual cortex and deep networks: learning invariant representations. MIT Press.\n8. https://academic.oup.com/imaiai/article-abstract/5/2/134/2363483?redirectedFrom=fulltext\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept