Test Time Robustification of Deep Models via Adaptation and Augmentation
Authors: Marvin Mengxin Zhang
Sergey Levine
Chelsea Finn
Venue: ICLR 2022 Submitted (2022)
Keywords: distribution shift
test time adaptation
data augmentation

Abstract: While deep neural networks can attain good accuracy on in-distribution test points, many applications require robustness even in the face of unexpected perturbations in the input, changes in the domain, or other sources of distribution shift. We study the problem of test time robustification, i.e., using the test input to improve model robustness. Recent prior works have proposed methods for test time adaptation, however, they each introduce additional assumptions, such as access to multiple test points, that prevent widespread adoption. In this work, we aim to study and devise methods that make no assumptions about the model training process and are broadly applicable at test time. We propose a simple approach that can be used in any test setting where the model is probabilistic and adaptable: when presented with a test example, perform different data augmentations on the data point, and then adapt (all of) the model parameters by minimizing the entropy of the model's average, or marginal, output distribution across the augmentations. Intuitively, this objective encourages the model to make the same prediction across different augmentations, thus enforcing the invariances encoded in these augmentations, while also maintaining confidence in its predictions. In our experiments, we demonstrate that this approach consistently improves robust ResNet and vision transformer models, achieving accuracy gains of 1-8% over standard model evaluation and also generally outperforming prior augmentation and adaptation strategies. We achieve state-of-the-art results for test shifts caused by image corruptions (ImageNet-C), renditions of common objects (ImageNet-R), and, among ResNet-50 models, adversarially chosen natural examples (ImageNet-A).

Under review as a conference paper at ICLR 2022
TEST TIME ROBUSTIFICATION OF DEEP MODELS VIA ADAPTATION AND AUGMENTATION
Anonymous authors
Paper under double-blind review
ABSTRACT
While deep neural networks can attain good accuracy on in-distribution test points, many applications require robustness even in the face unexpected perturbations in the input, changes in the domain, other sources of distribution shift. We study the problem of test time robustification, i.e., using the test input to improve model robustness. Recent prior works have proposed methods for test time adaptation, however, they each introduce additional assumptions, such as access to multiple test points, that prevent widespread adoption. In this work, we aim to study and devise methods that make no assumptions about the model training process and are broadly applicable at test time. We propose simple approach that can be used in any test setting where the model is probabilistic and adaptable: when presented with a test example, perform different data augmentations on the data point, and then adapt (all of) the model parameters by minimizing the entropy of the model's s average, or marginal, output distribution across the augmentations. Intuitively, this objective encourages the model to make the same prediction across different augmentations, thus enforcing the invariances encoded in these augmentations, while also maintaining confidence in its predictions. In experiments, we evaluate two baseline ResNet models, two robust ResNet-50 models, and a robust vision transformer model, and demonstrate that this approach achieves accuracy gains of 1-8% over standard model evaluation and also generally outperforms prior augmentation and adaptation strategies. For the setting in which only one test point is available, we achieve state-of totte--f-the-aae art results on the ImageNet-C. ImageNet-R, and, among ResNet-50 models, ImageNet-A distribution shift benchmarks.
INTRODUCTION
Deep neural network models have achieved excellent performance on many machine learning problems, such as image classification. but are often brittle and susceptible to issues stemming from distribution shift. For example, deep image classifiers may degrade precipitously in accuracy when encountering input perturbations, such noise changes in lighting (Hendrycks & Dietterich) 2019) or domain shifts which occur naturally in real world applications (Koh et al. (021). Therefore, robustification of deep models against these test shifts is an important and active area of study.
Most prior works in this area have focused on techniques for training time robustification, including utilizing larger models and datasets (Orhan 2019), various forms of adversarial training (Sagawal
et al. 2020 Wong et al. 2020), and aggressive data augmentation (Yin et al.] 2019: Hendrycks et al. al. 2021; Hendrycks et al. 2021a). Employing these techniques requires modifying the traming process, which may not be feasible if, e.g., it involves heavy computation or non-public data. Furthermore, these techniques do not rely on any information about the test points that the model must predict on, even though these test points may provide significant information for improving model robustness. Recently, several works have proposed methods for improving accuracy via adaptation after seeing the test data, typically by updating subset of the model's weights (Sun et al. 2020), normalization statistics (Schneider et al. 2020), or both (Wang et al.) 202 Zhang et al., 2021). Though effective handling test shifts, these methods sometimes still require specialized training procedures, and they typically rely on extracting distributional information via batches or even entire sets of test inputs, thus introducing additional assumptions.
2020

---

Under review as a conference paper at ICLR 2022
1
B
single leet Inained model augmentation lunctions
{al...., aM}
min H (po)
H
marginal entropy minimization
manginal outpul detributtion po (y(x
Po(p(x)
(o(y(x)
Figure 1: A schematic of our overall approach. Left: at test time, as detailed in Section 3] we have trained model that outputs probabilistic predictive distribution and has adaptable parameters single test input x, and set of data augmentation functions {a1,.... 4M ). Note that we do not assume access the model training procedure or multiple test inputs for adaptation. We perform different augmentations to and pass these augmented inputs to the model in order to estimate the marginal output distribution averaged over augmentations. Right: rather than using this distribution to make the final prediction, instead perform a gradient update on the model to minimize the entropy this marginal distribution, thus encouraging the model predictions to be invariant across different augmentations while maintaining confident predictions. The final prediction is then made on the original data point, i.e., the predictive distribution in the top right of the schematic.
We are interested in studying and devising methods for improving model robustness that are "plug and play", i.., they can be readily used with wide variety of pretrained models and test settings. Such methods may simply constitute a different, more robust way to perform inference with preexisting models, under virtually the same assumptions as standard test time inference. In this work, we focus on methods for test time robusiness, in which the specific test input may be leveraged in order to improve the model's prediction on that point. Though broad applicability is our primary goal, we also want methods that synergize with other robustification techniques, in order to achieve greater performance than using either set of techniques in isolation. To satisfy both of these desiderata, we devise a novel test time robustness method based on adaptation and augmentation. As illustrated in Figure when presented with test point, we propose adapt the model by augmenting the test point in different ways and ensuring that the model makes the same predictions across these augmentations, data augmentations. We further encourage the model
thus respecting the invariances encoded to make confident predictions, thus arriving at the proposed method: minimize the marginal entropy of the model's 's predictions across the augmented versions of the test point.
We refer to the proposed method marginal entropy minimization with ensembled augmenta- tions (MEME), and this is the primary contribution of our work. MEME makes direct use of pretrained models without any assumptions about their particular training procedure architecture, while requiring only a single test input for adaptation. In Section 4| we demonstrate empirically that MEME consistently improves the performance of ResNet (He et al., 2016) and vision trans- former (Dosoviskkiy et al., (02T) models on several challenging ImageNet distribution shift bench- marks, achieving several new saue-of-thh-eee results for these models in the setting which only one test point is available. In particular, MEME consistently outperforms non adaptive marginal distribution predictions (between 1-10% improvement) on corruption and rendition shifts tested by the ImageNet-C (Hendrycks & Dietterich] 2019) and ImageNet-R (Hendrycks et al. (221a) datasets, respectively- - indicating that adaptation plays crucial role in improving predictive accuracy. MEME encourages both invariance across augmentations and confident predictions, and an ablation study in Section 4hsows that both components are important for maximal performance gains. Also, MEME is, the best of our knowledge, the first adaptation method to improve performance (by 1-4% over standard model evaluation) on ImageNet-A (Hendrycks et al. (2021b), demonstrating that MEME is more broadly applicable wide range of distribution shifts.
2

---

Under review as a conference paper at ICLR 2022
2 RELATED WORK
The general problem of distribution shift has been studied under a number of frameworks (Quiñonero
Candela et al., 2009), including domain adaptation (Shimodaira, 2000; Csurka 2017 Wilson &
Cook] 2020), domain generalization (Blanchard et al. 2011 Muandet et al., 2013; Gulrajani & Lopez- Paz 2021 , and distributionally robust optimization (Ben-Tal al.J 2013] Hu et al. 2018
Sagawa et al. 2020). to name just few. These frameworks typically leverage additional training or test assumptions in order to make the distribution shift problem more tractable. Largely separate from these frameworks, various empirical methods have also been proposed for dealing with shift, such as increasing the model and training dataset size or using heavy training augmentations (Orhan] (219)] Yin et al. 2019; Hendrycks et al.) 2021a). The focus of this work is complementary to these efforts: the proposed MEME method is applicable to a wide range of pretrained models, including those trained via robustness methods, and can achieve further performance gains via test time adaptation.
Prior test time adaptation methods generally either make significant training or test time assumptions. Some methods update the model using batches even entire datasets of test inputs. such as by
computing batch normalization (BN) statistics on the test set (Li et al. 2017 Kaku et al. 2020 Nado
et al. 2020; Schneider et al.) 2020), minimizing the (conditional) entropy of model predictions across a batch of test data (Wang et al. (02TT. The latter approach is closely related to MEME, The differences are that MEME minimizes marginal entropy using single test points and data augmentation and adapts all of the model parameters rather than just those associated with normalization layers, thus not requiring multiple test points or specific model architectures. Other test time adaptation methods can be applied to single test points but require specific training procedures or models (Sun) et al.) 2020; Huang et al. 2020; Schneider et al. 2020). Test time training (TTT) (Sun et al. 2020) requires a specialized model with a rotation prediction head, as well as different procedure for training this model. [Schneider et al. (2020) show that BN adaptation can be effective even with only one test point, and we refer to this approach "single point" BN adaptation. As we discuss in Section 3. MEME synergizes well with single point BN adaptation.
A number of works have noted that varying forms of strong data augmentation on the training set can
improve the resulting model's 's robustness (Yin et al. 2019 Hendrycks et al. 2020 Li et al. 2021 Hendrycks et al., 2021ap. Data augmentations are also sometimes used the test data directly by averaging the model's outputs across augmented copies of the test point (Krizhevsky et al. 2012 [hhrrten & Khoshgoftaar 2019). i.e., predicting according the model's marginal output distribution. This technique, which we refer to as test time augmentation (TTA), has been shown to be useful both for improving model accuracy and calibration (Ashukha et al., 2020) as well as handling distribution shift (Molchanov et al.) 2020). We take this idea one step further by explicitly adapting the model such that its marginal output distribution has low entropy. This extracts an additional learning signal for improving the model, and furthermore, the adapted model can then make its final prediction on the clean test point rather than the augmented copies. We empirically show in Section 4 that these differences lead to improved performance over this non adaptive TTA baseline.
3 ROBUSTNESS VIA ADAPTATION AND AUGMENTATION
Data augmentations are typically used to train the model to respect certain invariances e.g.. changes in lighting or viewpoint do not change the underlying class label - but, especially when faced with distribution shift, the model is not guaranteed to obey the same invariances at test time. In this section, we introduce MEME, a method for test time robustness that adapts the model such that it respects these invariances on the test input. We use "test time robustness" specifically to refer to techniques that operate directly on pretrained models and single test inputs single point BN adaptation and TTA, as described in /Section 2 are examples of prior test time robustness methods.
In the test time robustness setting, we are given a trained model So X - y with parameters 0 E O. We do not require any special training procedure and do not make any assumptions about the model, except that 0 is adaptable and that fe produces a conditional output distribution pe (ylx) that is differentiable with respect to 0 All standard deep neural network models satisfy these assumptions. A single point E X' is presented to fo, for which it must predict a label ÿ E ) immediately. Note
'Single point BN adaptation also assumes that the model has batch normalization layers, and, shown empirically in (Secttion 4 this assumption that do not require but can also henefit from.
3

---

Under review as a conference paper at ICLR 2022
Algorithm 1 Test time robustness via MEME
Require: trained model fo, test point x, # augmentations B, learning rate n, update rule G i.d. 1: Sample A1,..., ag ~ U(A) and produce augmented points x; = ax(x) for E {l,..., B}
2: Compute Monte Carlo estimate p Σ x.)) = ()(((x) and = Hÿ) f(0;x)
3: Adapt model parameters via update rule 0' + &,
4: Predict i å arg maxy Per (y|x)
that this is precisely identical to the standard test time inference procedure for regular supervised learning models in effect, are simply modifying how inference is done, without any additional assumptions on the training process or on test time data availability. This makes test time robustness methods a simple "slot-in" replacement for the ubiquitous and standard test time inference process. We assume sampling access a set of augmentation functions A _ {aI..... aM} that can be applied to the test point x. We use these augmentations and the self-supervised objective detailed below to adapt the model before it predicts on x. When given set of test inputs, the model adapts and predicts on each test point independently. We do not assume access to any ground truth labels.
3.1 MARGINAL ENTROPY MINIMIZATION WITH ENSEMBLED AUGMENTATIONS
Given a test point x and set of augmentation functions A. we sample augmentations from and apply them to x in order to produce batch of augmented data _,...., &g. The model's 's average, or marginal, output distribution with respect to the augmented points given by
B
Po (o(y) x) A Ee(A)) [po(yla(x))] R
1
Σ po(u)))),
B
where the expectation is with respect to uniformly sampled augmentations a ~ U(A).
What properties do we desire from this marginal distribution? To answer this question, consider the role that data augmentation typically serves during training. For each training point (x (xtrain y the model fe is trained using multiple augmented forms of the input xetrain ..... x is trained to train
obey the invariances between the augmentations and the label should predict the same label train and it should do so confidently. We seek to devise a similar
matter the augmentation
learning signal during test time, when ground truth labels are available. That is, after adapting:
(1) the model fo predictions should be invariant across augmented versions of the test point, and (2) the model Se should be confident in its predictions, even for heavily augmented versions of the test point, due to the additional knowledge that all versions have the same underlying label.
Optimizing the model for more confident predictions can be justified from the assumption that the true underlying decision boundaries between classes lie in low density regions of the data space (Grandvalet & Bengio 2005). With these two goals in mind, we propose to adapt the model using the entropy of its marginal output distribution over augmentations (Equation Ip. i.c.,
<(0x))= H (po(-Ix)) = - Σ po (v(x) log (x)
Optimizing this objective encourages both confidence and invariance augmentations, since the entropy of Po(- is minimized when the model outputs the same (confident) prediction regardless of the augmentation. Given that is adaptable and pe po(y/x) is differentiable with respect to 8, we can directly use gradient based optimization to adapt 0 according to this objective. We use only one gradient step per test point, because empirically we found this be sufficient for improved performance while being more computationally efficient. After this step, can use the adapted model, which we denote før, to predict on the original test input x.
Algorithm 1 presents the overall method MEME for test time adaptation. Though prior test time adaptation methods must carefully choose which parameters to adapt in order to avoid degenerate solutions (Wang al.) (2021), adaptation procedure simply adapts all of the model's parameters 0 (line 3). Note that, as discussed above, the model fo adapts using augmented data but makes its final prediction the original point (line 4), which may be easier to predict on.
4

---

Under review as a conference paper at ICLR 2022
IN-C befora MEMO
"atingrmy"" 152 913
"diehrag" ( :2%5
"goose" 13.29%g
"stingray" (26.675)
"goldtien" Eark
"etingry"" (37 Borsl
meetingmyy (80-80% 800
"dugong"" (3 52%
"stingray"" (88 40%
"dlggeong 12 (0.805))
"window" 1060%
"stingray"" (99.00%)
"slligrry"" 224 CPN
"wock"" 15 2020
"alimgnry" 200
IN-C after MEMO
"alliggry""
"stingray"" (4%)
ray"
"sslinggaa ooy" 47%))
"stlngray"
"stlngray" (94.7216)
"dugong" 5404
(0%)
"stingray" (04.701
"sllingra"" (10 38%)
"bothbub" 12.40%
"stingra"" B2,2010
"stingray"
"stingray" (85 TVK1
"stingryy"
"Stingaay" 386 33.033
(02 10%
150 46%)
Figure We visualize augmentations of randomly chosen data point from the "Gaussian Noise level 3" ImageNet-C test set. Even for robust model trained with heavy data augmentations (Hendrycks) al. 2021a), both its predictive accuracy confidence drop sharply when encountering test distribution shift. As shown in the bottom two rows, these drops can be remedied via MEME.
3.2 COMPOSING MEME WITH PRIOR METHODS
An additional benefit of MEME is that synergizes with other approaches for handling distribution shift. In particular, MEME can be composed with prior methods for training robust models and adapting model statistics, thus leveraging the performance improvements of each technique.
Pretrained robust models. Since MEME makes no assumptions about, or modifications to, the model training procedure, performing adaptation on top of pretrained robust models, such as those trained with heavy data augmentations, is as simple as using any other pretrained model. Crucially, we find that, in practice, the set augmentations that we use at test time does not have to match the augmentations that were used to train the model. This is important properties from the test time augmentations: that they can easily sampled and are applied directly to the model input x. These properties do not hold for. e.g., data augmentation techniques based on image translation models, such as DeepAugment (Hendrycks et al.) (02Ia), or feature mixing, still models trained with these data augmentation techniques as our starting point for adaptation, thus allowing us to improve upon their state-of -fttr--rttttt results. As noted above, using pretrained models is not as easily accomplished for adaptation methods which require complicated or specialized training procedures and model architectures, such TTT (Sun et al., 2020)) ARM (Zhang al. 202T). In our experiments, use AugMix as our set of augmentations (Hendrycks et al. 2020). as it satisfies the above properties and still yields significant diversity when applied, as depicted Figure
require a few
such moment exchange (Li al. (021). However,
Adapting BN statistics. SSchnider et al. (2020) showed that, even when presented with just single test point, partially adapting the estimated mean and variance of the activations in each batch normalization (BN) layer of the model can still be effective in some cases for handling distribution shift, this setting, to prevent overfitting to the test point, the channelwise mean and variance [Meat 2o estimated from this point are mixed with the the mean and variance Murain o² computed during training according to prior strength N, i.c.,
N Murain
1
N
14
N Pest
N+1
iain
N 1 P
This technique is also straightforward to combine with MEME: we simply use the adapted BN statistics whenever computing the model's output distribution. That is. adapt the BN statistics alongside all the model parameters for MEME. Following the suggestion in Schneider et al. (2020). set N 16 for all of experiments in the next section.

---

Under review as a conference paper at ICLR 2022
4 EXPERIMENTS
Our experiments aim to answer the following questions:
(1) How does MEME compare to prior methods for test time adaptation, which make additional training and test assumptions, and test time robustness?
(2) Can MEME be successfully combined with wide range of pretrained models?
(3) Which aspects of MEME are the most important for strong performance?
We conduct experiments on two distribution shift benchmarks for CIFAR- 10 (Krizhessky 2009) and three distribution shift benchmarks for ImageNet (Russakovsky et al. 2015). Specifically, ly, for CIFAR-10, we evaluate on the CIFAR-10-C (Hendrycks & Dietterich 2019) and CIFAR-10.1 (Recht et all.. 2018 test sets, and for ImageNet, we evaluate on the ImageNet-C (Hendrycks & Dietterich 2019 ImageNet-R (Hendrycks et al. 2021a), and ImageNet-A (Hendrycks et al. 2021b) test sets.
To answer question (I), we compare to test time training (TTT) (Sun et al., 2020) in the CIFAR- 10 experiments, for which we train ResNet-26 models following their protocol and specialized architecture. We do not compare to TTT for the ImageNet experiments due to the computational demands of training tfae-oofthhe-art models and because (Sun al. (2020) do not report competitive ImageNet results. For the ImageNet experiments, we compare to Tent (Wang et al.) 2021) and BN adaptation, which can be used with pretrained models but require multiple test inputs (or even the entire test set) for adaptation. We provide BN adaptation with 256 test inputs at a time and, following Schneider et al. (2020), set the prior strength N 256 accordingly.
For Tent, we use test batch sizes of 64 and, for ResNet-50 models, test both "online" adaptation where the model adapts continually through the entire evaluation and "episodic" adaptation where the model is reset after each test batch. (Wang et al., (021). Note that the evaluation protocols are different for these two methods: whereas MEME tasked with predicting on each test point immediately after adaptation, BN adaptation predicts on a batch of 256 test points after computing BN statistics on the batch, and Tent predicts on a batch of 64 inputs after adaptation but also, in the online setting, continually adapts throughout evaluation. In all experiments, we further compare to single point BN adaptation (Schneider et al. 2020) and the TTA baseline that simply predicts according to the model's marginal output distribution over augmentations pe (ylx) [Equation 1) (Krizhevsky et al. 2012 Ashukha et al., 2020). Full details on our experimental protocol are provided in ||pppeddxx A
To answer question (2), we apply MEME top of multiple pretrained models with different architectures, trained via several different procedures. For CIFAR-10, we train our own ResNet- 26 He al. al. (016) models. For ImageNet, we the best performing ResNet-50 robust models from prior work, which includes those trained with Deepeugment and AugMix augmentations (Hendrycks) al., 2021a) well as those trained with moment exchange and CutMix (Li al., 2021). To evaluate the generality of prior test time robustness methods and MEME, we also evaluate the small robust vision transformer (RVT*-small) which provides superior performance on all three ImageNet distribution shift benchmarks compared to the robust ResNet-50 models (Mao et al. 2021).
first determine the relative im- portance of maximizing confidence (via entropy minimization) versus enforcing invariant predictions across augmented copies of each test point, and second to determine the importance of the particular augmentation functions used, The comparison to the non adaptive TTA baseline also helps determine whether simply augmenting the test point is sufficient or if adaptation is additionally helpful.
Finally, to answer (3), we conduct ablative studies lubbsecttiion 4.2)
4.1
MAIN RESULTS
We summarize results for CIFAR-10, CIFAR-10.1 and CIFAR-10-C in Table 1 with full CIFAR- C results in Appendix C. We use indentations to indicate composition, e.g., TTT is performed at test time on top of their specialized joint training procedure. Across all corruption types in CIFAR- -O-C. MEME consistently improves test error compared to the baselines, non adaptive TTA, and TTT. MEME also provides a larger performance gain on CIFAR-10.1 compared to TTT. We find that the non adaptive TTA baseline is competitive for these relatively simple test sets, though it is worse than MEME for CIFAR- 10-C. Of these three test sets, CIFAR- 10-C is the only benchmark that explicitly introduces distribution shift, which suggests that adaptation is useful when the test shifts

---

Under review as a conference paper at ICLR 2022
Table 1: Results for the original CIFAR-10 test set, CIFAR-10.1 and CIFAR-10- 10-C.
MEME outper-
forms TTT despite not making any training assumptions. "Results from Sun et al. 2020
CIFAR-10
CIFAR-10.1
CIFAR- 10-R-
Error (%)
Error (%)
Average Error (%)
ResNet-26 (He al. 2016
9.2
18.4
22.5
+ TTA
7.3 (-1.9)
14.8 (-3.6)
19.9 (-2.6)
+ MEME (ours)
7.3 (-1.9)
14.7 3.7)
19.6 2.9)
+ Joint training* Sun ct al. 2020
8.1
16.7
22.8
TTT
(Sun et al. 2020
7.9 (-0.2)
15.9 (-0.8)
21.5 (-1.3)
Table Test results for ImageNet-C, ImageNet-R, and ImageNet-A. MEME achieves new state-of- the-art performance each benchmark for ResNet-50 models for the single test point setting. For RVT MEME substantially improves performance across all benchmarks and reaches a new state of the art for ImageNet-C and ImageNet-R.
ImageNet-C mCE
ImageNet-R ImageNet-A Error (%)
Error (%)
Baseline ResNet-50 (He et al. 2016
76.7
63.9
100.0
TTA
77.9 (+1.2)
61.3 (-2.6)
98.4 (-1.6)
+ Single point BN
71.4 (-5.3)
61.1 (-2.8)
99.4 (-0.6)
MEME (ours)
69.9 (-6.8)
58.8 (-5.1)
99.1 (-0.9)
BN (N 256. n 256)
61.6 (-15.1)
59.7 (-4.2)
99.8 (-0.2)
Tent (online) (Wang et al. 2021
54.4 (-22.3)
57.7 (-6.2)
99.8 (-0.2)
Tent (episodic)
64.7 (-12.0)
61.0 (-2.9)
99.7 (-0.3)
+ DeepAuugmentAAugMiix (Hendrycks et al. 2021a
53.6
53.2
96.1
TTA
55.2 (+1.6)
51.0 (-2.2)
93.5 (-2.6)
Siiglee point BN
51.3 (-2.3)
51.2 (-2.0)
95.4 (-0.7)
+ MEME (ours)
49.8 (-3.8)
49.2 (-4.0)
94.8 (-1.3)
BBN (NV 256, 256)
45.4 (-8.2)
48.8 (-4.4)
96.8 (+0.7)
'Tnt (online)
43.5 (-10.1)
46.9 (-6.3)
96.7 (+0.6)
Tenn (cpisodic)
47.1 (-6.5)
50.1 (-3.1)
96.6 (+0.5)
+ MoEx+CutMix (Li et al. 2021
74.8
64.5
91.9
TTA
75.7 (+0.9)
62.7 (-1.)
89.5 (-2.4)
Single point BN
71.0 (-3.8)
62.6 (-1.9)
91.1 (-0.8)
МЕME (ours)
69.1 (-5.7)
59.4 (-3.3)
89.0 (-2.9)
BN (N = 256, n = 256)
60.9 (-13.)
61.6 (-2.9)
93.9 (+2.0)
Tent (online)
54.0 (-20.8)
58.7 (-5.8)
94.4 (+2.5)
'Tnt (episodic)
66.2 (-8.6)
63.9 (-0.6)
94.7 (+2.8)
RVT"-small (Mao al. 2021
49.4
52.3
73.9
+ TTA
53.0 (+3.6)
49.0 (-3.3)
68.9 (-5.0)
Single point BN
48.0 (-1.4)
51.1 (-1.2)
74.4 (+0.5)
MEME (ours)
40.6 (-88)
43.8 (-8.5)
69.8 (-4.)
BN (N = 256, n = 256)
44.3 (-5.1)
51.0 (-1.3)
78.3 (+4.4)
Tent (online)
46.8 (-2.6)
50.7 (-1.6)
82.1 (+8.2)
+ Tent (adapt all)
44.7 (-4.7)
74.1 (+21.8)
81.1 (+7.2)
more prominent. Both TTA and MEME also effective at improving performance for the original CIFAR-10 test set where there is no distribution shift, providing further support for the widespread of augmentations in standard evaluation protocols (Krizhevsky al. 2012 Ashukha et al. 2020).
We summarize results for ImageNet-C. C ImageNet-R, and ImageNet-A in Table 2
with complete ImageNet-C results in Appendix C) We again use indentations to indicate composition, e.g., the best results on ImageNet-C for our setting are attained through a combination starting from a model trained with DeepAugment and AugMix (Hendryeks al. 2021a) and using MEME
7

---

Under review as a conference paper at ICLR 2022
Table 3: Ablating the adaptation objective test pairwise cross entropy and conditional entropy (CE)
based adaptation. MEME generally performs the best, indicating that both encouraging invariance across augmentations and confidence are helpful in adapting the model.
CIFAR-10 Error (%)
CIFAR-10.1 10.1 Error (%)
CIFAR-IO-C Average Error (%)
ResNet-26 (He et al. 2016
9.2 7.3 (-1.9)
18.4
22.5
+ MEME (ours)
14.7 (-8.7)
19.6 (-2.9)
- Egquatinn 2 EpCE
7.6 (-1.6)
15.3 (-3.)
20.0 (-2.6)
Equaiion 2
fCE
7.6 (-1.6)
14.7 (-3.7)
20.0 (-2.5)
ImageNet-C
ImageNet-R Error (%)
ImageNet-A Error (%)
mCE 4
RVT*-smal (Mao et al. 2021
49.4
52.3
73.9
MEME (ours)
40.6 ( 8.8)
43.8
8.8)
69.8 (-4.1)
[qqutiin 2 ECr
11.2( 8.2)
44.2 ( 8.1)
69.7 4.2)
top. For both ImageNet-C and ImageNet-R, and for both the ResNet-50 and RVT -small models, combining MEME with robust training techniques leads to new state of the art performance among methods that observe only one test point at a time. We highlight in gray the methods that require multiple test points for adaptation, and we list in bold the best results from these methods which outperform the test time robustness methods. As AsTTale 2 and prior work both show (Schneider et al., 2020; Wang al. (021), accessing multiple test points can be powerful for benchmarks such as ImageNet-C and ImageNet-R, in which inferred statistics from the test input distribution may aid in prediction. In the case of Tent, which adapts online, the model has adapted using the entire test set by the end of evaluation. However, these methods do not help, and oftentimes even hurt, for ImageNet-A. Furthermore, we find that these methods are less effective with the RYT* -mall model, which may indicate their sensitivity to model architecture choices. Therefore, for this model, we also test modification of Tent which adapts all parameters, and we find that this version of Tent works better for ImageNet-C but is significantly worse for ImageNet-R.
MEME results in substantial improvement for ImageNet-A and is competitive with TTA on this problem. No prior test time adaptation methods have reported improvements on ImageNet-A, and some have reported explicit negative results (Schneider et al., 2020). As discussed, it is reasonable for adaptation methods that rely multiple test points to achieve greater success on other benchmarks such as ImageNet-C, in which a batch of inputs provides significant information about the specific corruption that must be dealt with. In contrast, ImageNet-A does not have such obvious characteristics associated with the input distribution, as it is simply a collection of images that are difficult to classify, As MEME instead extracts learning signal from single test points, it is, the best of knowledge, the first test time adaptation method to report successful results on this testbed. When used on top of a model trained with moment exchange and CutMix (Li et al. 202I). MEME achieves state- of-the-art performance among ResNet-50 models and single test point methods. TTA generally offers larger performance gains on ImageNet-A and also results in the highest overall accuracy when combined with the RVT* -small model; however, TTA performs worse than MEME on ImageNet-R and consistently decreases accuracy on ImageNet-C compared to standard evaluation. We view the consistency with which MEME outperforms the best prior methods, which change across different test sets, to be a major advantage of the proposed method.
4.2 ABLATIVE STUDY
MEME increases model robustness at test time via adaptation and augmentation. In this section, ablate the adaptation procedure, and in Appenddi B we ablate the use and choice of augmentations.
From the results above, we conclude that adaptation generally provides additional benefits beyond simply using TTA to predict via the marginal output distribution pe (y) x)) However, we can disentan- gle two distinct self-super learning signals that may be effective for adaptation: encouraging invariant predictions across different augmentations of the test point, and encouraging confidence via entropy minimization. The marginal entropy objective in Eqqation enncapsulaess both of these
8

---

Under review as a conference paper at ICLR 2022
learning signals, but it cannot easily be decomposed into these pieces. Thus, we instead use two ablative adaptation methods that each only make use of one of these learning signals.
First, consider optimizing the pairwise cross entropy between each pair of augmented points, i.e.,
(0; &
Σ i= j
Po(y)ı;) log (v)
B B
Where x; again refers to the i-th sampled augmentation applied to x. Intuitively, this loss function encourages the model to adapt such that it produces the same predictive distribution for all aug- mentations of the test point, but it does not encourage the model to produce confident predictions. Conversely, as objective that encourages confidence but not invariance, we also consider optimizing conditional entropy on the batch of augmented points, i.e.,
B
1
(0:(0: &
Σ (ppo(112i) B
This ablation is effectively a version of the episodic variant of Tent (Wang al., (021)) that produces augmented copies of single test point rather than assuming access to test batch. We first evaluate these ablations on the CIFAR-10 test sets. We use same adaptation procedure outlined in Algorithm I] with e replaced with the above objectives, and we keep the same hyperparameter values.
The results are presented in Table 3)
We see that MEME, i.e., marginal entropy minimization, generally performs better than adaptation with either of the alternative objectives. This supports the hypothesis that both invariance across augmentations and confidence are important learning poor performance
signals for self-supervised adaptation. When faced with CIFAR- 10.1, from the pairwise cross entropy based adaptation method. On the original CIFAR- 10 test set and CIFAR-I0-C, the ablations perform nearly identically and uniformly worse than MEME. To further test the fcE ablation, we also evaluate it on the ImageNet test sets for the RVT*-small model. We find that, similarly, minimizing conditional entropy generally improves performance compared the baseline evaluation. MEME is more performant for ImageNet-C and ImageNet-R, again indicating the benefits of encouraging invariance to augmentations, Adaptation via fcE performs slightly better for ImageNet-A, though for this problem, TTA is still the best method.
5 DISCUSSION
We presented MEME, a method for test time robustification again distribution shift via adaptation and augmentation. MEME does not require access or changes to the model training procedure and is thus broadly applicable for a wide range pretrained models. Furthermore, MEME adapts at test time using single test inputs, thus it does not assume access multiple test points in several recent methods for test time adaptation (Schneider et al. 2020 Wang et al.) 2021 Zhang et al. 2021). On range of distribution shift benchmarks for CIFAR-T0 and ImageNet classification, and for both ResNet and vision transformer models, MEME consistently improves performance at test time and achieves several new state-of-the-an results for these models the single test point setting.
Inference via MEME is more computationally expensive than standard model inference, primarily because adaptation is performed per test point and thus inference cannot be batched. When deployed in the real world, it is natural to expect that test points will arrive one at time and batched inference will be possible. However, MEME is also more computationally expensive due its augmentation and adaptation procedure. One interesting direction for future work is to develop techniques for selectively determining when adapt the model in order to achieve more efficient inference. For example, with well calibrated models (Guo et al.) 2017). we may run simple "feedforward" inference when the prediction confidence is over a certain threshold, thus achieving better efficiency. Additionally, would be interesting to explore MEME in the test setting where the model is allowed to continually adapt as more test data observed. In our preliminary experiments this setting, MEME tended to lead to degenerate solutions, e.g., the model predicting constant label with maximal confidence. and this may potentially be rectified by carefully choosing which parameters to adapt (Wang et al. (02T) regularizing the model such that it does not change too drastically from the pretrained model.

---

Under review as a conference paper at ICLR 2022
6 ETHICS STATEMENT
Distribution shift, in general, lies at the heart of many ethical concerns in machine learning. When machine learning models and research not adhere to ethical principles such as "contribute society and to human well-being", "avoid harm", and "be fair and take action to avoid discrimination", it can oftentimes be attributed at least partially to issues stemming from distribution shift. As such, research into ways combat and mitigate shift contribute to furthering the ethical goals and principles laid out by this conference and the broader community. Related to this work, we may imagine that models that more robust adaptable to shift may produce more fair results when considering underrepresented subpopulations and could be more trustworthy in safety critical applications. However, as mentioned in Section 5] the proposed method is also more computationally intensive, and care must be taken in general to not place the most powerful machine learning tools exclusively in the hands of the most privileged and resource rich individuals and organizations. Beyond this, do not see any immediate ethical concerns regarding the content, presentation, or methodology of this work.
REPRODUCIBILITY STATEMENT
The proposed method is relatively simple and, to the best of our ability, explained fully in [Section and Algorithm I This explanation is bolstered by complete description of all hyperparameters in Appendix A well as the example code provided in the supplementary materials. As briefly explained in the README in the example code, all datasets were downloaded from publicly accessible links and preprocessed only following the standard protocols. Upon publication, we will include link the full (non anonymous) code release containing full instructions for setting up the code, downloading datasets, and reproducing the results.
REFERENCES
A. Ashukha, A. Lyzhov, D. Molchanov, and D. Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling deep learning. International Conference on Learning Representations (ICLR), 2020.
A. Ben-Tal, D. den Hertog, A. De Waegenaere, B. Melenberg, and G. Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 2013.
G. Blanchard, G. Lee, and C. Scott. Generalizing from several related classification tasks to a new unlabeled sample. In Advances in Neural Information Processing Systems (NIPS), 2011.
G. Csurka. Domain adaptation for visual applications: A comprehensive survey. arXiv preprint 173.0:170. 2017.
A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
Y. Grandvalet and Y. Bengio. Semi- supervised learning by entropy minimization. In Advances in Neural Information Processing Systems (NIPS), 2005.
I. Gulrajani and D. Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations (ICLR), 2021.
C. Guo, G. Pleiss, Y. Sun, and K. Weinberger. On calibration of modern neural networks. In International Conference Machine Learning (ICML), 2017.
K. He, X. Zhang, S. Ren, and Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference Learning Representations (ICLR), 2019.
10

---

Under review as a conference paper at ICLR 2022
D. Hendrycks, N. Mu, E. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan, AugMix: A simple data processing method to improve robustness and uncertainty. In International Conference on Learning Representations (ICLR), 2020.
D. Hendrycks, S. Basart, N. Mu, S. Kadavath, F. Wang. E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, D. Song, J. Steinhardt, and Gilmer. The many faces of robustness: A critical analysis of generalization. In IEEE International Conference on Computer Vision (ICCV),
2021a.
D. Hendrycks, K. Zhao, S. Basart, J. Steinhardt, and D. Song. Natural adversarial examples. In Conference Computer Vision and Pattern Recognition (CVPR), 2021b.
W. Hu, G. Niu, I. Sato, and M. Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning (ICML), 2018.
Y. Huang, J. Gomet, S. Dai, Z. Yu, T. Nguyen, D. Tsao, and A. Anandkumar. Neural networks with recurrent generative feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
A. Kaku, S. Mohan, A. Parnandi, H. Schambra, and C. Fernandez-Granda. Be like water: Robustness extraneous variables via adaptive feature normalization. arXiv preprint arXiv:2002.04000 2020.
D. Kingma and J. Ba. Adam: A method for stochastic optimization. In International Conference Learning Representations (ICLR), 2015.
Koh, S. Sagawa, H. Marklund, S. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. Phillips, I. Gao, T. Lee, E. David, I. Stavness, W. Guo, B. Earnshaw, I. Haque, S. Beery, Leskovec, A. Kundaje, F. Pierson, S. Levine, C. Finn, and P. Liang. WILDS: benchmark of in-the-wild distribution shifts. International Conference on Machine Learning (ICML), 2021.
A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.
A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS), 2012.
B. Li, F. Wu, S. Lim, S. Belongie, and K. Weinherger. On feature normalization and data augmentation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Y. Li, N. Wang, Shi, J. Liu, and X. Hou. Revisiting batch normalization for practical domain adaptation. In International Conference Learning Representations Workshop (ICLRW), 2017.
I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference Learning Representations (ICLR), 2019.
D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. van der Maaten. Exploring the limits of weakly supervised pretraining. In European Conference Computer Vision (ECCV), 2018.
X. Mao, G. Qi, Y. Chen, X. Li, R. Duan. S. Ye, Y. He, and H. Xue. Towards robust vision transformer. arXiv preprint arXiv: 10t.029055.0966 2021.
D. Molchanov, A. Lyzhov, Y. Molchanova, A. Ashukha, and D. Vetrov. Greedy policy search: A simple baseline for learnable test-time augmentation. In Conference on Uncertainty in Artificial Intelligence (UAI), 2020.
K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representa- tion. In International Conference on Machine Learning (ICML), 2013.
Z. Nado, S. Padhy, D. Sculley, A. D'Amour, B. Lakshminarayanan, and J. Snock. Evaluat- ing prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963, 2020.
A. Orhan. Robustness properties of Facebook's ResNeXt WSL models. arXiv preprint 1rXiv:1907.076444 2019.
11

---

Under review as a conference paper at ICLR 2022
J. Quiñonero Candela, M. Sugiyama, A. Schwaighofer, and N. Lawrence. Dataset Shift in Machine Learning. The MIT Press, 2009.
B. Recht, R. Roelofs, L. Schmidt, and V. Shankar. Do CIFAR-10 classifiers generalize to CIFAR-10? arXiv preprini arXiv: 1806. 1806.0047,, 2018.
O. Russakovsky, J. Deng, H. Su, Krause, Satheesh, S. Ma, Huang, A. Karpathy, A. Khosla, M. Bernstein, A. Berg, and F. Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 2015.
Sagawa, P. Koh, T. Hashimoto, and P. Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Confer- ence on Learning Representations (ICLR), 2020.
S. Schneider, E. Rusak. L. Eck, O. Bringmann, W. Brendel, and M. Bethge. Improving robustness against common corruptions by covariate shift adaptation. In Advances Neural Information Processing Systems (NeurIPS), 2020.
H. Shimodaira. Improving predictive inference under covariate shift by weighting the log-likelihood function. Journal of Statistical Planning and Inference (JSPI), 2000.
C. Shorten and Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 2019.
Y. Sun, X. Wang, Z. Liu, J. Miller, A. Efros, and M. Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International Conference on Machine Learning (ICML), 2020.
D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations (ICLR), 2021.
G. Wilson and D. Cook. A survey of unsupervised deep domain adaptation. ACM Transactions Intelligent Systems and Technology (TIST), 2020.
E. Wong, L. Rice, and J. Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations (ICLR), 2020.
Y. Wu and K. He. Group normalization. European Conference on Computer Vision (ECCV), 2018
Xie, R. Girshick, P. Dollár, Z. Tu, and K. He. Aggregated residual transformations for deep neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
D. Yin, R. Lopes, J. Shlens, E. Cubuk, and J. Gilmer. A Fourier perspective model robustness in computer vision. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
M. Zhang, H. Marklund, A. Gupta, S. Levine, and C. Finn. Adaptive risk minimization: Learning to adapt to domain shift, In Advances Neural Information Processing Systems (NeurIPS), 2021.
12

---

Under review as a conference paper at ICLR 2022
A EXPERIMENTAL PROTOCOL
We select hyperparameters using the four disjoint validation corruptions provided with CIFAR-10-C and ImageNet-C (Hendryeks& & Dietterich] 2019). As the other benchmarks are only test sets and do not provide validation sets, we use the same hyperparameters found using the corruption validation sets and do not perform any additional tuning. For the ResNet models that we evaluate, we use stochastic gradients as the update rule G; for ResNet-26 models, we set the number of augmentations B 32 and the learning rate 77 = 0.005; and for ResNet-50 models, we set B 64 and 17 0.00025. For the robust vision transformer. we use AdamW W (Loshchilov & Hutter 2019) as the update rule G, with learning rate rj = 0.00001 and weight decay 0.01, and B 64.
In the CIFAR evaluation, we compare to TTT, which, as noted, can also be applied to single test inputs but requires a specialized training procedure (Sun et al. 2020)). Thus, the ResNet-26 model we use for our method closely follows the modifications that Sun et al. (2020) propose, in order to provide fair point comparison. In particular, |Sun et al. (2020) elect to use group normalization (Wu & Hej 2018) rather than BN, thus single point BN adaptation is not applicable for this model architecture. As noted before, TTT also requires the joint training of a separate rotation prediction head, thus further changing the model architecture, while MEME directly adapts the standard pretrained model.
The TTA results are obtained using the same AugMix augmentations as for MEME. point BN adaptation results use N 16, suggested by /Schneider et al. (2020). As noted, the BN adaptation results (using multiple test points) are obtained using N = 256 as the prior strength and batches of 256 test inputs for adaptation. For Tent, we use the hyperaarameters suggested in Wang et al. (202T): we use stochastic gradients with learning rate 0.00025 and momentum 0.9, the adaptation is performed with test batches of 64 inputs, and the method is run online, i.e., prediction and adaptation occur simultancously and the model is allowed to continuously adapt through the entire test epoch. Since Wang et al. (222T) did not experiment with transformer models, we also attempted to run Tent with Adam (Kingma & Ba 2013) and Adam (Loshchilov & Hutter] 2019) and various hy hyprraaraeeeess for the RVT* small model; however, we found that this generally resulted in worse performance than using stochastic gradient updates with the aforementioned hyperparameters.
The single
We obtain the baseline ResNet-50 parameters directly from the torchvis sin library. The parameters for the ResNet-50 trained with DeepAugment and AugMix are obtained from https: //tpiv/ google. com/ wwle/e/////////ee 2t1/1KKmm_pppa The parameters for the ResNet-50 trained with moment exchange and CutMix are obtained from https: google. wwt./////////c/////cc//ccaccac/e cttpa//ccileee///cccccocooopa/a/ jrtp:pp---jjjsttt.aaaa
99aat 9Eab1 QDA. The parameters for the small robust vision transformer -mal"-small) model obtained from
https: htt.::/...........o...ooooooooon
B ADDITIONAL EXPERIMENTS
In this section, we analyze the importance of using augmentations during adaptation, study the trade- offs between efficiency and accuracy for MEME, and present results with ResNext101 models (Xie et al. 2017, Mahajan et al. 2018: Orhan 2019) on ImageNet-A.
B.1 ANALYSIS ON AUGMENTATIONS
One may first wonder: are augmentations needed in the first place? In the test time robustness setting when only one test point is available, how would simple entropy minimization fare? We answer this question in Table by evaluating the episodic variant of Tent (i.e., with model resetting after each batch) with test batch size of 1. This approach is also analogous to a variant of MEME that does not use augmentations, since for one test point and no augmented copies, conditional and marginal entropy are the same. Similar to MEME, we also incorporate single point BN adaptation with N = 16, in place of the standard BN adaptation that Tent typically employs using batches of test inputs. The results in Table 4 indicate that entropy minimization provides no additional performance gains beyond just single point BN adaptation. This empirically shows that using augmentations is important for achieving the reported results.
single test point generally
We also wish to understand the importance of the choice of augmentation functions A. As mentioned, used AugMix (Hendrycks et al., 2020) in the previous experiments it best fit our criteria:
13

---

Under review as a conference paper at ICLR 2022
Table 4: Evaluating the episodic version of Tent with batch size of 1, which corresponds simple entropy minimization approach for the test time robustness setting. This approach also uses single point BN adaptation, and entropy minimization does not provide much, if any, additional gains.
ImageNet-C mCE +
ImageNet-R ImageNet-A Error (%)
Error (%)
Baseline ResNet-50 (He et al. 2016
76.7
63.9
100.0
Single point BN
71.4 (-5.3)
61.1 (-2.8)
99.4 (-0.6)
MEME (ours)
69.9 (-6.8)
58.8 (-5.1)
99.1 (-0.9)
Tent (episodic, batch size 1) Wang et al.
2021
71.4 (-5.3)
61.2 (-2.7)
99.4 (-0.6)
+ DeepAuugmentAAugMixx (Hendrycks al. 2021a
53.6
53.2
96.1
Single point BN
51.3 (-2.3)
51.2 (-2.0)
95.4 (-0.7)
+ MEME (ours)
49.8 (-3.8)
49.2 (-4.0)
94.8 (-1.3)
Tent (episodic, batch size 1) Wang et al.
2021
51.2 (-2.4)
50.9 (-2.3)
95.4 (-0.7)
+ MoEx+CutMix (Li et al. 2021
74.8
64.5
91.9
Single point BN
71.0 (-3.8)
62.6 (-1.9)
91.1 (-0.8)
МЕME (ours)
69.1 (-5.7)
59.4 (-3.3)
89.0 (-2.9)
Tent (episodic, batch size 1)
Wang et al. 2021
70.9 (-3.9)
62.6 (-1.9)
91.1 (-0.8)
RsTalsmall (Mao et al. 2021
49.4
52.3
73.9
Single point BN
48.0 (-1.4)
51.1 (-1.2)
74.4 (+0.5)
MEME (ours)
40.6 (-8.8)
43.8 (-8.5)
69.8 (-4.1)
+ Tent (episodic, batch size 1) (Wang et al. 2021
48.1 (-1.3)
50.7 (-1.6)
74.4 (+0.5)
Table 5: Ablating the augmentation functions to test standard augmentations (random resized cropping and horizontal flips). When changing the augmentations used, the post-adaptation performance generally does not change much, though it suffers the most on CIFAR-10-C.
CIFAR-10
CIFAR-10.1 Eror (%)
CIFAR-10-4 Average Error (%)
Error (%)
ResNet-26 (He et al. 2016
9.2
18.4
22.5
+ MEME (ours)
7.3 (-1.9)
14.7 (-3.7)
19.6 (-2.9)
AugMix (Hendrycks et al. 2020 standard augs 7.2 (-2.0)
14.6 (-3.8)
20.2 (-2.3)
Aug requires only the input x, and randomly sampled augmentations lead to diverse augmented data points. A simple alternative to instead use the "standard" set of augmentations commonly used in ImageNet training, i.c., random resized cropping and random horizontal flipping. We evaluate this ablation of using MEME with standard augmentations also on the CIFAR- 10 test sets, again with the same hyperparameter values. From the results in Table 5] we can see that MEME is still effective with simpler augmentation functions. This is true particularly for the cases where there no test shift, as in the original CIFAR-10 test set, or subtle shifts in CIFAR-10.1: however, for the more severe and systematic CIFAR-10-C shifts, using heavier AugMix data augmentations leads to greater performance gains over the standard augmentations. Furthermore, this ablation conducted using the ResNet-26 model, which was trained with standard augmentations for robust models such as those in Table 2 AugMix may offer greater advantages at test time since these models were exposed to heavy augmentations during training.
B.2 ANALYZING THE TRADEOFF BETWEEN EFFICIENCY AND ACCURACY
In Figure 3] we analyze the test error of MEME adaptation on ImageNet-R as function of the efficiency of adaptation, measured in seconds per evaluation. We achieve various tradeoff's by varying the number of augmented copies B = {1, 2, 4,8, 8, 16, 32, 64, 1283. We note that small values of B such 4 and 8 can already provide significant performance gains, indicating that practical tradeoff between efficiency and accuracy is possible. For large B, the wall clock time is dominated by computing the augmentations in implementation, we do not compute augmentations in
14

---

Under review as a conference paper at ICLR 2022
Baseline Reslect50
smal
DeepAugment AugMix
47.0
MoEx CutNix
62.5
46.5
Eron (19
Error
45.5
33.0
41.5
16
50.0
128
Seconds per evaluction
Seconds per evallsstion
Figure 3: Plotting MEME efficiency as seconds per evaluation axis) and test error on ImageNet-R (y axis) for the ResNet-50 models (left) and RVT*-small (right) while varying B (1,2, 4, 8, 16, 32, 64, 128]. Note the log scale on the x axis.
Table 6: ImageNet-A results for the ResNext- 101 models.
ImageNet-A A Error (%)
Baseline ResNext- 101 (32x8d) (Xie et al. 2017
90.0
TTA
83.2 (-6.8)
Single point BN
88.8 (-1.2)
MEME (ours)
84.3 (-5.7)
WSL on billions of images (Mahajan et al. 2018
54.9
TTA
49.1 (-5.8)
MEME (ours)
43.2 (-11.7)
parallel, though in principle this is possible for AugMix and should improve efficiency overall. These experiments used four Intel Xcon Skylake 6130 CPUs and one NVIDIA TITAN RTX GPU,
B.3 EVALUATING RESNEXTI01 MODELS ON IMAGENET-A
ResNext-101 models (Xie et al. 2017) have been found achieve higher accuracies ImageNet-A (Hendrycks et al., 2021b). particularly when trained with massive scale weakly supervised pretraining (Mahajan et al. 2018 Hendrycks et al., 2021a). In this section, evaluate whether MEME can successfully adapt these models and further improve perfor- mance on this challenging set. We use the same hyperparameters as for the robust vision transformer with no additional tuning: AdamW (Loshchilov & Hutter (201)) as the update rule G. learning rate 7/ 0.00001, weight decay 0.01, and B 64. We obtain the baseline ResNext- 101 (32x8d) parameters, pretrained on ImageNet, directly from the torchviss sion library. We also evaluate ResNext- 101 (32x8d) pretrained with weakly supervised learning (WSL) on billions of Instagram images (Mahajan et al.) 2018). and we obtained the parameters from https: / /owwn oad. pytorch. org/ org/model model s/ ae/llss/iieeeeee resnegeeeeeellleeeeeee 101 1011_3xx-c3330
0e5 pth. For the WSL model, did not use single point BN adaptation we found this technique to be actually harmful to performance, and this corroborates previous findings (Schneider et al. 2020).
Table 6/sumarizes the results. We can see that, similar to Table 2
Both TTA and MEME significantly improve upon the baseline model evaluation. TTA performs best for the baseline ResNext- 101 model. significant margin, as it more successful at adapting the WSL model, which has a much higher accuracy. This suggests that combining MEME with other large pretrained models may be an interesting direction for future work.
However, MEME ultimately achieves the best accuracy
15

---

Under review as a conference paper at ICLR 2022
C FULL CIFAR-10-C AND IMAGENET-C RESULTS
In the following tables, we present test results broken down by corruption and level for CIFAR-1O-C for the methods evaluated in Table I] We omit joint training and TTT because these results available from Sun et al. (020). Our test results for ImageNet-C are provided in the CSV files in the supplementary material.
Table 7: Test error (%) on CIFAR-IO-C level corruptions.
gauss
shot
impul
defoc
glass
frost
fog
brit
contr clast
pixel
ipeg
ResNet-26
48.4
44.8
50.3
24.1
47.7
24.5
24.1
24.1
33.1
28.0
14.1
29.7
25.6
43.7
28.3
+TA
43.4
39.6
42.9
28.3
44.7
26.3
26.3
21.4
28.5
23.3
12.1
32.9
21.7
43.2
21.7
MEME (ours)
43.3
26.4
44.4
25.1
25.0
20.9
28.3
22.8
11.9
28.3
21.1
42.8
21.7
Table 8:
Test error (%) on CIFAR-10-C level corruptions.
gauss
shot
impul
defoc
glass
motn
frost
fog
brit
contr elast pixel
jpeg
ResNet-22
37.2
39.3
14.8
48.0
19.9
18.7
22.0
24.9
15.1
11.4
16.8
19.1
27.9
24.9
+TA
45.0
20.9
20.2
12.9
17.7
25.7
MEME (ours)
32.2
45.0
20.0
19.2
18.7
21.
12.5
16.9
25.2
Table 9:
Test error (%) on CIFAR-IO-C level 3 corruptions.
shot
impul
defoc
glass
moin
zoom
snow
frost
fog
brit
contr
elust
pixel
ipeg
ResNet-26
40.0
33.8
26.4
11.5
37.3
20.0
16.6
20.0
24.7
12.2
10.5
13.6
15.0
18.4
22.7
TTA
34.3
27.7
20.3
11.3
32.9
20.7
16.7
16.3
9.8
8.5
12.5
13.7
14.5
17.2
MEME (ours)
34.4
27.9
20.5
10.8
32.8
19.8
16.1
16.1
20.9
9.6
8.6
11.7
13.2
14.5
17.2
Table 10:
Test error (%) on CIFAR- 10-C level corruptions.
gauss
shot
impul
defoc
glass motn
zoom
snow
frost
fog
brit contr elast pixel jpeg
ResNet-26
30.1
21.8
21.1
9.7
38.3
15.3
13.8
21.2
17.6
10.5
11.6
12.9
15.4
21.3
TTA
25.3
16.9
15.8
8.5
33.8
15.3
13.7
17.8
14.5
8.7
7.8
10.0
11.0
12.0
16.1
MEME (ours)
25.3
16.9
15.9
8.4
33.5
14.6
13.0
17.7
14.4
8.5
7.7
9.6
10.7
11.9
16.2
Table 11:
Test error (%) on CIFAR- level corruptions.
gauss
shot
impul
defoc glass motn
zoom
snow
frost
fog
brit contr elast pixel
jpeg
ResNet-26
20.8
16.5
15.8
38.9
11.8
12.8
13.9
13.4
9.7
9.6
13.1
12.0
16.4
TTA
15.8
12.8
7.3
35.1
10.8
12.5
11.0
10.8
7.4
7.7
11.4
MEME (ours)
16.1
12.9
34.7
10.4
11.0
10.7
7.4
7.5
10.9
16


Reviews:
[{"review_id": "nSJ9gRnO1e", "reviewer": "Reviewer_ZgMY", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I think that the general idea is interesting. It is indeed a short-coming of BatchNorm adaptation that one needs large batch sizes for this technique to really be effective. It is very relevant to have a method that works for single images. This would allow adapting models in an online-adaptation setting where there is only one image present from a certain distribution shift.\n\nThe authors should explicitly define the difference between f and p since currently, they seem to be using both interchangeably. Is p just f+softmax? Based on algorithm 1, I understand the procedure as following: The authors use one single test image and create B augmentations of it, pass B through the network, get B embeddings, calculate the mean over the embeddings, then calculate the loss based on the mean, i.e. for calculating the loss, they use p_tilde which is one number per dimension. The difference to the second ablation is that in the second ablation, they average after calculating H. Please make the distinction clearer in the text.\n\nI appreciate the authors uploading their code. I actually went through it to better understand the marginal entropy loss function and several other things.\n\nComparison to TENT. The authors write “This ablation is effectively a version of Tent (Wang et al., 2021) that is suitable for our test setting.” To me, the most important difference to TENT is that the model is reset after each image while in TENT, the model continually adapts to the full ImageNet-C set. Therefore, ablation 2 is NOT equivalent to TENT and this should be made clear.\n\nI think that the results are rather limited and several important ablations are missing. Especially, for a purely experimental paper I think more results should be presented.\n1.\tThe most relevant difference to other gradient-based test-time approaches is that the model is reset after each new image. It has not been tested whether TENT still works if one resets the model after each batch. \n2.\tIt is not clear to me how many gradient steps are performed on each image. In the code, n_iter is set to one but I think it is a hyperparameter that should be tuned.\n3.\tHow is B selected? I would appreciate a study for different values of B. In the limit case of B=1, one recovers the TENT setting, right? Currently, it is not clear to me that TENT does not work for B=1 since it has not been shown and TENT (with model resetting) actually performs equally well to MEME in Table 2 for a batch size of 64. -> Showing that TENT (+ model resetting) indeed does not work for small batch sizes and one needs this averaging over augmentations would be important.\n4.\tCurrently, the authors only study ResNet50 models and the Vision Transformer. For the ImageNet-A results, any results on ResNet50 models are biased (see my points below) and it is not clear how to adapt vision transformers (see my points below), making the results on both architectures hard to judge. I would suggest to the authors to test other model architectures, e.g., they could test the ResNext101 models (both vanilla and the IG-3.5B variants), which would be interesting because the IG-3.5B variant could not be adapted successfully with Batch Norm adaptation. \n4.b. In the abstract, the authors write \"In our experiments, we demonstrate that this approach consistently improves robust ResNet and vision transformer models\". To avoid over-claiming the success of the method, I would suggest to the authors to include all relevant robust ResNet50 models, e.g., the ones from Table 1 in Schneider et al. Alternatively, the authors have to remove the claim that MEME improves upon all robust ResNet50 models and write that MEME improves upon the vanilla, the DeepAug+Augmix and the MoEx+CutMix model.\n5.\tConsidering the second ablation point: “Conversely, as an objective that encourages confidence but not invariance…”, I find the presented results not conclusive enough to state that both confidence and invariance are important, as the difference are only 0.4 percent max. Possibly, more experiments should be performed to really stress this point. \n6.\tAugMix augmentations come in different severity levels. Checking the code, it seems the authors used a severity of 1 which is the smallest severity possible. This might explain the ablation result why the difference to standard augmentations is not that big. I think varying this parameter would be important.\n7. I would find it very interesting to see how MEME performs for larger batch sizes. The authors could plot a curve like Figure 1 in Schneider et al to see the effect of different batch sizes (alongside with the BN adapt curve). Ideally, the MEME curve should always be below the \"BN adapt\" curve. In the limes of the full dataset size as sample size, MEME should converge to the TENT result. Using the full dataset size as the batch size is obviously not possible due to GPU constraints, but one should see the asymptotic trend approaching that number. Maybe the authors could use batch sizes of [1,2,4,8,16... max_batch_size_per_GPU] for this analysis.\n\n### Comments on achieving a new state of the art claims:\nThe authors claim to achieve a new state of the art on various benchmarks on several occasions. \n1. “We achieve state-of-the-art results for test shifts caused by image corruptions (ImageNet-C), renditions of common objects (ImageNet-R), and, among ResNet-50 models, adversarially chosen natural examples (ImageNet-A).”\n2. “In Section 4, we demonstrate empirically that MEME consistently improves the performance of ResNet (He et al., 2016) and vision transformer (Dosovitskiy et al., 2021) models on several challenging ImageNet distribution shift benchmarks, achieving several new state-of-the-art results for these models.”\n3. “In particular, as we empirically show in Section 4, MEME can be composed with prior methods for training robust models and adapting model statistics, resulting in improvements over the prior state of the art for several ImageNet distribution shift benchmarks and model architectures.”\n4. “For both ImageNet-C and ImageNet-R, and for both the ResNet-50 and RVT_-small models, combining MEME with robust training techniques leads to new state of the art performance.”\n\nWhen speaking about state-of-the-art results on either ImageNet-C or ImageNet-R, one can either claim (1) a new state of the art for a certain architecture or (2) a new state of the art on ImageNet-C overall. \nConsidering case (1), the authors claim state-of-the-art results on ImageNet-C for a ResNet50 architecture which is wrong since both TENT and BN adaptation outperform their approach.\nConsidering case (2), the authors report a mCE of 40.6% on ImageNet-C and an error of 43.8% on ImageNet-R for their vision transformer. These are not state of the art overall, because for example, the Noisy Student model achieves a mCE of 28.3% on ImageNet-C and an error of 23.5% on ImageNet-R. \n\nThe authors achieve good results for the vision transformer model, but this is not a standard architecture (yet) so judging these numbers is hard. It is in general not very clear how transformers should be adapted. In conv-nets, usually the batchnorm layers are adapted (like in TENT), but maybe in transformers, one needs to adapt other parameters. For example, [5] and [6] experiment with “adapter” and “compacter” layers, respectively, to adapt transformers. The authors also raise this point when discussing their results: “Furthermore, we find that these methods are less effective with the RVT-small model, which may indicate their sensitivity to model architecture choices since vision transformer models are comparatively less reliant on BN layers (Steiner et al., 2021).” Yes, exactly. But to me, it does not mean that TENT and BN adaptation do not work per se, but that one needs to think about which layers need to be tuned.\n-> It might be that TENT would work if one used the layers suggested there, but not if one uses conventional BN adaptation.\n\nOn ImageNet-A, the Noisy Student model from [3] reports a top1 error of 16.5% which is much better than what the authors report here. Sure, the authors manage to use MEME to decrease the top1 error of a ResNet50 backbone to 89% (which is a slight improvement from 92%), but I find this result not very interesting, because it is still very bad and no one would actually use this model on ImageNet-A. In either case, their ImageNet-A results are not state of the art. In general, all ResNet50 results on ImageNet-A are very biased, because this dataset was explicitly designed to fool vanilla ResNet50 classifiers. So results reported for a ResNet50 do not generalize to other architectures. One can argue that MEME is the only method that improves upon the RVT results, but here again the issue from above arises where it is not clear how to adapt a vision transformer properly.\n\nTable 2: “A composition of robust training via strong data augmentations and MEME adaptation achieves new state-of-the-art performance on each benchmark for ResNet-50 models.” -> This is wrong: DeepAugment+Augmix+TENT achieve lower error rates on ImageNet-R and ImageNet-C for the ResNet50 backbone. I also find the bolding of the numbers inconsistent: If the best numbers are supposed to be bold, why is MEME bold as well despite being worse than TENT? The authors write “and we list in bold\nthe best results from these methods which outperform the test time robustness methods.” I don’t get it: BN adaptation and TENT are also test time robustness methods?\n\n\nAs a summary of this point, I think the authors need to rephrase their interpretation of the results since they do not achieve SotA results neither on ImageNet-C nor on ImageNet-R nor on ImageNet-A. \n", "summary_of_the_paper": "The authors study test-time adaptation to distribution shifts. They address the problem of single-sample adaptation and propose a method where they create several augmentations of a single image and average over the augmentations. They present results on different common robustness benchmarks.", "summary_of_the_review": "I really like the general idea of the paper of trying to find a method where adaptation to a single test sample is possible, with resetting the model after each sample. To my knowledge, this has not been done yet and it would be an interesting contribution to the field and highly relevant for practitioners. This is actually very similar to how robustness is evaluated when model weights are frozen and inference is done on every point separately. Here, one wants to adapt to each test data point.\n\nIt is obviously a harder problem since so far, models have been allowed to adapt to the whole test set, e.g. TENT studied this setting. And indeed, single-point BN adaptation is currently the setting closest to this one. So it is fine to not claim state-of-the-art results on ImageNet-C in this hard setting. Of course, methods that are allowed to see the full test set and adapt to it will be better. Therefore, I think the authors should refrain from wanting to claim state-of-the-art results over methods that are allowed to see the full test set, since it is an unreasonable request. Methods that are allowed to see the full test set should always perform better, and if they don’t, there is likely an optimization issue like with the vision transformers.\n\nI would suggest to the authors to pitch their paper in this setting: It is a new problem that no one has really studied so far which we propose a solution for and we ran all the necessary control experiments. That said, if one has access to many samples from the target domain, it should still be better to use all of them for adaptation. I think writing the pitch in this way should make the paper much more clear and easier to understand and put into context.\n\nIf one were to pitch the paper in this setting, one would need to test whether other test time adaptation methods work with model resetting, i.e. does TENT work for a batch size of 1 and with model resetting? Does one really need the different augmentations as claimed here?\n\nI think that the paper cannot be accepted in the current state, but I think the general setting is very interesting and it can become a relevant contribution if the pitch is changed and more results are added. I am open to have a discussion with the authors during the rebuttal period whenever they wish to. I am happy to raise my score if my points are addressed.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "knsiVlfckQ9", "reviewer": "Reviewer_8PMD", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength:\n * The problem of addressing domain shift is a highly significant topic\n * The proposed method can do adaptation on a per-datapoint level (in contrast to related works that require larger batches of inputs from the same target domain)\n * The experiments compare to the most related work on a large set of domain shift benchmarks and baseline models. Results are well-structured\n * The paper is very well written and structured.\n\nWeaknesses:\n * The proposed method has little novelty since it is a straightforward combination of existing methods: test-time augmentation, test-time entropy minimization (Wang et al., 2021), and adapting BatchNorm statistics (Schneider et al. 2020). \n * The proposed method considerably increases inference cost (by a factor of B) - this is undesirable in practice\n * As discussed below, some ablations/alternative design choices are missing\n\nSome alternative design choice that could strengthen the paper when included in the experiments/ablation studies:\n * Equation 1 computes the average of class predictions over augmented samples. Alternatively, the average over logits could be considered. \n * Varying the number of augmentations B in the experiments and discussing the trade-off between inference time and error\n * in general, repeating every run several times with different random seeds and report average error in the results\n * discuss why there is a \"stop-gradient\" in the loss $\\ell_{PCE}$ and study its effect\n *  $\\ell_{PCE}$ is not invariant to permutation of the batch since the second sum runs only over $j=(i+1)...B$. Moreover, this is particularly problematic since the gradient flows through the $\\tilde{x}_j$-inference for j-1 terms (those terms with i < j) .Thus samples with higher index j have larger weight in the loss. A simple fix would be to use a sum over $j \\neq i$.\n * since the authors use AugMix augmentations and AugMix-pretrained models, it would be natural to also add a JSD-type term to the loss at test-time that explicitly aims at encouraging invariance against augmentations.\n\nMinor comments:\n * In the tables, it is confusing to report errors and then a reduction of the error by X percent point with +X (it should be -X since it is a reduction of error)\n * how many iterations of SGD/ADAM are performed per datapoint? I did not find this information in the paper. Is it only one? If yes: why?", "summary_of_the_paper": "The paper aims at increasing robustness of image classifiers against domain shift using test-time adaptation on a single-instance level. For this, the authors apply a large set of random augmentation on the respective test input and adapt the model using entropy minimization on the marginal distribution of model outputs across augmentations. The proposed method is called \"Marginal Entropy Minimization with Ensembled Augmentations\" (MEME). MEME is combined with prior methods for compensating domain shift and evaluated on several domain shift benchmarks.", "summary_of_the_review": "The main contribution of the paper, the method MEME consists of plugging together two existing techniques, namely test-time augmentation and test-time entropy minimization (Wang et al., 2021). This combination is straight-forward and the technical novelty of the paper is quite small. \nOn the other hand, the empirical evaluation is relatively extensive and well structured . Consistent albeit small gains of the proposed method are observed. As discussed above, there are some ablations/alternative design choice that could be added to the experiments. Including those would give  a more complete picture.\n\nIn summary, I see this paper as marginally below the decision threshold, due to its limited novelty and incomplete ablations.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "iaWv4H5N7qu", "reviewer": "Reviewer_fKFu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n* Motivation is clear\n* Well-written and structured\n* Addressing an important and challenging problem setting\n* No assumptions on the availability of the entire test dataset\n* A simple approach that is easy to implement\n* Provided ablation study to understand the importance of the augmentations, confidence maximization and encouraging consistent predictions.\n* Significant improvements are noticed on vision transformer model that are less reliant on BatchNorm layers, e.g., about 8% improvement on ImageNet-C and ImageNet-R over the baseline.\n\nWeaknesses:\nMy major concerns lie on author's claims, novelty of the approach and effectiveness of the approach on CNNs.\n* Novelty of the approach is limited. Prior works have utilized augmentations during test time and entropy minimization for test time adaptation independently. This work can be seen as combination of these two existing approaches for test time adaptation.\n* Results on CIFAR-10 are similar or comparable to TTA.\n* Individual analysis of corruptions and severities on ImageNet-C reveal that improvements are marginal and range between 0%-2%.\n* Authors emphasize that invariances across the augmentation is significant but do not reflect that in the results, e.g., results in Table 3 show that major gains are obtained from confidence maximization of multiple augmentations but encouraging the invariance produce only marginal improvements. I assume similar behavior on ImageNet-C: considerable improvements are seen with confidence maximization of multiple augmentations but only marginal improvements by enforcing invariance.\n* TENT produce worse results on three ImageNet benchmarks with vision transformer. Since this architecture is less reliant on BatchNorm layers, how do updating all the parameters of this model would perform with TENT?\n", "summary_of_the_paper": "The paper focus on single image test time adaptation of deep neural network for distribution shift in classification task. Prior works perform test time adaptation on a batch of images or entire test dataset to capture the distribution statistics. The authors propose to create augmented copies of a provided test input and encourage invariance across different augmentations by promoting same prediction. Authors propose to minimize marginal entropy for confident predictions across the augmented versions of the input. Improvements are shown on CIFAR-10-C, CIFAR-10.1, ImageNet-C, ImageNet-R and ImageNet-A.\n", "summary_of_the_review": "Given the simplicity of the approach for single image test time adaptation, its improvements on vision transformer and my concerns listed above, I marginally accept this paper. I am willing to reconsider my rating based on other reviews and authors responses.\n\n-------- post rebuttal ---------------\n\nThe paper studies an interesting problem setting and also bring improvements in this setting, particularly on vision transformer. However, my concerns on authors claims still persist after the rebuttal and I also see that the novelty of the paper is limited. After considering the authors response and other reviewers comments, I rate this paper as borderline and keep my original score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "1: The main claims of the paper are incorrect or not at all supported by theory or empirical results.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "CLeCNhYMglT", "reviewer": "Reviewer_6Xgd", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths\n1. The proposed method is general since it doesn't need to alter the model training and supports online adaptation with one sample at a time.\n2. The single objective embraces two intuitions behind: invariance across augmentations and prediction confidence.\n3. The method is simple and easy to implement.\n\nWeaknesses\n1. The paper claimed that MEME could adapt only BN statistics in Section 3.2, but there seems no results presented in Section 4's tables.\n2. It is unclear how the testing is conducted. Does testing happen after adapting the model over all the test samples (one epoch or several epochs)? Or do we test on a sample immediately after adapting to it? Do different methods use the same testing procedure? More details are necessary at the beginning of the experiment section.\n3. According to Table 2, MEME can't beat other methods in several cases. It usually gets higher errors on datasets ImageNet-C and ImageNet-R than BN and Tent. Although it gets better results on ImageNet-A, it still has a higher error than the test-time augmentation baseline.\n4. Considering the entropy minimization has been explored in prior work Tent, the main novelty is bringing invariance to augmentations into test-time adaptation.\n5. If making model adaptation in deployment, the inference efficiency will suffer from forwarding multiple augmentations and updating the model weights. Any investigation on efficiency? It would be better to add some discussions on how to improve efficiency.", "summary_of_the_paper": "This paper studies the test-time model adaptation, where training data is not accessible and only one test example is available. It adapts model parameters by minimizing the output entropy on augmented test samples. Given one test sample, it first samples some augmentation operations, resulting in different augmentations. The entropy is computed on the average distribution of augmentations' outputs. Experiments on CIFAR-10 CIFAR-10.1 CIFAR-10-C, ImageNet-C, ImageNet-R, and ImageNet-A show that the proposed method effectively reduces classification errors.", "summary_of_the_review": "Test-time model adaptation is an interesting problem, providing a new perspective to improve model robustness. The proposed method builds on intuitive assumptions that are easy to understand. There are mainly two concerns regarding novelty and effectiveness. I think the authors may need to rethink designing experiments to outstand advantages.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree that test-time model adaptation is an interesting problem, providing a new perspective to improve model robustness. The proposed method builds on intuitive assumptions that are easy to understand. However, there are mainly two concerns regarding novelty and effectiveness. The paper can improve in these two aspects to meet ICLR standard.
Decision: Reject