MEMO: Test Time Robustness via Adaptation and Augmentation
Authors: Marvin Mengxin Zhang
Sergey Levine
Chelsea Finn
Venue: NeurIPS 2022 Accept (2022)
Keywords: distribution shift
test time adaptation
data augmentation

Abstract: While deep neural networks can attain good accuracy on in-distribution test points, many applications require robustness even in the face of unexpected perturbations in the input, changes in the domain, or other sources of distribution shift. We study the problem of test time robustification, i.e., using the test input to improve model robustness. Recent prior works have proposed methods for test time adaptation, however, they each introduce additional assumptions, such as access to multiple test points, that prevent widespread adoption. In this work, we aim to study and devise methods that make no assumptions about the model training process and are broadly applicable at test time. We propose a simple approach that can be used in any test setting where the model is probabilistic and adaptable: when presented with a test example, perform different data augmentations on the data point, and then adapt (all of) the model parameters by minimizing the entropy of the model's average, or marginal, output distribution across the augmentations. Intuitively, this objective encourages the model to make the same prediction across different augmentations, thus enforcing the invariances encoded in these augmentations, while also maintaining confidence in its predictions. In our experiments, we evaluate two baseline ResNet models, two robust ResNet-50 models, and a robust vision transformer model, and we demonstrate that this approach achieves accuracy gains of 1-8% over standard model evaluation and also generally outperforms prior augmentation and adaptation strategies. For the setting in which only one test point is available, we achieve state-of-the-art results on the ImageNet-C, ImageNet-R, and, among ResNet-50 models, ImageNet-A distribution shift benchmarks.

MEMO: Test Time Robustness via Adaptation and Augmentation
Marvin Zhang's Sergey Levine', Chelsea Finn? UC Berkeley Stanford University
Abstract
While deep neural networks attain good accuracy in-distribution test points, many applications require robustness even in the face unexpected perturbations in the input, changes in the domain, other sources of distribution shift. We study the problem of test time robustification, i.e., using the test input to improve model robustness. Recent prior works have proposed methods for test time adaptation, however, they each introduce additional assumptions, such as access to multiple test points, that prevent widespread adoption. In this work, we aim to study and devise methods that make no assumptions about the model training process and are broadly applicable at test time. We propose simple approach that can be used in any test setting where the model is probabilistic and adaptable: when presented with test example, perform different data augmentations on the data point, and then adapt (all of) the model parameters by minimizing the entropy of the model's s average, or marginal, output distribution across the augmentations. Intuitively, this objective encourages the model to make the same prediction across different augmentations, thus enforcing the invariances encoded in these augmentations, while also maintaining confidence in its predictions. In our experiments, we evaluate two baseline ResNet models, two robust ResNet-50 models, and a robust vision transformer model, and demonstrate that this approach achieves accuracy gains of 1-8% over standard model evaluation and also generally outperforms prior augmentation and adaptation strategies. For the setting in which only one test point is available, we achieve state-of-the-en results on the ImageNet-C. ImageNet-R, and, among ResNet-50 models, ImageNet-A distribution shift benchmarks.
Introduction
Deep neural network models have achieved excellent performance many machine learning problems, such as image classification, but are often brittle and susceptible to issues stemming from distribution shift. For example, deep image classifiers may degrade precipitously in accuracy when encountering input perturbations, such as noise or changes lighting [12] or domain shifts which occur naturally in real world applications 21 Therefore, robustification of deep models against these test shifts is an important and active area of study.
Most prior works in this area have focused on techniques for training time robustification, including utilizing larger models and datasets 64 various forms of adversarial training [3948]
48 and aggressive data augmentation [5] IK! 24 14]. Employing these techniques requires modifying the training process, which may not feasible if, e-g., it involves heavy computation or non public data. Furthermore, these techniques do not rely on any information about the test points that the model must predict on, even though these test points may provide significant information for improving model robustness. Recently, several works have proposed methods for improving accuracy via adaptation after seeing the test data, typically by updating a subset of the model's weights [44] 45 Though effective handling test shifts, these
18 normalization statistics 40 both 40 5
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

sligget teeil inpult
tramed model augmentation functions
(v) x))
{ aM
min H (po) 0
H
merginal entropy minimization
manginal outpuit po (y)x)
Pe(p/x)
pe(y
Figure 1: Left: at test time, as detailed Section we single test input x, set of data augmentation functions {ar,.... a.s) J. and a trained model that outputs a probabilistic predictive distribution and has adaptable parameters 0. We perform different augmentations on x and pass these augmented inputs to the model in order estimate the marginal output distribution averaged over augmentations. Right: we perform gradient update on the model minimize the entropy of this marginal distribution, thus encouraging the model predictions be invariant across different augmentations while maintaining confident predictions. The final prediction is then made on the original data point, i.e., the predictive distribution in the top right of the schematic.
methods sometimes still require specialized training procedures, and they typically rely on extracting information via batches or even entire sets of test inputs, thus introducing additional assumptions.
In this work, we focus on methods for test time robustness, in which the specific test input may be leveraged in order to improve the model's s prediction on that point. We are interested in studying and devising methods for improving model robustness that readily used with wide variety of pretrained models and test settings. We also want methods that synergize with other robustification techniques, in order to achieve greater performance than using either in isolation. With these goals in mind, we devise a novel test time robustness method based on adaptation and augmentation. As illustrated in Figure when presented with a test point, we adapt the model by augmenting the test point in different ways while encouraging the model to make consistent predictions, thus respecting the invariances encoded in the data augmentations. We further encourage the model to make confident predictions, thus arriving at the proposed method: minimize the marginal entropy the model's predictions across the augmented versions of the test point.
"plug and play", i.e., they can be
We refer to the proposed method as marginal entropy minimization with one test point (MEMO), and this is the primary contribution our work. MEMO makes direct use of pretrained models without any assumptions about their particular training procedure or architecture, while requiring only single test imput for adaptation. In Section 4 we demonstrate empirically that MEMO consistently improves the performance of ResNet IIII and vision transformer ? models on several challenging ImageNet distribution shift benchmarks, achieving several new seate-of-the-ane results for these models in the setting in which only one test point is available. MEMO consistently outperforms non adaptive marginal distribution predictions (between 1-10% improvement) on the ImageNet-C [12 and Net-e |4] test sets, indicating that adaptation plays crucial role in improving predictive accuracy. MEMO encourages both invariance across augmentations and confident predictions, and an ablation study in Section 4 shows that both components are important for maximal performance gains. Also, MEMO is, to the best of our knowledge, the first adaptation method to improve performance (by 1-4% over standard model evaluation) on the ImageNet-A test set [15].
2 Related work
Distribution shift has been studied under number of frameworks [36], including domain adaptation
41 1 4 domain generalization (5] 32 9 and distributionally robust optimization E! 16 39) These
frameworks typically leverage additional training test assumptions in order to make the distribution shift problem more tractable. Largely separate from these frameworks, various empirical methods have also been proposed for dealing with shift, such increasing the model and training dataset

---

size or using heavy training augmentations these efforts: MEMO is applicable to a wide range of pretrained models, including those trained via robustness methods, and can achieve further performance gains via test time adaptation.
34 51 14
The focus of this work is complementary to
Prior test time adaptation methods generally either make significant training or test time assumptions. Some methods update the model using batches or even entire datasets of test inputs, such as by computing batch normalization (BN) statisties on the test set [25]
computing class prototypes T8]. or minimizing the (conditional) entropy of model predictions across a batch of test data 46 The latter approach is closely related MEMO. The differences are that MEMO minimizes marginal entropy using single test points and data augmentation and adapts all of the model parameters rather than just those associated with normalization layers, thus not requiring multiple test points or specific model architectures, Other test time adaptation methods can be applied to single test points but require specific training procedures or models [44]
19
3 3D,
40 6) Test time training (TTT) 44 requires a specialized model with a rotation prediction head and different procedure for training this model. Schneider et al. 40 show that BN adaptation can be effective even
with only one test point. As we discuss in Secciinn 3 MEMO synergizes well with this technique of "single point" BN adaptation. Mao et al. 22 propose a test time adaptation method based on input perturbations for robustness to adversarial attacks. Concurrently with our work, Sivaprasad and Fleuret 43 propose a similar method for test time adaptation by encouraging invariance to data augmentations, and they test their method on the corrupted CIFAR and VisDA (35] datasets.
number of works have noted that varying forms of strong data augmentation on the training set can improve the resulting model's robustness 51 13 24 [4] Data augmentations are also sometimes used on the test data directly by averaging the model's outputs across augmented copies of the test point 2 42 i.e., predicting according to the model's 's marginal output distribution. When using cropping as the augmentation, this technique is often referred to as multicrop evaluation (2). We instead use the term test time augmentation (TTA),
we use additional augmentations beyond cropping 13. TTA has been shown to be useful both for improving model accuracy and calibration 2 as well as handling distribution shift [3. We take this idea one step further by explicitly adapting the model such that its marginal output distribution has low entropy. This extracts an additional learning signal for improving the model, and furthermore, the adapted model can then make its final prediction on the clean test point rather than the augmented copies. We empirically show in inSection 4 that these differences lead to improved performance over this non adaptive TTA baseline.
3 Augmenting and Adapting at Test Time
Data augmentations are typically used to train the model to respect certain invariances e.g., changes in lighting or viewpoint do not change the underlying class label - but, especially when faced with distribution shift, the model is not guaranteed to obey the same invariances at test time. In this section, we introduce MEMO, method for test time robustness that adapts the model such that it respects these invariances on the test input. We use "test time robustness" specifically to refer to techniques that operate directly on pretrained models and single test inputs single point BN adaptation and TTA, as described in Section 2 are examples of prior test time robustness methods.
In the test time robustness setting, we given a trained model fe with parameters 0 E e. We do not require any special training procedure and do not make any assumptions about the model, except that 0 is adaptable and that fo produces conditional output distribution (x(x) that is differentiable with respect to M All standard deep neural network models satisfy these assumptions. A single point x E X' is presented to fe, for which it must predict label ÿ E y immediately. Note that this is precisely identical to the standard test time inference procedure for regular supervised learning models in effect, we simply modifying how inference is done, without any additional assumptions on the training process or on test time data availability. This makes test time robustness methods a simple "slot-in" replacement for the ubiquitous and standard test time inference process. We assume sampling access to a set of augmentation functions A A {ai,.... amr) } that can be applied to the test point x. We use these augmentations and the self-supervised objective detailed below to adapt the model before it predicts on x. When given a set of test inputs, the model adapts and predicts on each test point independently. We do not assume access to any ground truth labels.
Siiggle point BN adaptation also assumes that the model has BN layers as shown empirically in Section 4 this is assumption that we do not require but can also benefit from.
3

---

Algorithm 1 Test time robustness via MEMO
Require: trained model fe, test point x, number of augmentations B, learning rate 7) update rule G
1: Sample 01...., ag i.d. (() and produce augmented points x; = a, ax(x) for E {1 fI..... B}
2: Compute estimate p = Ei=1 (x)) = Po (v(x) and = (p) x x(0;x), x) i.c. Eq. 2
3: Adapt parameters via update rule O' + C)
4: Predict g _ arg maxy Pe (y/x)
3.1 Marginal Entropy Minimization with One test point
Given test point x and set of augmentation functions A, we sample B augmentations from A and apply them to x in order to produce batch of augmented data x,,..., xp. The model's average, or marginal, output distribution with respect to the augmented points is given by
pe Po(y|x) x) B(((A) (po(yla(x)))
Σ po(y)&;), B
where the expectation with respect to uniformly sampled augmentations a ~ U(A).
What properties do we desire from this marginal distribution? To answer this question, consider the role that data augmentation typically serves during training. For each training point (x!rain, mrain) the model fa is trained using multiple augmented forms of the input x|"*,, ...., X f is trained to obey the invariances between the augmentations and the label - no matter the augmentation on xitainn f
should predict, with confidence, the same label ش We seek to devise similar learning signal during test time, without any ground truth labels. That is, after adapting:
(1) the model fe predictions should be invariant across augmented versions of the test point, and (2) the model fe should be confident in its predictions, even for heavily augmented versions of the test point, since all versions have the same underlying label.
Optimizing the model for more confident predictions can be justified from the assumption that the true underlying decision boundaries between classes lie in low density regions of the data space [X] With these two goals in mind, we propose to adapt the model using the entropy of its marginal output distribution over augmentations (Eg. 1). i...
E(0: x) - H (po(-
Σ log
Note that this objective is not the same as optimizing the average conditional entropy the model's predictive distributions across augmentations, i.e.,
lee (f=(0::0 x)
e
Σ Hpoo--1x:) B
1
(3)
model which predicts confidently but differently across augmentations would minimize Eq. but
not Bq. 2 Optimizing Eq. 2 encourages both confidence and invariance, since the entropy of pa
(x)
minimized when the model outputs the same (confident) prediction regardless of the augmentation.
Algorithm I presents the overall method MEMO for test time adaptation. Though prior test time adaptation methods must carefully choose which parameters to adapt in order to avoid degenerate solutions [А], our adaptation procedure simply adapts all of the model's parameters 0 (line 3). Given that po po(ylx) x) is differentiable with respect to 0, can directly
gradient based optimization to adapt according to I...9 2, We use only one gradient step per test point, because empirically we found this to be sufficient for improved performance while being more computationally efficient. After this step, we use the adapted model fe to predict on the original test input x (line 4).
3.2 Composing MEMO with Prior Methods
An additional benefit of MEMO is that it synergizes with other approaches for handling distribution shift. In particular, MEMO can be composed with prior methods for training robust models and adapting model statistics, thus leveraging the performance improvements of each technique.

---

IN-C bafore MEMO
"stligraa" (52.91%))
"dishrag" (1.32%)
"goose" (3.29%)
"stligray" (26.67%)
"goldfieh" (40.59%%
"stingray" (37. 69%)
"stlngray" (20 80%)
"dugong" (3.52%)
"stlngray" (89.433%)
"dugong" (3.58%%
"window" (0.80%)
"stingray" (50.33%)
"stingray" (24.07%)
"sock" (5.20%)
"stingray" (40.20%)
"stingray" (44.77%%
IN-C after MEMO
"stingray" (98 47%)
"wall clock" [1.88 (1.80%)
"stingray" (63. (63.24%)
"stingray" (78.80%)
"stingray" (73.47%)
"stingray" (74.36%)
"stingray" 72%)
"dugong" (16 (496))
"stingray" (84.76%)
"stingray" (18.59%)
"bathtub" (2.45%)
"stingray" (92.25%)
"stingray" (82.18%)
"stingray"
"stingray" (83, 18%)
"stingray" (6.80%%
(68. 11 11%)
Figure 2: We visualize augmentations of randomly chosen data point from the "Gaussian Noise level 3" ImageNet-C test Even for robust model trained with heavy data augmentations [14. both predictive accuracy and confidence (as shown the top two rows) drop sharply when encountering test shift, As shown in the bottom two rows, these drops can be remedied via MEMO adaptation.
Pretrained robust models. Since MEMO makes assumptions about, modifications to, the model training procedure, performing adaptation on top of pretrained robust models, such as those trained with heavy data augmentations, is as simple as using any other pretrained model. Crucially, we find that, in practice, the set of test augmentations A does not have to match the augmentations that were used train the model. For simplicity and efficiency, easily sampled and are applied directly to the model input x. These propertics do not hold for, e.g., data augmentation techniques based image translation models, such DeepAugment [14), or feature mixing. such as moment exchange 24.. However, we can still use models trained with these data augmentation techniques
augmentations that can be
starting point for adaptation, thus allowing their state-of-the-ard results. As noted above, using pretrained models is not as easily accomplished for adaptation methods which require complicated
improve upon
specialized training procedures and model architectures, such as TTT ! or ARM (52). In our experiments, we use Aug as our set of satisfies the above properties and still yields significant diversity when
augmentations [K]] applied, as depicted in Figure 2 Note that AugMix explicitly does not use augmentations that are similar the corruptions in the CIFAR-IO-C and ImageNet-C test
Adapting BN statistics. Schneider et al. 40 showed that, even when presented with just single test point, partially adapting the estimated mean and variance of the activations in each batch normalization (BN) layer of the model can still be effective in some cases for handling distribution shift, this setting, to prevent overfitting to the test point, the channelwise mean and variance [Most Tin)] estimated from this point are mixed with the the mean and variance Marain o2 computed during training according to prior strength N. That is, for v E {14, o²).
+1 Ptrain +
N W+1 Vest
N
This technique is also straightforward to combine with MEMO: we simply use the adapted BN statistics whenever computing the model's output distribution. We find in our experiments that this technique never degrades, and generally improves, the performance of test time adaptation, thus we combine MEMO with this technique by default whenever applicable. Following the suggestion in Schneider et al.
40
we set N 16 for all of our experiments in the next section.
5

---

4 Experiments
Our experiments aim to answer the following questions:
(1) How does MEMO compare prior methods for test time adaptation and test time robustness?
(2) Can MEMO be combined with a wide range of model architectures and pretraining methods?
(3) Which aspect of MEMO, the adaptation or augmentation, is the most important?
We evaluate MEMO on total of five distribution shift benchmarks. We conduct CIFAR-10 experiments on the CIFAR-I0-C 12] and CIFAR-I0. 37 test sets, and we conduct ImageNet experiments on the ImageNet-C
3X
ImageNet-R
and ImageNet-A
15
test sets.
To answer question (1), we compare to test time training (TTT) 44 in the CIFAR-10 10 experiments, for which we train 26 models following their protocol and specialized architecture. We do not compare to TTT for the ImageNet experiments due to the computational demands of training state-of-the-art models and because Sun al. [44] do not report competitive ImageNet results. For the ImageNet experiments, we compare to Tent 46 and BN adaptation, which can be used with pretrained models but require multiple test inputs (or even the entire test set) for adaptation. We provide BN adaptation with 256 test inputs at time and set the prior strength N = 256 400
For Tent, we use test batch sizes of 64 and, for ResNet-50 models, test both "online" adaptation where the model adapts continually through the entire evaluation and "episodic" adaptation where the model is reset after each test batch [6). Note that the evaluation protocols different for these two methods: whereas MEMO is tasked with predicting on each test point immediately after adaptation, BN adaptation predicts a batch of 256 test points after computing BN statistics the batch, and Tent predicts on a batch of 64 inputs after adaptation but also, in the online setting, continually adapts throughout evaluation. In all experiments, we further compare to single point BN adaptation 20 and the TTA baseline that simply predicts according to ( x) (Eq. 23 2]. Full
details on our experimental protocol are provided in Appendix A
To answer question (2), we apply MEMO on top of multiple pretrained models with different architectures, trained via several different procedures. For CIFAR-10, train own ResNet- 26 [I]] models. For ImageNet, we use the best performing ResNet-50 robust models from prior work, which includes those trained with DeepAugment and AugMix augmentations [4] as well as those trained with moment exchange and CutMix [24]. To evaluate the generality of prior test time robustness methods and MEMO, we also evaluate the small robust vision transformer -RVTT-small), which provides superior performance on all three ImageNet distribution shift benchmarks compared to the robust ResNet-50 models (0). Finally, we evaluate ResNext- 101 models (50 28 on ImageNet-A, as these models previously achieved the strongest results for this test set [4].
Finally, to answer (3), we conduct ablative studies in subsection 4.2 first to determine the relative importance of maximizing confidence (via entropy minimization) versus enforcing invariant predic- tions across augmented copies of each test point, second to determine the importance of the particular augmentation functions used, and third to determine the required number of augmented samples per inference. The comparison to the non adaptive TTA baseline also helps determine whether simply augmenting the test point is sufficient or if adaptation is additionally helpful. In Appendix B] we provide further experiments ablating the augmentation component specifically.
4.1 Main Results
We summarize results for CIFAR-10, CIFAR- and CIFAR- in Table 1 with full CIFAR- 10- C results in Appendix C We use indentations to indicate composition, e.g., TTT is performed at test time on top of their specialized joint training procedure. Across all corruption types in CIFAR-IO-C, MEMO consistently improves test error compared the baselines, non adaptive TTA, and TTT. MEMO also provides a larger performance gain on CIFAR-10.1 compared to TTT. We find that the non adaptive TTA baseline is competitive for these relatively simple test sets, though than MEMO for CIFAR- 10-C. Of these three test sets, 1FARR-1--C is the only benchmark that explicitly introduces distribution shift, which suggests that adaptation useful when the test shifts are more prominent. Both TTA and MEMO are also effective at improving performance for the original CIFAR-10 test set where there is no distribution shift, providing further support for the widespread use of augmentations in standard evaluation protocols 2 2
worse

---

Table 1: Results for CIFAR-10, 10, CIFAR-10.1, and CIFAR-1O-C. "Results from Sun et al.
44
CIFAR-10
CIFAR-10.1 Error (%)
CIFAR-IO-C Average Error (%)
Error (%)
ResNet-26 ||]
9.2
18.4
22.5
+ TTA
7.3 (-1.9)
14.8 (-3.6)
19.9 (-2.6)
MEMO (ours)
7.3 (-1.9)
14.7 (-3.7)
19.6 (-2.9)
+ Joint training"
8.1
16.7
22.8
TTT*
7.9 (-0.2)
15.9 (-0.8)
21.5 (-1.3)
Table 2: Test results for the ImageNet test sets. MEMO achieves new state-of-the-art performance on each benchmark for ResNet-50 models for the single test point setting. For RVT -mall, MEMO improves performance across all benchmarks and reaches new state of the art for ImageNet-C and ImageNet-R, Compared to prior approaches, MEMO offers more consistent improvements.
ImageNet-C mCE
ImageNet-R Error (%)
ImageNet-A Error (%)
Baseline ResNet-50
76.7
63.9
100.0
TTA
77.9 (+1.2)
61.3 (-2.6)
98.4 (-1.0)
Single point BN
71.4 (-5.3)
61.1 (-2.8)
99.4 (-0.6)
+ MEMO (ours)
69.9 (-6.8)
58.8 (-5.1)
99.1 (-0.9)
BN (N 256, = 256)
61.6
59.7 (-1.2)
99.8 (-0.2)
+ Tent (online) 46
54.4 (-22.3)
57.7 (-6.2)
99.8 (-0.2)
Tent (episodic)
64.7 (-12.0)
61.0 (-2.9)
99.7 (-0.3)
+ AeepAuument+AugMiix 14
53.6
53.2
96.1
TTA
55.2 (+1.6)
51.0 (-2.2)
93.5 (-2.6)
+ Single point BN
51.3 (-2.3)
51.2 (-2.0)
95.4 (-0.7)
MEMO (ours)
49.8 (-3.8
49.2 (-4.0)
94.8 (-133)
+ BN (N = 256, n 256)
45.4 (-8.2)
48.8 (-4.4)
96.8 (+0.7)
Tent (online)
43.5 (-10.1)
46.9 (-6.3)
96.7 (+0.6)
+ Tent (cpisodic)
47.1 (-6.5)
50.1 (-3.1)
96.6 (+0.5)
+ MoEx+CutMix 24
74.8
64.5
91.9
TTA
75.7 (+0.9)
62.7 (-1.8)
89.5 (-2.4)
Single point BN
71.0 (-3.8)
62.6 (-1.9)
91.1 (-0.8)
+ MEMO (ours)
69.1 (-5.7)
59.4 (-3.3)
89.0 (-2.9)
+ BBN (N 256, n 256)
60.9 (-13.9)
61.6 (-2.9)
93.9 (42.0)
Tent (online)
54.0 (-20.)
58.7 (-5.8)
94.4 (+2.5)
Tent (episodic)
66.2 8866)
63.9 (-0.6)
94.7 (+2.8)
RVT'-small [30
49.4
52.3
73.9
TTA
53.0 (+3.6)
49.0 (-3.3)
68.9 (-5.0)
+ Single point BN
48.0 (-1.4)
51.1 (-1.2)
74.4 (+0.5)
MEMO (ours)
40.6 (-8.8)
43.8 (-8.5)
69.8 (-4.1)
+ BN (N 256, n 256)
44.3 (-5.1)
51.0 (-1.3)
78.3 (+1.)
Tent (online)
46.8 (-2.6)
50.7 (-1.6)
82.1 (48.)
+ Tent (adapt all)
44.7 (-4.7)
74.1 (+21.8)
81.1 (+7.2)
We summarize results for ImageNet-C. ImageNet-R, and ImageNet-A Table 2 with complete ImageNet-C results in Appenddx C We again use indentations to indicate composition, e.g., the best results on ImageNet-C for our setting are attained through a combination of starting from model trained with DeepAugment and Aug [4] and using MEMO on top. For both ImageNet-C and ImageNet-R. and for both the ResNet-50 and RVT* models, combining MEMO with robust training techniques leads to new state-of-the art performance among methods that observe only one test point time. We highlight gray the methods that require multiple test points for adaptation, and we list in bold the best results from these methods which outperform the test time robustness methods. As Table and prior work both show id accessing multiple test points can be powerful for benchmarks such ImageNet-C. and ImageNet-R, in which inferred statistics
7

---

from the test input distribution may aid in prediction. However, these methods do not help, and oftentimes even hurt, for ImageNet-A. Furthermore, we find that these methods are less effective with the RVT model, which may indicate their sensitivity to model architecture choices. Therefore, for this model, we also test modification Tent which adapts all parameters, and we find that this version of Tent works better for ImageNet-C but is significantly worse for ImageNet-R.
MEMO also results in substantial improvement for ImageNet-A. No prior test time adaptation methods have reported improvements on ImageNet-A, and some have reported explicit negative results 40 As discussed, it reasonable for adaptation methods that rely on multiple test points to achieve greater success on other benchmarks such as ImageNet-C, in which a batch of inputs provides significant information about the specific corruption that must be dealt with. In contrast, ImageNet-A does not have such obvious characteristics associated with the input distribution, as it is simply a collection of images that are difficult to classify. As MEMO instead extracts learning signal from single test points, it is, to the best of our knowledge, the first test time adaptation method to report successful results on this testhed. We view the consistency with which MEMO outperforms the best prior methods, which change across different test sets, major advantage of the proposed method.
TTA is the most competitive prior method for ImageNet-A, e.g., it results in larger improvements than MEMO for the RVT -mall model. MEMO, how- ever, achieves state-of-the-an performance among ResNet-50 models. To further compare MEMO to TTA, in Table 3] we evaluate whether MEMO can successfully adapt ResNext-101 models [50) and further improve performance on this challenging test set. We evaluate both a ResNext- 101 (32x8d) baseline model pretrained on Ima- geNet, as well as the same model pretrained with weakly supervised learning (WSL) billions of Instagram images 28 For the WSL model, we did not use single point BN adaptation for MEMO found this technique be actually harmful to performance, and this corroborates previous findings [40]. From the results, we that, although both TTA and MEMO significantly improve upon the baseline model evaluation, MEMO ultimately achieves the best accuracy by a significant margin as it is more successful at adapting the WSL model. This suggests that MEMO may synergize well with large scale pretraining, and further exploring this combination is an interesting direction for future work.
Table 3: ImageNet-A results for the ResNext-1011
ImageNet-A Error (%)
ResNext- 101 50
90.0
+ TTA
83.2 (-6.8)
Single point BN
88.8 (-1.2)
+ MEMO (ours)
84.3 (-5.7)
+ WSL. 25
54.9
+ TTA
49.1 (-5.8)
Single point BN
58.9 (+1.0)
+ MEMO (ours)
43.2 (-11.7)
4.2 Ablative Study
MEMO uses both adaptation and augmentations. In this section, we ablate the adaptation procedure and the number of augmentations, and in Appendix B we ablate the choice of augmentations.
Adaptation procedure. From the results above, conclude that adaptation generally provides additional benefits beyond simply using TTA to predict via the marginal output distribution fo (yl x). However, we can disentangle two distinct self-supervised learning signals that may be effective for adaptation: encouraging invariant predictions across different augmentations of the test point, and encouraging confidence via entropy minimization. The marginal entropy objective in Eq. encapsulates both of these learning signals. but it cannot easily be decomposed into these pieces. We instead use two ablative adaptation methods that each only make use of one of these learning signals.
First, we consider optimizing the pairwise cross entropy between each pair of augmented points, i.e.,
B
1
H Hpoo (x),, 1x)))
B
Where x; again refers to the i-th sampled augmentation applied to x. Intuitively, this loss function encourages the model to adapt such that it produces the same predictive distribution for all aug- mentations of the test point, but it does not encourage the model to produce confident predictions.
8

---

Table 4: Ablating the adaptation objective test pairwise entropy and conditional entropy (CE)
based adaptation. MEMO generally performs the best. indicating that both encouraging invariance across augmentations and confidence are helpful in adapting the model.
CIFAR-10 Error (%)
CIFAR-IO.. 10.1 Error (%)
CIFAR-IOS -0-C
Average Error (%)
ResNet-26 I
9.2
18.4
22.5
+ MEMO (ours)
7.3 (-1.9)
14.7 (-3.7)
19.6 (-2.9)
-g 2 PCC
7.6 (-1.6)
15.3 (-3.1)
20.0 (-2.3)
- L6. + cE
7.6 (-1.6)
14.7 (-3.7)
20.0 (-2.5)
ImageNet-C mCE $
ImageNet-R Error (%)
ImageNet-A Error (%)
RVT*-smal 30
49.4 40.6 (-8.8)
52.3
73.9
MEMO (ours)
43.8 (-8.8)
69.8 (-4.1)
f... 2) fcE
41.2 (-8.2)
44.2 (-8.1)
69.7 (-4.2)
Baseline Reshat-5n
Despliugment AugMix McEx CutNix
47.0
4b.5
Test Enor 196
60.0
Emror (%6
46.0
45.5
55.0
45.0
44.5
50.0
128
44.0
128
0.6 0.8
1.0
0.2
0.4 0.6
1.0
Seconds per evaluation
per
Figure 3: Plotting MEMO efficiency seconds per evaluation (x axis) and % test ImageNet-R (y axis) for the ResNet-50 models (left) and RVT*-small (right) while varying B (1, 4, 8, 16, 32, 64, 128). Note the log scale the axis.
Conversely, as an objective that encourages confidence but not invariance, we also consider opti- mizing the conditional entropy objective detailed Eq. This ablation effectively a version of the episodic variant of Tent [46 that produces augmented copies of a single test point rather than assuming access to test batch. We first evaluate these ablations on the CIFAR-10 10 test sets. We use the same adaptation procedure and hyperparameters, with l replaced with the above objectives.
The results are presented in Table 4 We see that MEMO, i.e., marginal entropy minimization, generally performs better than adaptation with either of the alternative objectives. This supports the hypothesis that both invariance across, and confidence on, the augmentations are important learning signals for self-supervised adaptation. When faced with CIFAR- 10.1, we see poor performance from pairwise cross entropy based adaptation method. the original CIFAR- 10 test set and CIFAR- 10- C, the ablations perform nearly identically and uniformly worse than MEMO. To further test the lcE ablation, which is the stronger the two ablations, we also evaluate it on the ImageNet test sets for the RVT* model. We find that, similarly, minimizing conditional entropy generally improves performance compared to the baseline evaluation. MEMO is more performant for ImageNet-C and ImageNet-R. Adaptation via fcE performs slightly better for ImageNet-A, though for this problem and model, TTA is still the best method. Thus, MEMO results relatively small, but consistent, performance gains compared to only maximizing confidence on the augmentations.
Number of augmentations. In Figure 3 tion on ImageNet-R as function of the efficiency of adaptation, measured seconds per evaluation. We achieve various tradeoffss by varying the number of augmented copies B
we analyze the % test error of MEMO adapta-

---

(1, 8, 16, 32, 64, 128). We note that small values of B such 4 and 8 can already provide significant performance gains, thus a practical tradeoff between efficiency and accuracy is possible.
For large B, the wall clock time is dominated by computing the augmentations. For the baseline ResNet-50 model, single point BN adaptation requires average of 0.0252 seconds per test point. TTA and MEMO with / = 64 are much slower - 0.7742 and 0.9746 seconds, respectively, per test point- but can be made significantly more efficient by using B 4 augmented samples 0.0631 and 0.1037 seconds, respectively. In our implementation, we do not compute augmentations in parallel, though in principle this is possible for Mug and should drastically improve efficiency overall. These experiments used four Intel Xeon Skylake 6130 CPUs and one NVIDIA TITAN RTX GPU.
5 Discussion
We presented MEMO, a method for test time robustification again distribution shift via adaptation and augmentation. MEMO does not require access or changes to the model training procedure and is thus broadly applicable for a wide range of model architectures pretrained in number of different ways. Furthermore, MEMO adapts test time using single test inputs, thus does not assume access to multiple test points as in several recent methods for test time adaptation 440 46 On a range of CIFAR-10 and ImageNet distribution shift benchmarks, and for ResNet, vision transformer, and, to an extent, ResNext models, MEMO consistently improves performance at test time and achieves several new state-of-the-arr results for these models in the single test point setting.
Inference via MEMO is more computationally expensive than standard model inference due to its augmentation and adaptation procedure though, as the experiments above show, more favorable tradcoffs between efficiency and accuracy are possible with smaller values of B, the number of augmentations per test point. One interesting direction for future work is to develop techniques for selectively determining when to adapt the model in order to achieve more efficient inference. For example, with well calibrated models [TO). we may run simple "feedforward" inference when the prediction confidence is over a certain threshold, thus achieving better efficiency. Additionally, it would be interesting to explore MEMO in the test setting where the model is allowed to continually adapt as more test data is observed. In our preliminary experiments in this setting, MEMO tended to lead to degenerate solutions, e.g., the model predicting a constant label with maximal confidence regardless of the input. This failure mode may potentially be rectified by carefully choosing which parameters to adapt, such as only adapting the parameters in BN layers [46], or regularizing the model such that it does not change too drastically from the pretrained model 26
Acknowledgments and Disclosure of Funding
We thank members of the Robotic Al and Learning Lab and Berkeley Al Research for helpful discussions and feedback. MZ was supported part by an NDSEG fellowship. CF a CIFAR fellow. This research was partially supported by ARL DCIST CRA W911NF--1-2-0081 and ARO W911NF-21-1-0097.
References
[1] F. Alet, M. Bauza, K. Kawaguchi, N. Kuru, T. Lozano-Péree and L. Kaelbling. Tailoring: Encoding inductive biases by optimizing unsupervised objectives at prediction time. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[2] A. Ashukha, Lyzhov, D. Molchanov, and D. Vetrov. Pitfalls of in-domain uncertainty estima- tion and ensembling in deep learning. In International Conference Learning Representations (ICLR), 2020.
[3] A. Bartler, A. Bühler, F. Wiewel, M. Döbler, and B. Yang. MT3: Meta test-time training for self-supervised test-time adaption. In International Conference on Artificial Intelligence and Statistics (AISTATS), 2022.
[4] A. Ben-Tal, den Hertog, A. De Waegenaere, B. Melenherg, and G. Rennen. Robust solutions of optimization problems affected by uncertain probabilities. Management Science, 2013.
10

---

[5] G. Blanchard, G. Lee, and C. Scott. Generalizing from several related classification tasks to new unlabeled sample. In Advances in Neural Information Processing Systems (NIPS), 2011.
[6] G. Csurka. Domain adaptation for visual applications: A comprehensive survey, arXiv preprint arXiv:1702.05374, 2017.
[7] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations (ICLR), 2021.
[8] Y. Grandvalet and Bengio. Semi-supervised learning by entropy minimization. In Advances in Neural Information Processing Systems (NIPS), 2005.
[9] I. Gulrajani and D. Lopez-Paz. In search of lost domain generalization. In International Conference on Learning Representations (ICLR), 2021.
[10] C. Guo, G. Pleiss, Y. Sun, and K. Weinberger. On calibration of modern neural networks. In International Conference on Machine Learning (ICML), 2017.
[11] K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
[12] D. Hendrycks and T. Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In International Conference on Learning Representations (ICLR), 2019.
[13] D. Hendrycks, N. Mu, E. Cubuk, B. Zoph, J. Gilmer, and B. Lakshminarayanan. AugMix: A simple data processing method to improve robustness and uncertainty. In International Conference on Learning Representations (ICLR), 2020.
[14] D. Hendrycks, Basart, N. Mu, S. Kadavath, F. Wang. E. Dorundo, R. Desai, T. Zhu, S. Parajuli, M. Guo, D. Song, Steinhardt, and J. Gilmer. The many faces of robustness: A critical analysis of out-of-distributio generalization. In IEEE International Conference on Computer Vision (ICCV), 2021.
[15] D. Hendrycks, K. Zhao, S. Basart, Steinhardt, and D. Song. Natural adversarial examples. In Conference Computer Vision and Pattern Recognition (CVPR), 2021.
[16] W. Hu, G. Niu, I. Sato, and M. Sugiyama. Does distributionally robust supervised learning give robust classifiers? In International Conference on Machine Learning (ICML), 2018.
[17] Y. Huang, J. Gornet, S. Dai, Z. Yu, T. Nguyen, D. Tsao, and A. Anandkumar. Neural networks with recurrent generative feedback. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[18] Y. Iwasawa and Y. Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[19] A. Kaku, Mohan, A. Parnandi, H. Schambra, and C. Frandne Be like wa- ter: Robustness to extraneous variables via adaptive feature normalization. arXiv preprint arXiv:2002.04019 2020.
[20] D. Kingma and Ba. Adam: A method for stochastic optimization. In International Conference on Learning Representations (ICLR), 2015.
[21] Koh, S. Sagawa, H. Marklund, S. Xie, M. Zhang, A. Balsubramani, W. Hu, M. Yasunaga, R. Phillips, I. Gao, T. Lee, E. David, I. Stavness, W. Guo, B. Earnshaw, Haque, S. Beery, J. Leskovee, A. Kundaje, E. Pierson, S. Levine, C. Finn, and P. Liang. WILDS: A benchmark of in-the-wild distribution shifts. In International Conference on Machine Learning (ICML), 2021.
[22] A. Krizhevsky. Learning multiple layers of features from tiny images. Technical report, University of Toronto, 2009.

---

[23] A. Krizhevsky, I. Sutskever, and G. Hinton. ImageNet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS), 2012.
[24] B. Li, F. Wu, S. Lim, S. Belongie, and K. Weinberger. On feature normalization and data augmentation. In Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[25] Y. N. Wang, J. Shi, J. Liu, and X. Hou. Revisiting batch normalization for practical domain adaptation. In International Conference on Learning Representations Workshop (ICLRW), 2017.
[26] Y. Liu, P. Kothari, B. van Delft, B. Bellot-Gurlet, T. Mordan, and A. Alahi. TTT++; When does self-supervised test-time training fail or thrive? In Advances in Neural Information Processing Systems (NeurIPS), 2021.
[27] I. Loshchilov and F. Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019.
[28] D. Mahajan, R. Girshick, V. Ramanathan, K. He, M. Paluri, Y. Li, A. Bharambe, and L. van der Maaten. Exploring the limits of weakly supervised pretraining. In European Conference on Computer Vision (ECCV), 2018.
[29] C. Mao, M. Chiquier, H. Wang, J. Yang, and C. Vondrick. Adversarial attacks are reversible with natural supervision. In IEEE International Conference on Computer Vision (ICCV), 2021.
[30] X. Mao, G. Qi, Y. Chen, X. Li, R. Duan, S. Ye, Y. He, and H. Xue. Towards robust vision transformer. In Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[31] D. Molchanov, A. Lyzhov, Y. Molchanova, A. Ashukha, and D. Vetrov. Greedy policy search: A simple baseline for learnable test-time augmentation. In Conference on Uncertainty in Artificial Intelligence (UAI), 2020.
[32] K. Muandet, D. Balduzzi, and B. Schölkopf. Domain generalization via invariant feature representation. In International Conference on Machine Learning (ICML), 2013.
[33] Z. Nado, S. Padhy, D. Sculley, A. D' A'mour,, B. Lakshminarayanan, and J. Snock. Evaluat- ing prediction time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10000006..99.. !0963, 2020.
[34] A. Orhan. Robustness properties of Facebook's ResNeXt WSL models. arXiv preprint 1771v:190.007640, 2019.
[35] X. Peng, B. Usman, N. Kaushik, J. Hoffman, D. Wang, and K. Saenko. VisDA: The visual domain adaptation challenge. arXiv preprint arXiv: 2017.
[36] Quiñonero Candela, M. Sugiyama, Schwaighofer, and N. Lawrence. Dataset Shift in Machine Learning. The MIT Press, 2009.
[37] B. Recht, Roelofs, L. Schmidt, and V. Shankar. Do CIFAR-10 classifiers generalize to CIFAR-10? arXiv preprint arXiv: 0045/, 2018.
[38] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla. M. Bernstein, A. Berg, and F. Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV), 2015.
[39] S. Sagawa, P. Koh, T. Hashimoto, and P. Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In International Conference on Learning Representations (ICLR), 2020.
[40] S. Schneider, E. Rusak, L. Eck, O. Bringmann, W. Brendel, and M. Bethge. Improving robustness against common corruptions by covariate shift adaptation. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
[41] H. Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of Statistical Planning and Inference (JSPI), 2000.
12

---

[42] C. Shorten and T. Khoshgoftaar. A survey on image data augmentation for deep learning. Journal of Big Data, 2019.
[43] Sivaprasad and F. Fleuret. Test time adaptation through perturbation robustness. arXiv 2021.
preprint
[44] Y. Sun, X. Wang. Z. Liu, J. Miller, A. Efros, and M. Hardt. Test-time training with self- supervision for generalization under distribution shifts. In International Conference on Machine Learning (ICML), 2020.
[45] Varsavsky, M. Orbes-Arteager C. Sudre, M. Graham, Nachev, and M. Cardoso. Test-time unsupervised domain adaptation. arXiv preprint arXiv:2010.01926 2020.
[46] D. Wang, E. Shelhamer, S. Liu, B. Olshausen, and T. Darrell. Tent: Fully test-time adaptation by entropy minimization. International Conference on Learning Representations (ICLR), 2021.
[47] G. Wilson and D. Cook. A survey of unsupervised deep domain adaptation. ACM Transactions on Intelligent Systems and Technology (TIST), 2020.
[48] E. Wong, L. Rice, and J. Kolter. Fast is better than free: Revisiting adversarial training. In International Conference on Learning Representations (ICLR), 2020.
[49] Y. Wu and K. He. Group normalization. In European Conference on Computer Vision (ECCV), 2018.
[50] S. Xie, R. Girshick, P. Dollár, 7. Tu, and K. He. Aggregated residual transformations for deep neural networks. In Conference on Computer Vision and Pattern Recognition (CVPR). 2017.
[51] D. Yin, R. Lopes, J. Shlens, E. Cubuk, and Gilmer. A Fourier perspective on model robustness in computer vision. In Advances in Neural Information Processing Systems (NeurIPS), 2019.
[52] M. Zhang, H. Marklund, A. Gupta, S. Levine, and C. Finn. Adaptive risk minimization: Learning to adapt to domain shift. In Advances in Neural Information Processing Systems (NeurlPS), 2021.
13

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 5
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes] See supplemen-
tary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Appendix A
(c) Did you report error bars (e.g., with respect to the random seed after running experi- ments multiple times)? [N/A]
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [No]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] All assets are publicly and freely
available.
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're 're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "ltsmvI149gq", "reviewer": "Reviewer_bMhD", "summary": "This paper proposed an approach that performs test-time adaptation using one test point. Most past test-time adaptation work requires accessing multiple test points under the same distribution shifts. The method first augments the query test data with different augmentation instances from AugMix, and then use the classifier to estimate the predictions. Then the whole model backbone is adapted to produce invariant predictions across augmented versions of the test data, and also optimized to be confident on the prediction, which is different from past work where only BN layer is optimized. Experiment on ImageNet-A, C, R are presented. Ablations studies on the effect of invariance and confidence are presented.", "questions": "None", "limitations": "None", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. This work performs test-time adaptation on single test point, which is more realistic than past work setup. It is a simple and effective approach.\n2. The method is evaluated over more than 4 OOD benchmark, and demonstrates improvement on ImageNet-A for the first time.\n3. The ablation study shows the effect of each component of the proposed approach.\n4. Presentation is clear.\n\nWeakness:\n1. What is the inference time the test time adaptation method spends compared with non-adaptation method and other test-time adaptation method? A Table should be provided.\n2. Missing related work: [1] uses test time adaptation on augmented input images to defend adversarial distribution shifts. \n\n\n[1] Mao et al. Adversarial Attacks are Reversible with Natural Supervision. ICCV 2021.  https://arxiv.org/abs/2103.14222", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "cy-PPJPYDx", "reviewer": "Reviewer_dZZX", "summary": "## Post rebuttal\nI'm bumping up my rating to Accept. \n\n## Original\nThis paper proposes a method for test-time adaptation for a single sample by augmenting the data point and reducing the marginal entropy over these augmented inputs. The experiments shown on Imagenet-C, -A, -R and CIFAR-10, 10-C, 10.1 show that MEMO is capable to reducing test errors with access to a single point. ", "questions": "* How does Memo fare with batches? I see single point BN results, and Tent with batch size 1 (in Table 5) but it might be interesting to see how Memo does with access to mini-batches comparable to the ones used in Tent. \n* Why do the authors use AugMix specifically? Are there any specific properties of Augmix that are useful for the proposed method, since they show in Table 6 that using standard augs gets very close ($\\approx$ 0.5\\%)?", "limitations": "Limitations of the method would be limited technical novelty, but has been more than made up through extensive experiments. Also a theoretical understanding of the method is completely lacking, though I understand this might warrant its own paper. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths:\nThe paper is clearly written, and well motivated. The ablations are shed further light on the importance of each component of the method. The results on Imagenet-A are especially impressive. \n\n### Weakness:\n\n* The method is a black-box. Why does encouraging same prediction across augmentations  necessarily leads to better results?\n* The paper is missing references to several new paper on test-time adaptation [1, 2] etc. Several improvements to Tent have been proposed, and comparison to those is needed to place the importance of the current paper in the latest literature. Additionally, a method [3] that is quite similar to the current one has not been discussed.\n* A discussion on what architectural choices lead to better results is missing i.e., improvements over Tent are small (in most cases), and all the architectures use some normalization layer. How would the authors attribute their method's successes to each kind of network layer? Would the method be effective on networks without normalization? \n* The paper can be expanded by experiments on various other shifts (domain adaptation problems), or tasks like semantic segmentation (as shown in Tent). \n\n\n[1] Test-time classifier adjustment module for model-agnostic domain generalization\n[2] MT3: Meta Test-Time Training for Self-Supervised Test-Time Adaption\n[3] Test time Adaptation through Perturbation Robustness https://openreview.net/forum?id=GbBeI5z86uD", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Xt9v0kbUeSk", "reviewer": "Reviewer_yhos", "summary": "The paper proposes a method for single-point test-time adaptation, i.e., test-time adaptation without an access of multiple i.i.d. samples in test-time, by (a) using a set of data augmentation techniques instead to build a batch to adapt, and by (b) minimizing the marginal entropy of the predictions from the batch of (augmented) samples. Experimental results on CIFAR and ImageNet, along with a diverse set of distribution shift benchmarks - CIFAR-10-C, CIFAR-10.1 and ImageNet-C/R/A and architectures covering both convolutional networks and vision transformer, show that the proposed method can outperform previous single-point adaptation methods. ", "questions": "- The paper states that MEMO adapts every parameters in the model unlike previous BN-adaptation based methods. But how does MEMO perform under BN-only adaptation? This would give a clearer comparison with Single-point BN baseline.\n- Given that MEMO uses augmentations from AugMix, I wonder how the paper handled in their evaluation for corruptions that are in common to the AugMix augmentations - some corruption types in CIFAR-10-C may share some semantics with some types of augmentations in AugMix, which means that the method thus may leveraged an additional prior knowledge of augmentations compared to baselines, i.e., TTA and Single-point BN.", "limitations": "The paper does include an adequate discussion for the limitations, but apparently does not for the potential societal impacts. ", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is clearly-written and easy-to-follow.\n- The proposed method is simple and easy-to-use.\n- The paper performs extensive experiments across diverse datasets and architectures to show the effectiveness.\n- The paper tackles a challenge that has been crucial in the literature of test-time adaptation, i.e., the requirement of batch information.\n\n**Weaknesses**\n\n- The overall gains can be marginal considering its additional computational costs: although the paper demonstrates that the proposed MEMO could outperform two main baselines, i.e., TTA and Single-point BN, the performance gap between MEMO and Single-point BN is overall quite small while MEMO has several times more inference time in my understanding.\n- The ablation study is inconclusive on identifying which part of the method is the most crucial to improve results. I think currently several important ablations are missing in the ablation study: e.g., MEMO without Single-BN adaptation, more extensive ablations on the data augmentations used, # of augmented samples used per inference.\n- Regarding the reported ablations (in Table 4), on the other hand, I found the performance of the ablation \"$+ \\ell_{\\mathrm{CE}}$\" works quite similarly with the proposed marginal entropy based loss, which means that a Tent variant is actually performs similarly to MEMO, and the actual form adaptation objective does not much matter. This can decrease the technical novelty of the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "GOc12u-35Q3", "reviewer": "Reviewer_iZWy", "summary": "- This work presents a simple, empirically-driven approach (MEMO) for test-time adaptation of arbitrary parametric deep models, given only a single test point. The approach is modular, and the authors examine it in conjunction with other relevant approaches in a wide test suite.\n- The approach centers on augmenting the given test point before evaluating the model on all augmentations, then applying an entropy minimization loss on the average of all evaluations' prediction vectors, and taking a single gradient step to update the model. The procedure is well-described in Algorithm 1 and section 3.1.\n\nThe flow of the work is to:\n- Introduced related work.\n- Introduce the MEMO procedure and motivate the associated loss function.\n- Describe auxiliary approaches that can be combined with MEMO.\n- Examine error rates across different datasets when evaluating with MEMO, other adaptation approaches, and combinations of approaches.\n- Ablate MEMO in several ways and show that both of the loss' targets (confidence and agreement) are necessary.", "questions": "Questions:\n- I have no outstanding questions regarding this work. Any such questions (e.g. questions about the effects of variation in the augmentation pools) are answered in the appendices. I believe that the authors are generally aware of the effects and limitations of their work, and communicate its scope well and honestly.\n", "limitations": "Limitations:\n- The authors argue that their method presents small, but consistent gains, which is true. They do not overstate the empirical result.\n- In the ImageNet-A problem on RVT*-small model, the MEMO approach does not outperform other adaptation methods. The authors correctly call this out.\n- The authors correctly call out and demonstrate that access to larger numbers of test points at once can improve adaptation in some settings, and argue that their method only represents SOTA gains on certain datasets _under the restriction of accessing a single test point for adaptation._\n\n- This paper's scope is largely isolated from societal impacts, and I do not believe a discussion of them is needed.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- The approach is, to my knowledge, novel, although it draws inspiration from a variety of works on data augmentation and test-time adaptation.\n- The approach is well-communicated, with Figure 1 and Algorithm 1 capturing the key details. The choice of loss function is clearly motivated in 3.1 as optimizing for confidence of prediction as well as agreement between predictions on different augmentations.\n- The approach presents impactful improvements for test-time adaptation on variants of ImageNet\twith ResNet-50, as well as on ImageNet-A on ResNext-101.\n- The ablative study split across the main paper and appendices is extensive, ablating not only each part of the loss objective (agreement and confidence), but also the choice of augmentations (Appendix B), as well as the number of augmentations sampled for evaluation (Appendix B).\n- Communication is clear.\n\nWeaknesses:\n- Improvements on variants of CIFAR-10 are slim. This is not an issue with the paper or approach, so much as an observation that MEMO is not a panacea and it is more useful in some settings than others.\n- This work is not, in my view, a new literature paradigm. It combines several established techniques (augmentation, test-time adaptation, entropy minimization) into a novel module to add onto other models and robustification approaches, in order to induce performance gains. Most of the interesting future directions this work opens (and addresses in Discussion section) are further improvements on the MEMO procedure, as opposed to broadly new ways to think about the problem of test-time adaptation.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept