Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: - This work presents a simple, empirically-driven approach (MEMO) for test-time adaptation of arbitrary parametric deep models, given only a single test point. The approach is modular, and the authors examine it in conjunction with other relevant approaches in a wide test suite.
- The approach centers on augmenting the given test point before evaluating the model on all augmentations, then applying an entropy minimization loss on the average of all evaluations' prediction vectors, and taking a single gradient step to update the model. The procedure is well-described in Algorithm 1 and section 3.1.

The flow of the work is to:
- Introduced related work.
- Introduce the MEMO procedure and motivate the associated loss function.
- Describe auxiliary approaches that can be combined with MEMO.
- Examine error rates across different datasets when evaluating with MEMO, other adaptation approaches, and combinations of approaches.
- Ablate MEMO in several ways and show that both of the loss' targets (confidence and agreement) are necessary.
strengths_and_weaknesses: Strengths:
- The approach is, to my knowledge, novel, although it draws inspiration from a variety of works on data augmentation and test-time adaptation.
- The approach is well-communicated, with Figure 1 and Algorithm 1 capturing the key details. The choice of loss function is clearly motivated in 3.1 as optimizing for confidence of prediction as well as agreement between predictions on different augmentations.
- The approach presents impactful improvements for test-time adaptation on variants of ImageNet	with ResNet-50, as well as on ImageNet-A on ResNext-101.
- The ablative study split across the main paper and appendices is extensive, ablating not only each part of the loss objective (agreement and confidence), but also the choice of augmentations (Appendix B), as well as the number of augmentations sampled for evaluation (Appendix B).
- Communication is clear.

Weaknesses:
- Improvements on variants of CIFAR-10 are slim. This is not an issue with the paper or approach, so much as an observation that MEMO is not a panacea and it is more useful in some settings than others.
- This work is not, in my view, a new literature paradigm. It combines several established techniques (augmentation, test-time adaptation, entropy minimization) into a novel module to add onto other models and robustification approaches, in order to induce performance gains. Most of the interesting future directions this work opens (and addresses in Discussion section) are further improvements on the MEMO procedure, as opposed to broadly new ways to think about the problem of test-time adaptation.
questions: Questions:
- I have no outstanding questions regarding this work. Any such questions (e.g. questions about the effects of variation in the augmentation pools) are answered in the appendices. I believe that the authors are generally aware of the effects and limitations of their work, and communicate its scope well and honestly.

limitations: Limitations:
- The authors argue that their method presents small, but consistent gains, which is true. They do not overstate the empirical result.
- In the ImageNet-A problem on RVT*-small model, the MEMO approach does not outperform other adaptation methods. The authors correctly call this out.
- The authors correctly call out and demonstrate that access to larger numbers of test points at once can improve adaptation in some settings, and argue that their method only represents SOTA gains on certain datasets _under the restriction of accessing a single test point for adaptation._

- This paper's scope is largely isolated from societal impacts, and I do not believe a discussion of them is needed.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: ## Post rebuttal
I'm bumping up my rating to Accept. 

## Original
This paper proposes a method for test-time adaptation for a single sample by augmenting the data point and reducing the marginal entropy over these augmented inputs. The experiments shown on Imagenet-C, -A, -R and CIFAR-10, 10-C, 10.1 show that MEMO is capable to reducing test errors with access to a single point. 
strengths_and_weaknesses: ### Strengths:
The paper is clearly written, and well motivated. The ablations are shed further light on the importance of each component of the method. The results on Imagenet-A are especially impressive. 

### Weakness:

* The method is a black-box. Why does encouraging same prediction across augmentations  necessarily leads to better results?
* The paper is missing references to several new paper on test-time adaptation [1, 2] etc. Several improvements to Tent have been proposed, and comparison to those is needed to place the importance of the current paper in the latest literature. Additionally, a method [3] that is quite similar to the current one has not been discussed.
* A discussion on what architectural choices lead to better results is missing i.e., improvements over Tent are small (in most cases), and all the architectures use some normalization layer. How would the authors attribute their method's successes to each kind of network layer? Would the method be effective on networks without normalization? 
* The paper can be expanded by experiments on various other shifts (domain adaptation problems), or tasks like semantic segmentation (as shown in Tent). 


[1] Test-time classifier adjustment module for model-agnostic domain generalization
[2] MT3: Meta Test-Time Training for Self-Supervised Test-Time Adaption
[3] Test time Adaptation through Perturbation Robustness https://openreview.net/forum?id=GbBeI5z86uD
questions: * How does Memo fare with batches? I see single point BN results, and Tent with batch size 1 (in Table 5) but it might be interesting to see how Memo does with access to mini-batches comparable to the ones used in Tent. 
* Why do the authors use AugMix specifically? Are there any specific properties of Augmix that are useful for the proposed method, since they show in Table 6 that using standard augs gets very close ($\approx$ 0.5\%)?
limitations: Limitations of the method would be limited technical novelty, but has been more than made up through extensive experiments. Also a theoretical understanding of the method is completely lacking, though I understand this might warrant its own paper.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The paper proposes a method for single-point test-time adaptation, i.e., test-time adaptation without an access of multiple i.i.d. samples in test-time, by (a) using a set of data augmentation techniques instead to build a batch to adapt, and by (b) minimizing the marginal entropy of the predictions from the batch of (augmented) samples. Experimental results on CIFAR and ImageNet, along with a diverse set of distribution shift benchmarks - CIFAR-10-C, CIFAR-10.1 and ImageNet-C/R/A and architectures covering both convolutional networks and vision transformer, show that the proposed method can outperform previous single-point adaptation methods. 
strengths_and_weaknesses: **Strengths**

- The paper is clearly-written and easy-to-follow.
- The proposed method is simple and easy-to-use.
- The paper performs extensive experiments across diverse datasets and architectures to show the effectiveness.
- The paper tackles a challenge that has been crucial in the literature of test-time adaptation, i.e., the requirement of batch information.

**Weaknesses**

- The overall gains can be marginal considering its additional computational costs: although the paper demonstrates that the proposed MEMO could outperform two main baselines, i.e., TTA and Single-point BN, the performance gap between MEMO and Single-point BN is overall quite small while MEMO has several times more inference time in my understanding.
- The ablation study is inconclusive on identifying which part of the method is the most crucial to improve results. I think currently several important ablations are missing in the ablation study: e.g., MEMO without Single-BN adaptation, more extensive ablations on the data augmentations used, # of augmented samples used per inference.
- Regarding the reported ablations (in Table 4), on the other hand, I found the performance of the ablation "$+ \ell_{\mathrm{CE}}$" works quite similarly with the proposed marginal entropy based loss, which means that a Tent variant is actually performs similarly to MEMO, and the actual form adaptation objective does not much matter. This can decrease the technical novelty of the paper.
questions: - The paper states that MEMO adapts every parameters in the model unlike previous BN-adaptation based methods. But how does MEMO perform under BN-only adaptation? This would give a clearer comparison with Single-point BN baseline.
- Given that MEMO uses augmentations from AugMix, I wonder how the paper handled in their evaluation for corruptions that are in common to the AugMix augmentations - some corruption types in CIFAR-10-C may share some semantics with some types of augmentations in AugMix, which means that the method thus may leveraged an additional prior knowledge of augmentations compared to baselines, i.e., TTA and Single-point BN.
limitations: The paper does include an adequate discussion for the limitations, but apparently does not for the potential societal impacts.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposed an approach that performs test-time adaptation using one test point. Most past test-time adaptation work requires accessing multiple test points under the same distribution shifts. The method first augments the query test data with different augmentation instances from AugMix, and then use the classifier to estimate the predictions. Then the whole model backbone is adapted to produce invariant predictions across augmented versions of the test data, and also optimized to be confident on the prediction, which is different from past work where only BN layer is optimized. Experiment on ImageNet-A, C, R are presented. Ablations studies on the effect of invariance and confidence are presented.
strengths_and_weaknesses: Strengths:
1. This work performs test-time adaptation on single test point, which is more realistic than past work setup. It is a simple and effective approach.
2. The method is evaluated over more than 4 OOD benchmark, and demonstrates improvement on ImageNet-A for the first time.
3. The ablation study shows the effect of each component of the proposed approach.
4. Presentation is clear.

Weakness:
1. What is the inference time the test time adaptation method spends compared with non-adaptation method and other test-time adaptation method? A Table should be provided.
2. Missing related work: [1] uses test time adaptation on augmented input images to defend adversarial distribution shifts. 


[1] Mao et al. Adversarial Attacks are Reversible with Natural Supervision. ICCV 2021.  https://arxiv.org/abs/2103.14222
questions: None
limitations: None
review_title: null

Meta-review: metareview: The authors propose MEMO, a heuristic for adapting classifiers at inference time based on a single test point. The adaptation procedure consists of constructing augmented versions of the test point and optimizing an objective that encourages "confidence and consensus" among the predictions on the augmented versions of the test point. The reviewers found the idea to be interesting, the experiments compelling, and were satisfied with the author responses during the discussion period. Thus, I am recommending acceptance. 
confidence: Certain
Decision: Accept