Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper focus on single image test time adaptation of deep neural network for distribution shift in classification task. Prior works perform test time adaptation on a batch of images or entire test dataset to capture the distribution statistics. The authors propose to create augmented copies of a provided test input and encourage invariance across different augmentations by promoting same prediction. Authors propose to minimize marginal entropy for confident predictions across the augmented versions of the input. Improvements are shown on CIFAR-10-C, CIFAR-10.1, ImageNet-C, ImageNet-R and ImageNet-A.

main_review: Strengths:
* Motivation is clear
* Well-written and structured
* Addressing an important and challenging problem setting
* No assumptions on the availability of the entire test dataset
* A simple approach that is easy to implement
* Provided ablation study to understand the importance of the augmentations, confidence maximization and encouraging consistent predictions.
* Significant improvements are noticed on vision transformer model that are less reliant on BatchNorm layers, e.g., about 8% improvement on ImageNet-C and ImageNet-R over the baseline.

Weaknesses:
My major concerns lie on author's claims, novelty of the approach and effectiveness of the approach on CNNs.
* Novelty of the approach is limited. Prior works have utilized augmentations during test time and entropy minimization for test time adaptation independently. This work can be seen as combination of these two existing approaches for test time adaptation.
* Results on CIFAR-10 are similar or comparable to TTA.
* Individual analysis of corruptions and severities on ImageNet-C reveal that improvements are marginal and range between 0%-2%.
* Authors emphasize that invariances across the augmentation is significant but do not reflect that in the results, e.g., results in Table 3 show that major gains are obtained from confidence maximization of multiple augmentations but encouraging the invariance produce only marginal improvements. I assume similar behavior on ImageNet-C: considerable improvements are seen with confidence maximization of multiple augmentations but only marginal improvements by enforcing invariance.
* TENT produce worse results on three ImageNet benchmarks with vision transformer. Since this architecture is less reliant on BatchNorm layers, how do updating all the parameters of this model would perform with TENT?

summary_of_the_review: Given the simplicity of the approach for single image test time adaptation, its improvements on vision transformer and my concerns listed above, I marginally accept this paper. I am willing to reconsider my rating based on other reviews and authors responses.

-------- post rebuttal ---------------

The paper studies an interesting problem setting and also bring improvements in this setting, particularly on vision transformer. However, my concerns on authors claims still persist after the rebuttal and I also see that the novelty of the paper is limited. After considering the authors response and other reviewers comments, I rate this paper as borderline and keep my original score.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This paper studies the test-time model adaptation, where training data is not accessible and only one test example is available. It adapts model parameters by minimizing the output entropy on augmented test samples. Given one test sample, it first samples some augmentation operations, resulting in different augmentations. The entropy is computed on the average distribution of augmentations' outputs. Experiments on CIFAR-10 CIFAR-10.1 CIFAR-10-C, ImageNet-C, ImageNet-R, and ImageNet-A show that the proposed method effectively reduces classification errors.
main_review: Strengths
1. The proposed method is general since it doesn't need to alter the model training and supports online adaptation with one sample at a time.
2. The single objective embraces two intuitions behind: invariance across augmentations and prediction confidence.
3. The method is simple and easy to implement.

Weaknesses
1. The paper claimed that MEME could adapt only BN statistics in Section 3.2, but there seems no results presented in Section 4's tables.
2. It is unclear how the testing is conducted. Does testing happen after adapting the model over all the test samples (one epoch or several epochs)? Or do we test on a sample immediately after adapting to it? Do different methods use the same testing procedure? More details are necessary at the beginning of the experiment section.
3. According to Table 2, MEME can't beat other methods in several cases. It usually gets higher errors on datasets ImageNet-C and ImageNet-R than BN and Tent. Although it gets better results on ImageNet-A, it still has a higher error than the test-time augmentation baseline.
4. Considering the entropy minimization has been explored in prior work Tent, the main novelty is bringing invariance to augmentations into test-time adaptation.
5. If making model adaptation in deployment, the inference efficiency will suffer from forwarding multiple augmentations and updating the model weights. Any investigation on efficiency? It would be better to add some discussions on how to improve efficiency.
summary_of_the_review: Test-time model adaptation is an interesting problem, providing a new perspective to improve model robustness. The proposed method builds on intuitive assumptions that are easy to understand. There are mainly two concerns regarding novelty and effectiveness. I think the authors may need to rethink designing experiments to outstand advantages.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors study test-time adaptation to distribution shifts. They address the problem of single-sample adaptation and propose a method where they create several augmentations of a single image and average over the augmentations. They present results on different common robustness benchmarks.
main_review: I think that the general idea is interesting. It is indeed a short-coming of BatchNorm adaptation that one needs large batch sizes for this technique to really be effective. It is very relevant to have a method that works for single images. This would allow adapting models in an online-adaptation setting where there is only one image present from a certain distribution shift.

The authors should explicitly define the difference between f and p since currently, they seem to be using both interchangeably. Is p just f+softmax? Based on algorithm 1, I understand the procedure as following: The authors use one single test image and create B augmentations of it, pass B through the network, get B embeddings, calculate the mean over the embeddings, then calculate the loss based on the mean, i.e. for calculating the loss, they use p_tilde which is one number per dimension. The difference to the second ablation is that in the second ablation, they average after calculating H. Please make the distinction clearer in the text.

I appreciate the authors uploading their code. I actually went through it to better understand the marginal entropy loss function and several other things.

Comparison to TENT. The authors write “This ablation is effectively a version of Tent (Wang et al., 2021) that is suitable for our test setting.” To me, the most important difference to TENT is that the model is reset after each image while in TENT, the model continually adapts to the full ImageNet-C set. Therefore, ablation 2 is NOT equivalent to TENT and this should be made clear.

I think that the results are rather limited and several important ablations are missing. Especially, for a purely experimental paper I think more results should be presented.
1.	The most relevant difference to other gradient-based test-time approaches is that the model is reset after each new image. It has not been tested whether TENT still works if one resets the model after each batch. 
2.	It is not clear to me how many gradient steps are performed on each image. In the code, n_iter is set to one but I think it is a hyperparameter that should be tuned.
3.	How is B selected? I would appreciate a study for different values of B. In the limit case of B=1, one recovers the TENT setting, right? Currently, it is not clear to me that TENT does not work for B=1 since it has not been shown and TENT (with model resetting) actually performs equally well to MEME in Table 2 for a batch size of 64. -> Showing that TENT (+ model resetting) indeed does not work for small batch sizes and one needs this averaging over augmentations would be important.
4.	Currently, the authors only study ResNet50 models and the Vision Transformer. For the ImageNet-A results, any results on ResNet50 models are biased (see my points below) and it is not clear how to adapt vision transformers (see my points below), making the results on both architectures hard to judge. I would suggest to the authors to test other model architectures, e.g., they could test the ResNext101 models (both vanilla and the IG-3.5B variants), which would be interesting because the IG-3.5B variant could not be adapted successfully with Batch Norm adaptation. 
4.b. In the abstract, the authors write "In our experiments, we demonstrate that this approach consistently improves robust ResNet and vision transformer models". To avoid over-claiming the success of the method, I would suggest to the authors to include all relevant robust ResNet50 models, e.g., the ones from Table 1 in Schneider et al. Alternatively, the authors have to remove the claim that MEME improves upon all robust ResNet50 models and write that MEME improves upon the vanilla, the DeepAug+Augmix and the MoEx+CutMix model.
5.	Considering the second ablation point: “Conversely, as an objective that encourages confidence but not invariance…”, I find the presented results not conclusive enough to state that both confidence and invariance are important, as the difference are only 0.4 percent max. Possibly, more experiments should be performed to really stress this point. 
6.	AugMix augmentations come in different severity levels. Checking the code, it seems the authors used a severity of 1 which is the smallest severity possible. This might explain the ablation result why the difference to standard augmentations is not that big. I think varying this parameter would be important.
7. I would find it very interesting to see how MEME performs for larger batch sizes. The authors could plot a curve like Figure 1 in Schneider et al to see the effect of different batch sizes (alongside with the BN adapt curve). Ideally, the MEME curve should always be below the "BN adapt" curve. In the limes of the full dataset size as sample size, MEME should converge to the TENT result. Using the full dataset size as the batch size is obviously not possible due to GPU constraints, but one should see the asymptotic trend approaching that number. Maybe the authors could use batch sizes of [1,2,4,8,16... max_batch_size_per_GPU] for this analysis.

### Comments on achieving a new state of the art claims:
The authors claim to achieve a new state of the art on various benchmarks on several occasions. 
1. “We achieve state-of-the-art results for test shifts caused by image corruptions (ImageNet-C), renditions of common objects (ImageNet-R), and, among ResNet-50 models, adversarially chosen natural examples (ImageNet-A).”
2. “In Section 4, we demonstrate empirically that MEME consistently improves the performance of ResNet (He et al., 2016) and vision transformer (Dosovitskiy et al., 2021) models on several challenging ImageNet distribution shift benchmarks, achieving several new state-of-the-art results for these models.”
3. “In particular, as we empirically show in Section 4, MEME can be composed with prior methods for training robust models and adapting model statistics, resulting in improvements over the prior state of the art for several ImageNet distribution shift benchmarks and model architectures.”
4. “For both ImageNet-C and ImageNet-R, and for both the ResNet-50 and RVT_-small models, combining MEME with robust training techniques leads to new state of the art performance.”

When speaking about state-of-the-art results on either ImageNet-C or ImageNet-R, one can either claim (1) a new state of the art for a certain architecture or (2) a new state of the art on ImageNet-C overall. 
Considering case (1), the authors claim state-of-the-art results on ImageNet-C for a ResNet50 architecture which is wrong since both TENT and BN adaptation outperform their approach.
Considering case (2), the authors report a mCE of 40.6% on ImageNet-C and an error of 43.8% on ImageNet-R for their vision transformer. These are not state of the art overall, because for example, the Noisy Student model achieves a mCE of 28.3% on ImageNet-C and an error of 23.5% on ImageNet-R. 

The authors achieve good results for the vision transformer model, but this is not a standard architecture (yet) so judging these numbers is hard. It is in general not very clear how transformers should be adapted. In conv-nets, usually the batchnorm layers are adapted (like in TENT), but maybe in transformers, one needs to adapt other parameters. For example, [5] and [6] experiment with “adapter” and “compacter” layers, respectively, to adapt transformers. The authors also raise this point when discussing their results: “Furthermore, we find that these methods are less effective with the RVT-small model, which may indicate their sensitivity to model architecture choices since vision transformer models are comparatively less reliant on BN layers (Steiner et al., 2021).” Yes, exactly. But to me, it does not mean that TENT and BN adaptation do not work per se, but that one needs to think about which layers need to be tuned.
-> It might be that TENT would work if one used the layers suggested there, but not if one uses conventional BN adaptation.

On ImageNet-A, the Noisy Student model from [3] reports a top1 error of 16.5% which is much better than what the authors report here. Sure, the authors manage to use MEME to decrease the top1 error of a ResNet50 backbone to 89% (which is a slight improvement from 92%), but I find this result not very interesting, because it is still very bad and no one would actually use this model on ImageNet-A. In either case, their ImageNet-A results are not state of the art. In general, all ResNet50 results on ImageNet-A are very biased, because this dataset was explicitly designed to fool vanilla ResNet50 classifiers. So results reported for a ResNet50 do not generalize to other architectures. One can argue that MEME is the only method that improves upon the RVT results, but here again the issue from above arises where it is not clear how to adapt a vision transformer properly.

Table 2: “A composition of robust training via strong data augmentations and MEME adaptation achieves new state-of-the-art performance on each benchmark for ResNet-50 models.” -> This is wrong: DeepAugment+Augmix+TENT achieve lower error rates on ImageNet-R and ImageNet-C for the ResNet50 backbone. I also find the bolding of the numbers inconsistent: If the best numbers are supposed to be bold, why is MEME bold as well despite being worse than TENT? The authors write “and we list in bold
the best results from these methods which outperform the test time robustness methods.” I don’t get it: BN adaptation and TENT are also test time robustness methods?


As a summary of this point, I think the authors need to rephrase their interpretation of the results since they do not achieve SotA results neither on ImageNet-C nor on ImageNet-R nor on ImageNet-A. 

summary_of_the_review: I really like the general idea of the paper of trying to find a method where adaptation to a single test sample is possible, with resetting the model after each sample. To my knowledge, this has not been done yet and it would be an interesting contribution to the field and highly relevant for practitioners. This is actually very similar to how robustness is evaluated when model weights are frozen and inference is done on every point separately. Here, one wants to adapt to each test data point.

It is obviously a harder problem since so far, models have been allowed to adapt to the whole test set, e.g. TENT studied this setting. And indeed, single-point BN adaptation is currently the setting closest to this one. So it is fine to not claim state-of-the-art results on ImageNet-C in this hard setting. Of course, methods that are allowed to see the full test set and adapt to it will be better. Therefore, I think the authors should refrain from wanting to claim state-of-the-art results over methods that are allowed to see the full test set, since it is an unreasonable request. Methods that are allowed to see the full test set should always perform better, and if they don’t, there is likely an optimization issue like with the vision transformers.

I would suggest to the authors to pitch their paper in this setting: It is a new problem that no one has really studied so far which we propose a solution for and we ran all the necessary control experiments. That said, if one has access to many samples from the target domain, it should still be better to use all of them for adaptation. I think writing the pitch in this way should make the paper much more clear and easier to understand and put into context.

If one were to pitch the paper in this setting, one would need to test whether other test time adaptation methods work with model resetting, i.e. does TENT work for a batch size of 1 and with model resetting? Does one really need the different augmentations as claimed here?

I think that the paper cannot be accepted in the current state, but I think the general setting is very interesting and it can become a relevant contribution if the pitch is changed and more results are added. I am open to have a discussion with the authors during the rebuttal period whenever they wish to. I am happy to raise my score if my points are addressed.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper aims at increasing robustness of image classifiers against domain shift using test-time adaptation on a single-instance level. For this, the authors apply a large set of random augmentation on the respective test input and adapt the model using entropy minimization on the marginal distribution of model outputs across augmentations. The proposed method is called "Marginal Entropy Minimization with Ensembled Augmentations" (MEME). MEME is combined with prior methods for compensating domain shift and evaluated on several domain shift benchmarks.
main_review: Strength:
 * The problem of addressing domain shift is a highly significant topic
 * The proposed method can do adaptation on a per-datapoint level (in contrast to related works that require larger batches of inputs from the same target domain)
 * The experiments compare to the most related work on a large set of domain shift benchmarks and baseline models. Results are well-structured
 * The paper is very well written and structured.

Weaknesses:
 * The proposed method has little novelty since it is a straightforward combination of existing methods: test-time augmentation, test-time entropy minimization (Wang et al., 2021), and adapting BatchNorm statistics (Schneider et al. 2020). 
 * The proposed method considerably increases inference cost (by a factor of B) - this is undesirable in practice
 * As discussed below, some ablations/alternative design choices are missing

Some alternative design choice that could strengthen the paper when included in the experiments/ablation studies:
 * Equation 1 computes the average of class predictions over augmented samples. Alternatively, the average over logits could be considered. 
 * Varying the number of augmentations B in the experiments and discussing the trade-off between inference time and error
 * in general, repeating every run several times with different random seeds and report average error in the results
 * discuss why there is a "stop-gradient" in the loss $\ell_{PCE}$ and study its effect
 *  $\ell_{PCE}$ is not invariant to permutation of the batch since the second sum runs only over $j=(i+1)...B$. Moreover, this is particularly problematic since the gradient flows through the $\tilde{x}_j$-inference for j-1 terms (those terms with i < j) .Thus samples with higher index j have larger weight in the loss. A simple fix would be to use a sum over $j \neq i$.
 * since the authors use AugMix augmentations and AugMix-pretrained models, it would be natural to also add a JSD-type term to the loss at test-time that explicitly aims at encouraging invariance against augmentations.

Minor comments:
 * In the tables, it is confusing to report errors and then a reduction of the error by X percent point with +X (it should be -X since it is a reduction of error)
 * how many iterations of SGD/ADAM are performed per datapoint? I did not find this information in the paper. Is it only one? If yes: why?
summary_of_the_review: The main contribution of the paper, the method MEME consists of plugging together two existing techniques, namely test-time augmentation and test-time entropy minimization (Wang et al., 2021). This combination is straight-forward and the technical novelty of the paper is quite small. 
On the other hand, the empirical evaluation is relatively extensive and well structured . Consistent albeit small gains of the proposed method are observed. As discussed above, there are some ablations/alternative design choice that could be added to the experiments. Including those would give  a more complete picture.

In summary, I see this paper as marginally below the decision threshold, due to its limited novelty and incomplete ablations.
review_title: null

Meta-review: The reviewers agree that test-time model adaptation is an interesting problem, providing a new perspective to improve model robustness. The proposed method builds on intuitive assumptions that are easy to understand. However, there are mainly two concerns regarding novelty and effectiveness. The paper can improve in these two aspects to meet ICLR standard.
Decision: Reject