Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: $Paper$ $Summary$

This paper proposes a method to adapt a pre-trained model to a target domain, without the need to access samples from the source domain - on which the model was originally trained. The idea is to adapt layer normalization parameters at test time, by learning affine transformations. This is applied in tandem with the re-collection of the domain statistics.

$Pros$

- The paper is very well written: it is easy to understand the core idea and its applicability in the context of the broader literature. Figures and Tables are also well designed and placed.

- The method is reasonable and simple, and results are strong. As the Authors claim, it is true that the proposed approach has significantly wider applicability than UDA methods and TTT.

$Cons$

- While interesting, I believe Section 4.2 ('Target-only Domain Adaptation') would require additional experiments to properly assess how competitive the proposed approach is with respect to the state of the art of UDA. For instance, several better performing methods could be included in Table 3 (different papers from CVPR/ICLR/etc 2018-2020 achieve significantly higher results on the SVHN -> MNIST split, basically closing the gap with target models - see for example  "A DIRT-T Approach to Unsupervised Domain Adaptation" [Shu et al. ICLR 2018]). The better performing methods still need more data to train (source + target), so higher numbers for the competitors would not undermine the proposed method, but they would provide the reader with a more realistic perspective. 

- Results associated with more challenging splits should be provided; for instance, can the proposed method handle MNIST -> SVHN adaptation? This would also clarify one concern I have, which is <how good the source model should be for the method to be effective>. The MNIST -> SVHN split would help clarifying this point, since MNIST models are severely under-performing on SVHN (typically accuracy is ~30%).  

- One important baseline that is missing is "Domain Adaptation in the Absence of Source Data", [Chidlovskii et al. SIGKDD 2016]. Can the authors comment on this? It seems to me that some of the Algorithms proposed in this related work could be applied here. 

$Minor$ $points$/$suggestions$

- Is the performance of UDA-SS in Table 3 correct? The error is larger than the Source model. I am also checking at the original paper.

- The paper "On Calibration of Modern Neural Networks" [Guo et al. ICML 2017] shows that deep neural networks are poorly calibrated, as over-confident on samples they are wrong about. Since high-confidence generally implies low-entropy, their results are not aligned with Figure 1, that seems to suggest proper calibration. It would be nice to comment on this in the manuscript.  

$Review$ $summary$

I believe this is a strong paper, proposing a simple method with large applicability - hence I recommend acceptance. Still, it presents some weaknesses at this stage: I would be happy to read the Author response on these points and iterate the discussion.

---- Post-rebuttal comments----

The rebuttal and the paper revision address my concerns. I fully recommend acceptance.
review_title: A simple method with broad applicability and good results

final_score: rating: 7: Good paper, accept
review_content: 
Presents Test-time Entropy (TENT) minimization, an algorithm for adapting deep models at test time to distributionally shifted data, without requiring access to source training data. At test time, the algorithm updates batch-norm parameters (that control channel-wise normalization and transformation) to minimize predictive entropy over target data. This simple approach is found to lead to state of the art performance on various corruption benchmarks for image classification, and competitive performance on simple DIGITS recognition-based domain adaptation shifts. 

Strengths

– The approach appears very simple to implement and seems to work well, particularly on adapting to corruptions

– The paper is well-written, clearly motivated, and very easy to follow. In particular, the source free-assumption is a compelling feature of the method.

– The analysis on rank-correlation b/w change in entropy and loss, and applicability to different architectures, strengthen the claims of the paper

Weaknesses

– While it’s clear that entropy minimization is correlated with correctness, the motivation beyond updating (only) batch-norm parameters to minimize it is unclear to me. Sec 3.2 states that the reason is “stability and efficiency”, but I think a more comprehensive ablation study would validate this choice better. 

– I appreciate the fact that the method is benchmarked on multiple tasks – visual corruptions and DA. However the DA experiments/comparisons appear quite cursory. DIGITS is an easy benchmark, and the DA point of comparison (RevGrad) is not competitive with the current SoTA. Without results on other more challenging benchmarks (VisDA, DomainNet, OfficeHome, etc.) I’m not convinced of the usefulness of this method as a DA technique. 
To be clear, it would be completely fine if the method *does not* outperform prior work that uses source data or additional computation, but I think it is important to at least benchmark its performance to understand whether it is a viable DA strategy. The qualitative results on semantic segmentation in supplementary appears promising but a quantitative comparison would have been more convincing.

Additional questions / suggestions

– In Fig. 5, how is source performance measured for TENT? Is this performance on source data after applying TENT, with the updated batch-norm parameters? If so, how does TENT achieve identical performance on the source test set?

– It would be interesting to benchmark the performance of TENT for online DA – beyond not requiring source data, being able to also adapt (even reasonably well) online could be very useful.

– A more descriptive caption for Figure 2 would be helpful. Does opacity correspond to the severity of corruption?

– In Fig. 5, it would be good to break down performance of ANT by corruption type as is done for other methods

– Sec 4.2: “Tent needs less computation, but still improves with more”: Does this hold indefinitely? Does performance degrade after a certain number of epochs, or does it remain stable?

Overall comments

Interesting paper on test-time-adaptation that proposes a simple entropy-minimization based objective to update batch norm parameters, and works well on robustness benchmarks. I have concerns around the motivation behind the algorithm’s design, and its viability as a DA method, but would be willing to reevaluate based on the author response.

---- Post-rebuttal comments----

The author rebuttal + revised draft adequately addresses most of my concerns – in particular, the experiments on online adaptation and semantic segmentation are strong, and the additional context on the DA results is helpful. I would still have liked to see DA results on more challenging benchmarks but nevertheless think that the paper proposes an interesting approach and is worth accepting.

review_title: Interesting approach, needs some clarifications

final_score: rating: 8: Top 50% of accepted papers, clear accept
review_content: Summary: This paper tackles an interesting problem setting — fully test-time adaptation with only target data. The proposed method is to minimize the test-time entropy, and the loss is used to update the feature modulation layer only. The proposed method compares favorably with the state of the arts, on the ImageNet-C benchmark and unsupervised domain adaptation tasks.

Strength:
+ The problem setting is interesting and meaningful
+ The proposed method is simple, efficient, network-agnostic, and generally applicable to many tasks
+ The proposed method achieves competitive performance on the ImageNet-C benchmark and unsupervised domain adaptation tasks

Weakness:
- When comparing with UDA methods (i.e., RG and UDA-SS), I am not sure if the current setting is fair or not. I think they can use the target test set instead of the target training set during training? In this case, it should be a fair comparison with the offline adaptation. 
- I wonder if the improvement comes from modulating the features to the target domain distribution, or because the network is optimized specifically on the test data. Run online/offline adaptation on the target training set, and then directly apply this model on the target test set without optimization might be interesting.
- Figure 6 is a bit not intuitive to me. Are the authors trying to convey that “entropy reduction is correlated with classification loss reduction”? Probably a better visualization is needed.
- In Section 4.3 “Tent needs feature modulation”, I wonder where are the exact numbers that I can refer to.
- What is the message the authors try to convey in Figure 7? What does “BN brings them back” mean? I think this study is interesting and important, more discussions and insights are appreciated.

Other comments:
* I wonder when this method will fail. I guess if the target test set is very small, the proposed method might not improve over the source model too much?
* Neural networks sometimes tend to output overconfident (but wrong) predictions even when given out-of-distribution data as input. In this case, I wonder if test entropy is still a good supervision signal to use.

---- Post-rebuttal comments----

Thanks for the response. The rebuttal addresses all my concerns. I am willing to increase my score to 8 and recommend acceptance.
review_title: Interesting and useful setting.

Meta-review: The paper is proposing a test time adaptation method without modifying the training. The proposed idea is simple and effective, adapting the normalization layers using the entropy of the model predictions as a loss function. The paper presents an extensive empirical study. Paper received unanimously accept scores. It also has potential to be impactful as it is easy to apply without any strong assumption/requirement. A clear accept!
Decision: Accept (Spotlight)