Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper addresses a problem arising from the generality of source-based conventional batch normalization (CBN) or test-based batch normalization (TBN), with the former biasing the architecture to source distribution and the latter suffering from inaccuracies for small test batch sizes. The paper proposes combining CBN and TBN, and exposing a fraction of different layers in the architecture to TBN and CBN, thereby identifying those layers which need not be as sensitive to the test distribution. The results show promising performance of the proposed method, TTN, on the CIFAR and some DG benchmarks across several batch sizes.

strength_and_weaknesses: Strengths:

The technique is clear and fairly straightforward to implement.
Motivation describing pitfalls of CBN and TBN, and the need for better online test-time adaptation under flexible batch sizes are described well

Weaknesses:

It is unclear how the results would generalize across architectures. What will the results look like for transformers, which are currently ubiquitous in several applications? Will some tokens in a layer be adapted to the test set and the rest to the source? What would this mean in practice?

In Figure 2, why cannot alpha be learnt or adapted online? If it can be, then these methods should be compared with prompting (freezing certain layers will be analogous to using source-based normalization, while the prompting layers will be adapted to test set). It is unclear why an alpha learnt offline and frozen will continue to yield optimal results across a range of different test-time distributions.

The significance of results from Table 6 is not clear. These appear to be tested specifically on batch size 2 where TBN is  known to falter, but the results on only a few datasets appear significant. Similarly for the corruption benchmark, with a batch size of 1 or 2 (which is a fairly common real world scenario - data streaming and not evaluated in batches), TTN seems to be beat by one of the baselines. A better understanding and explanation of this is required. Is this because of optimizing the alphas offline?

Finally, what are the restrictions of how far the data distribution that the TTN model is optimized on and the real streaming test time data are? The results appear to indicate that there will be scenarios and assumptions mainly surrounding the dataset that alpha is optimized upon.

clarity,_quality,_novelty_and_reproducibility: The paper has good clarity and enough details are provided to replicate the implementation. The novelty is marginal as this appears to be a fairly straightforward combination of two existing methods.
summary_of_the_review: Overall, I would rate the paper as below the acceptance threshold. While the method appears to show promise in certain settings, I would urge the authors to consider a wider selection of baseline architectures. Also, the results are not convincing enough about its use in a wider array of datasets or real-world scenarios (see the weakness section above).
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a test-time batch normalization method to tackle domain shifts. The core idea is to learn interpolating vectors to combine running statistics and test-time batch statistics in batch normalization layers. The authors also propose to use a gradient distance score to initialize and regularize the interpolating vectors during training. The proposed method achieves promising results on several image classification and segmentation benchmarks. The authors also conduct extensive ablation studies to validate the design choices.
strength_and_weaknesses: **Strength**
- The proposed method is simple yet effective
- The proposed method can handle various different test settings and achieve promising performance
- The authors conduct detailed ablation studies to help the readers to better understand the method

**Weakness**
1. Missing references
- [A1][A2] are two recent state-of-the-art test-time adaptation methods, it would be good to add and compare with them
- The idea of learning interpolating weight is quite similar to [A4]. I know [A4] should be counted as concurrent work and the authors do not have to cite it. But I suggest the authors add and compare with it, to provide a better context to the readers in this area. 
2. More comparisons
- I think (Schneider et al. 2020) is a strong baseline and should be included.
- The two state-of-the-art contrastive learning-based methods should also be compared
- Maybe add more comparisons for the semantic segmentation experiments as well. For example, [A3][A4].
3. Others.
- Which TBN method is used in Figure 1? According to my experience, at least (Schneider et al. 2020) should have a reasonable performance (not significantly worse than CBN). Please add more details here.
- The authors propose a mixed domain evaluation setting, where each batch contains a different domain. However, it is also possible that a single batch contains multiple domains of data. I wonder whether the proposed method can handle this case. [A3][A4] can both handle this and I think the authors can have some study or discussion about this setting.
- How to select the $\lambda$ in Eqn. (6) to balance the two loss terms?
- It is quite interesting to see that the proposed method actually performs worse than non-learning-based normalization methods when the batch size is very small (2 or 1). Can the authors comment on that? Will a dynamic batch size in the post-training stage improve the performance in this case?
- Why the normalization methods are not shown in Table 2 and Table 3? This looks like normalization methods cannot be used in these settings, but actually they can.
- While adding the proposed method improves upon the two optimization-based methods in Table 3, the results are even worse than applying the proposed method alone. This may weaken the proposed method and probably the authors can move it to the appendix.

**Reference**
- [A1] Liu et al. TTT++: When Does Self-Supervised Test-Time Training Fail or Thrive? NeurIPS 2021
- [A2] Chen et al. Contrastive Test-time Adaptation. CVPR 2022
- [A3] Khurana et al. SITA: Single Image Test-time Adaptation.
- [A4] Zou et al. Learning Instance-Specific Adaptation for Cross-Domain Segmentation. ECCV 2022

clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is easy to read and understand.
- Quality: Good.
- Novelty: Interesting and somewhat novel idea.
summary_of_the_review: Overall, I think this paper tackles an interesting and important task (test-time adaptation), and the proposed method is well-motivated and effective. I tend to accept this paper. But I suggest the authors add some comparisons and discussion with the missing references I mentioned above.

-----

Post-rebuttal

After reading other reviews and the response, I decided to increase my rating.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a test-time normalization (TTN) method that combines the normalization statistics of source domain and target domain (during test), by adjusting the importance between them according to the domain-shift sensitivity of each BN layer. The motivation of the proposed method is clear, and the comprehensive experiments conducted in this paper demonstrate the effectiveness of the proposed method for test-time adaptation, particularly in the scenes for small batch size adaptation.  
strength_and_weaknesses: 
**Strengths:**

1.The motivation is clear, and the effectiveness of the proposed method makes sense, based on the previous works.

2. This paper provides a comprehensive discussion to the related work, as to me. 

3. This paper conducts comprehensive experiments, and the experiments demonstrate the proposed method is empirically successful for test-time adaptation, particularly in the scenes for small batch size adaptation.  

**Weaknesses:** 

1.The technical novelty and contributions are incremental. Test-time adaptation using BN is recently widely investigated in previous works, as stated in the related work. Furthermore, the idea of combining the statics of BN from the source domain and targeted domain is not new from previous work. Besides, the method (calculate the domain shift sensitivity) to obtain the prior of $\alpha$ is from Choi et al. (2022), the optimization (for the interpolating weight $\alpha$) using cross entropy is also shown in several previous work (even though not specific to the interpolating weight $\alpha$). I donot well recognize new point for this paper. 

2. It should provide the code (including script with hyper-parameters), since this paper conducts comprehensive experiments and the configs of these experiments are complicated. Even though this paper provides details for setup of experiments in the supplementary material, I believe it is not easy to reproduce the results based on these descriptions. 

clarity,_quality,_novelty_and_reproducibility: The clarity is clear, the quality is overall good, and the novelty is somewhat incremental. I believe the experiments can be somewhat reproduced based on the descriptions of this paper.
summary_of_the_review: An empirical successful method for test-time adaptation but with incremental technical novelty and contribution. I am slightly positive to this paper, considering its empirical effectiveness, but not confident to accept it.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: **Summary:**

This paper presents a new test-time normalization (TTN) method that combines the training statistics and the test-time statistics via the importance between them. To obtain the Prior, the authors first augment the training samples as domain-shift ones, and then calculate the importance via the gradient distance score between the clean samples and domain-shift ones. The experimental results on CIFAR-C and ImageNet-C demonstrate the effectiveness of the proposed method.
strength_and_weaknesses: **Positive Points:**

 - The authors propose to adjust the balanced parameters between the training statistics and the test-time statistics based on the (augmented) training data.

 - The authors introduce a gradient distance score to calculate the balanced parameters $\alpha$, which is technically sound.

 - The authors consider plenty of test-time scenarios, such as various batch sizes, stationary, continuously changing, and mixed domain adaptation.

 **Negative Points:**

  - The authors obtain Prior based on the pair (clean sample x, and domain-shifted x'). And the domain-shifted x' greatly depends on the augmentation type and applying order. In this case, if the domain-shifted x' has an extensive domain gap from the test-time samples, the Prior would be estimated inaccurately, leading to poor TTA performance. Could the authors provide more discussion and explanations about this?

 - In the paragraph of "Optimize $\alpha$", the authors state "To simulate distribution shift, we use the augmented training data". Here, the authors have a strong assumption that the test-time shifted type is very similar to the augmented type. However, in my opinion, this assumption often does not hold in practice. More explanations are required.
 
 - I suggest the authors can provide an ablation study on the augmentation type, which is very crucial for the proposed method. I wonder if there is an augmentations combination that works well for all the corruption-shifted test-time samples? If no, the proposed method may be difficult for real-world applications.

 - Most experimental results are reported on CIFAR-C. I suggest the authors can report the main results on ImageNet-C, which will be more convincing. I think the computational and time cost on ImageNet-C is similar to that on CIFAR-C. In Table 9, I find the authors do not report the results of "Tent+TTN" and "SWR+TTN". But I think these two are crucial results.
clarity,_quality,_novelty_and_reproducibility: The paper is well-writen and technically sound. If the authors can address my concerns, I would raise my scoring to accept.
summary_of_the_review: I vote for reject since the proposed method has a strong assumption on the test samples, which may not be hold in practice.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: Thank you for your submission to ICLR.  The paper presents a new approach to normalization that interpolates between traditional batch normalization (using average statistics at test time) with test-time-applied batch normalization (which requires entire batches of data at test time, thus referred to as the tranductive batch norm, TBN, in this paper).

There was some disagreement about this paper from the reviewers, but ultimately I am more persuaded by the positive perspectives on this paper.  In particular, I think that the benefits and relative simplicity of the approach are compelling, and that most of the unaddressed negative comments were fairly generic (regarding why not evaluating on different architectures, etc), that may or may not really be relevant here (it's unclear, e.g., if the approach really needs to consider something that applies to architectures that don't contain BN layers).

The authors presented a vast number of experiments in response to reviewer concerns, including some fairly impressive results on larger-scale problems.  While these presentations border on presenting new results (typically not the goal of the rebuttal period), my general feeling is that they still represent compelling evidence for the method, and shouldn't be dismissed.  Overall, my main recommendation, then, is that the authors synthesize these many new results into the paper, perhaps even changing some of the focus given these rather impressive new results.
justification_for_why_not_higher_score:  I think the paper is still somewhat borderline, and several of the new results that are most compelling were only mentioned in the discussion (and thus likely were somewhat add-on experiments done after the original submission).  The core idea also seems to be a nice but still incremental method in a long line of test time adaptation approaches, which is great to include but likely not sufficient for a spotlight.
justification_for_why_not_lower_score:  While there were some negative scores for this paper, I think the additional experiments and overall simplicity and performance of the method make a very strong case for inclusion here.
Decision: Accept: poster