Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The authors present a method named NOTE to address the online adaptation for non-iid data streams without additional annotation. It has two key components: instance-aware batch norm (IABN) and prediction-balanced reservoir sampling (PBRS). The former computes the difference between learned knowledge and the current observation (i.e., the new instances from the online data stream). The latter aims to avoid overfitting towards the non-iid streams by mimicking the iid samples with a simulated memory. They evaluate the method with sota TTA baselines on a few datasets (mainly for computer vision) and show that NOTE is effective. 
strengths_and_weaknesses: ***Strengths***


- The paper is comprehensive and has a nice presentation. It introduces the background knowledge very well and the proposed components have clear motivation. 

- The IABN and PBRS are both novel and show effectiveness over other baseline methods. 

- This paper studies a very important problem of test-time adaptation without supervision. It can be used for many real-world applications. 

***Weakness***
- In the experiments, the authors should use more tasks such as some NLP data and more popular base models like Transformer-based ones for both CV and NLP. Otherwise, it's hard to measure how the proposed methods really work in modern AI systems. 
questions: N/A
limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Test-time adaptation updates the model on the test data to improve generalization to shifted data.
While this sort of adaptation can help, most existing methods assume the test data arrives i.i.d., and in particular without temporal dependence.
In temporal settings, such as robotics or autonomous driving, this work first demonstrates the weakness of existing methods, and then proposes a fix (NOn-i.i.d TEst-time adaptation, or NOTE) that adjusts normalization statistics and maintains a reservoir/queue of test data to stabilize test-time optimization.

This fix is an extension of TENT, a test-time adaptation method that updates batch normalization layers in two steps: first, it re-estimates the normalization statistics from test batches, and second, it updates affine scale and shift parameters to minimize the entropy of model predictions.
The normalization part of the fix generalizes instance norm and batch norm (IABN), reducing to each as a special case of the hyperparameters, and in essence correct outliers w.r.t. the batch statistics by using the instance statistics instead (Sec. 3.1).
The reservoir part of the fix adopts reservoir sampling to maintain a queue that is expected to be class-balanced and time-balanced in order to simulate access to i.i.d. data in the non-i.i.d. setting.
The balancing is accomplished by making use of the model predictions as pseudo-labels, in place of the true labels used for continual learning, for prediction based reservoir sampling (PBRS).
With the reservoir samples, current test inputs can be batched with past test inputs, which stabilizes both the normalization statistics and entropy gradients for updating the affine parameters.

Experiments on standard benchmarks for dataset shift (CIFAR-10/100-C), real temporal data with synthetic shift (KITTI), and lesser-known temporal datasets (HARTH, ExtraSensory) demonstrate that existing methods degrade in the temporal setting.
Matched comparison on standard and temporal/class-ordered CIFAR-10/100-C confirm that NOTE achieves comparable accuracy in both settings, rivaling existing methods in the i.i.d. case and signifcantly improving (~50% relative error) in the temporal case.
On the real temporal data (KITTI, HARTH, ExtraSensory) NOTE still does as well as better, although the KITTI result is within the error bars, while HARTH and ExtraSensory are not well-established as benchmarks for shift.
The main components of the method, IABN and PBRS, are ablated and the sensitivity of methods to temporal dependence w.r.t. the degree of non-uniformity and batch size are analyzed.

NOTE is the first work to identify the failure of test-time adaptation on temporal data and achieves state-of-the-art accuracies compared to existing methods in this setting.
strengths_and_weaknesses: Strengths

- Well-motivated: Temporal data is ubiquitous, and many cases that may need adaptation are indeed temporal, such as autonomous driving in anomalous weather or robotics applications in poorly-controlled conditions.
  The difficulty current methods have with temporal data is neatly summarized by Fig. 2, which shows the stark contrast between the i.i.d. and non-i.i.d. settings on a simple dataset (CIFAR-10-C).
- Clear illustrations: Fig. 1 shows two varieties of temporal data, with inputs and labels, where both clearly show the type of temporal correlation higlighted by this work.
- Sound experimental design on standard datasets: The use of CIFAR-10/100-C is standard and serves as a sanity check of the method in the established setting. The application of the Dirichlet distribution for sampling more dependent and less dependent orderings of the data according to its concentration hyperparameter is appropriate and effective for sweeping across degrees of dependence.
- Strong enough results: NOTE rivals existing methods in the existing setting, and does far better in the temporal setting it focuses on.
- Baselines: The experimental comparisons include the basic method, TENT, which is extended by NOTE, alongside the recent and strongest methods such as LAME and CoTTA which were only recently published at CVPR'22.
- Consistency: The same hyperparameters, such as the threshold for IABN, are used across experiments

Weaknesses

- NOTE has to alter training by replacing BN with IABN, and therefore cannot be applied to off-the-shelf models without re-training.
  The test-time adaptation setting is in part motivated by not having access to the source/training data, so this makes NOTE less of a good fit for this particular assumption.
- There is no large-scale evaluation in terms of dataset or model. For data, one would expect results on ImageNet-C, which serves as a gold standard benchmark of robustness to natural shifts like corruptions. For models, ResNet-50 is a common choice among test-time adaptation methods, including those compared to in this work.
- How NOTE updates is not fully specified. While Sec. 3.2 explains that inputs are sampled from the reservoir and batched with test inputs, it does not detail the proportion of test inputs to reservoir inputs and the number of gradient steps among other such considerations.
- The choice of datasets is not entirely justified. HARTH and ExtraSensory are indeed temporal, but at the same time they are unfamiliar for the purpose of evaluating adaptation. Are these datasets subject to shift across the given sources and targets? It would be reassuring to measure source accuracy on source and target to evidence drops and thus the presence of shift.
  The architectures for these datasets are likewise not standard, which would be fine, except the text does not explain how to they incorporate IANB (though this presumably follows the convolution layers).
questions: Questions

- How does NOTE perform on ImageNet-C in the i.i.d. and non-i.i.d. settings with ResNet-18 (and ideally ResNet-50)? For the non-i.i.d. setting, consider the same proposed dirichlet ordering, or simply ordering the data by classes.
- What is the full impact of PBRS on the amount of computation for prediction and adaptation? In particular, how much more time is needed per test input? Given that it is used for batching inputs, it seems that forward and backward would require linearly more time in the size of the reservoir. To put it another way, does using a reservoir of size 64 for a test input mean that inference is 64x times slower or at least requires that many more forward passes?
- Is it possible to adopt IABN during testing without altering training? That is, can NOTE operate in a fully test-time manner without having to (re-)train the model with IABN?
- For the real-world datasets, what is the source accuracy on the source data? This is important to report for measuring adaptation so that the severity of the shift and how much it is mitigated can be gauged. As presented, it is hard to know if HARTH and ExtraSensory are hard problems, and if the source model is not already doing quite well.
- (Minor) How are predictions made on the first samples before the reservoir is filled? Is adaptation deferred until the reservoir is full?

Other Feedback

- Are the KITTI, HARTH, and ExtraSensory datasets "real distributions with domain shift" (as in Sec. 4.2)? KITTI-Rain is real data, with a synthetic shift. HARTH and ExtraSensory are both real data, but the shift is not evident.
- Please clarify the hyperparameters for IABN in Equation 5 and the definition of $\psi$ that follows it. $k$ is often used for integer values, but in this case it seems to be a real-valued weight. Consider other notation, such as $alpha$, and introducing this weighting in its own sentence following Equation 5.
- Figure 3 tries to cover a lot, and is less accessible as a result. Part (3) on PBRS does not communicate a lot, as it just shows updated statistics, so consider dropping it in favor of focusing more on the IABN update. Consider more labeling of part (1) to express how IABN corrects BN or not.
- In Figure 3 (2), is the time arrow reversed? I would expect the latest sample $x_t$ to be new while the discarded sample $x_i$ should be old.
- To report failures in the temporal setting, consider highlighting results that are worse than the source model with special formatting. For example, these could be underlined, or typset in red.
limitations: The limitations covered are thoughtful, in particular when it comes to the potential to amplify bias in testing data. The reliance on batch normalization layers, at least as empirically examined, is honestly discussed. While this discussion is adequate, it would be further improved by underlining the need to alter training, in substituting BN with IABN, as other methods are agnostic to training (like TENT and BN).
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: Temporal correlated data violates the i.i.d. assumption and the existing test-time adaptation methods are prone to overfitting under high correlated test data. This work proposed two components, IABN and PBRS, to tackle the temporal correlation. IABN only updates BN parameters when test data distribution is sufficiently different and PBRS resamples data samples for calculating test data batch statistics. Experiments are carried out on simulated temporal correlated data and real datasets.
strengths_and_weaknesses: Strength:

1. Temporal correlation is a practical concern in test-time training. Addressing the challenges in TTA for temporal correlated data is important.
 
2. Experiments are extensive, covering both visual and audio modalities.

Weakness:

3. The improvement on KITTI is very marginal (as shown in Tab. 2). KITTI is the real temporally correlated dataset that motivates this work but the improvement is not significant to support the claims.

4. For temporally correlated test-time adaptation task, CIFAR10-C/100-C can hardly simulate the temporal correlation. How does the temporally correlated streams look like? It is important to give illustrations or analysis of why this synthesized data stream presents temporal correlation.

5. Adjusting Batchnorm has limitation to certain backbone networks. For example, ViT (transformer) does not have batchnorm layers, how does this approach apply to transformer backbones where BN is missing?

questions: Evaluation and discussion on how to generalize to transformer based backbones would further improve the quality of this work.

For streaming test, it would be interesting to report the real-time accuracy which illustrates the real adaptation power in test-on-stream manner.
limitations: Yes.
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: Test-time training is an emerging and promising approach for building robust machine learning models to distribute shifts. This paper claims that most existing TTA methods assume that the test samples come from i.i.d distribution, but it does not hold for many practical applications of TTA, such as self-driving, human activity recognition, etc. This paper then proposes two new techniques to adapt the model under non-i.i.d setup, namely Instance-Aware Batch Normalization and Prediction-Balanced Reservoir Sampling (PBRS). Experimental results on synthesized non-i.i.d setup and realistic non-i.i.d setup show that the proposed method significantly outperformed the existing methods. 

strengths_and_weaknesses: ---
**Strengths**

(1) Test-time adaptation is a timely and important topic. This paper pointed out important issues of existing TTA methods. 

(2) This paper validates the effectiveness of the proposed method on both synthetic and realistic benchmark tasks of non-i.i.d adaptation setup. 

---
**Weaknesses**

Overall, I think this paper provides a good contribution for the test-time adaptation research, with extensive experiments. However, I found several important details lacking in current manuscripts, including the detail of the proposed method (especially IABN part), explanation about why the proposed method should work better than prior works, and empirical validations.  See questions for the detailed comments. 

Minor comments/questions:
- Equation (4), the left hand side should be u_c/L?
- Lines 282-284, can you explain why LAME does not works well on i.i.d setup, or even worse than non-i.i.d setup?

questions: (1) Why do we need equation (4) and equation (5)? What happens if we always u_{b, c} to compute outputs? 

(2) What is the final output of IABN layer?

(3) Regarding the exponential moving average presented in 3.2, do you stop gradient for the past mean and sigma? 

(4) Why do we need lines 9-17 for PBRS? In other words, why don’t you randomly discard one instance in memory even if y_t \in L? 

(5) In table 1, why does the value for Source differ between i.i.d setup and non-i.i.d setup? 

(6) If I correctly understood the paper, PBRS can be incorporated with any prior studies, e.g., TENT, PL. Why did you choose to combine it with IABN? In other words, is IABN itself perform better than existing methods? 

limitations: This paper adequately addressed the limitation and potential negative social impact.
review_title: null

Meta-review: metareview: The paper proposed two test-time adaptation methods a) instance-aware batch normalization and b) prediction-balanced reservoir sampling and used these to show that the proposed method is better in the non-iid setting.

The reviewers found this to be an important problem the experiments generally convincing. Reviewers objected the choice of dataset (not commonly used to evaluate adaptation) and baseline models (not state of the art models) and the effect size. In the end all reviewers found the results strong enough and voted to accept.
confidence: Certain
Decision: Accept