Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: Test-time model adaptation updates model parameters during inference in order to reduce generalization error on shifted data. Continual test-time adaptation, the setting of this work, does so for varying shifts without knowledge of when the shift itself changes over time. The purpose of this work is to improve the efficiency of inference and adaptation to require less computation time and memory. In particular, the proposed memory economical continual test adaptation (MECTA) approach extends and tunes entropy minimization methods (Tent and EATA) so that their gradient updates do not need as much memory as the naive implementation of backpropagation during testing. This work highlights the memory usage of caching forward activations in particular, which is measured to be 5x the memory for inference with at ImageNet scale (ResNet-50, specifically). Efficiency is improved while maintaining accuracy by reducing batch size, extending normalization statistics to moving estimates rather than batch-wise estimates, dropping channels by a kind of test-time pruning or partial caching that is sparser than the standard backward pass, and gating updates by thresholding the change in layer-wise statistics. Any layer that is gated does not need its cache for updates, and so memory can be saved for the gated layer and the following layers. MECTA improves accuracy over Tent and EATA when memory is constrained (Table 1) while nearly achieving the same accuracy as EATA when memory is not limited (Table 2). These results are shown for the standard choices of datasets: ImageNet-C and CIFAR-10/100-C with common baseline models like ResNet-50.
strength_and_weaknesses: Strengths

- MECTA does reduce the peak memory usage of Tent and EATA while still improving accuracy on shifted data in the continual adaptation setting. MECTA allows for 4x larger batches and consequently improves the accuracy of EATA by more than 10 points. The accuracy of Tent is also improved, but Tent was not designed for continual adaptation in the same way as EATA, so this improvement is less surprising and significant.
- The stochastic dropping of channel-wise gradients ("Sparse gradients via stochastically-pruned caches") is a novel scheme to reduce gradient memory as far as I am aware. Dropping gradients instead of pruning channels has the advantage of preserving the forward computation and accuracy. (However, see weaknesses for a discussion of whether or not this can truly save memory given how deep networks are implemented.)
- The layer-wise gating of updates is more granular than the input-wise gating of EATA, and so more computation can potentially be spared. In particular, MECTA can stop caching as soon as a layer is gated, while EATA requires an entropy prediction and therefore a full forward (and the corresponding memory) for each input to decide its gating.
- MECTA has a regularizing side-effect on Tent that helps prevent forgetting, as shown by improved accuracy on _unshifted_ data after adaptation. This follows from its sparser updates and moving estimates.
- The implementation of MECTA is encapsulated in a variation on the batch normalization layer, which makes it easy to adopt by swapping it into a network with batch normalization.
- The experiments and in particular the analysis experiments in Figure 2 show that MECTA strictly dominates adaptation by BN for accuracy and memory usage. (However, without memory limits Tent and EATA can still do as well or better than MECTA, so this gain comes at the cost of specialization to this reduced computation setting.)

Weaknesses

- This work lacks simple computational baselines that would help prove the necessity and impact of MECTA's contributions. In particular, it does not investigate (1) updating fewer layers or (2) checkpointing gradients.
  - Fewer layers: this work ignores the option of simply updating fewer layers during testing, by for instance only updating the last/deepest layers of the network. Only updating deeper layers removes the need for caching of all earlier layers.
  - Gradient checkpointing: this work ignores the option of recomputation (a.k.a. rematerialization) by simply discarding forward caches and recomputing forward as needed for backward, and by doing so misses a simple baseline. This is now a common feature of deep learning frameworks, as is provided by the gradient checkpointing utility in PyTorch, for example.
- A prior paper on continual adaptation is missing from the related work: CoTTA at CVPR'22. CoTTA is a contemporary method from the same time as EATA, and so likewise deserves inclusion in the related work on continual adaptation. In the submission, it is only mentioned and disqualified for its computational cost in the experiments (Section 5), which is not appropriate.
- The proposed channel-sparse gradients may not save time or memory in practice. Most frameworks only support dense gradients for inputs and parameters, so zeroing out a particular channel may not alter the computation performed.
The analysis experiments or "Qualitative Studies" of Section 5.2 are done with a subset of the common corruptions. These fidings may or may not generalize to the full set, so it would be better to do them with the full set, for thoroughness and comparability with other results.

clarity,_quality,_novelty_and_reproducibility: Clarity:

- The exposition clearly identifies the dimensions controlling memory usage—(B)atch size, (C)hannels, and (L)ayers—and adequately explains how each is reduced by the proposed method. The core technical content of the paper is clear enough.
- Proofreading is needed throughout including in the abstract. The meaning can be determined, but the writing errors do make reading more effortful.
- The main contributions of this work could be better highlighted and named. MECTA norm is not a new norm, but follows the use of EMA by Yang et al. 2022 (cited) and Schneider et al. 2020 (uncited in the method section, though cited in related work). The point of calling it a normalization layer seems more to indicate the desired interface to the method, which is to incorporate each part into a substitute layer for batch normalization.

Quality:

- The claims of memory usage are grounded and measured by considering peak memory usage and its different components during inference and adaptation (Sec. 1, cache measurements in the main results of Tables 1 & 2).
- The chosen benchmarks and experimental design are standard and established by prior work such as EATA, so the results can be understood, and they can be compared with existing papers without additional effort.
- The analysis experiments (Section 5.2) check key properties such as the use of the adaptive statistics threshold auto $\beta$, and the results support the design choices of MECTA.


Novelty:

- This work is not the first to emphasize computational efficiency during test-time adaptation. Tent made a first step by only requiring one forward and backward per point, and EATA further considered when and when not to update. Nevertheless, this work provides the first detailed empirical study of memory usage in the context of test-time adaptation.
- The masking of gradients during backward is related to forward masking like Dropout and DropConnect, but nevertheless it is different. - To the best of my knowledge this is a new use of stochastic masking during optimization. It is a simple trick, but it does have a use in controlling memory usage, and it serves its purpose in the experiments.

Reproducibility:

- This work should be reproducible. It makes use of common datasets in an already-defined experimental setting, the method section (Sec. 4) is sufficiently detailed, and the code will be released.


Miscellaneous Feedback

- For Table 4, the "K, k" notation for old and new batches is hard to parse. Consider alternative letters like O for number of Old batches and N for number of New batches.

summary_of_the_review: The deployment of test-time adaptation does require efficiency so that the updates made during inference do not delay predictions too much. Even more seriously, the adaptation computation needs to be feasible at all by fitting in device constraints such as memory, which is the constraint addressed by this work. Precise measurements of memory are provided, and the proposed MECTA offers several techniques that combine to reduce peak memory usage, at least in principle. The reduction of batch size and the gating of layer updates should reduce memory in practice also. On the other hand, the pruning/masking of gradient channels may or may not reduce memory depending on the implementation, so this requires clarification. While the experiments show the effect of the proposed techniques, they do not cover simple baselines, such as updating fewer layers or applying gradient checkpointing.

The amount of memory saved can be as much as ~70% of the original usage for a ResNet-50, but the lack of computational baselines makes it unclear how much this amount matters.

Questions:

- How much memory is saved and accuracy maintained by simply updating different numbers of layers in the network, starting with the deepest and then adding more shallow layers? Please Consider only a single layer, say 10% of the layers, and 50% of the layers.
- What is the memory/time trade-off of test-time adaptation with gradient checkpointing when compared to MECTA? In particular, what is this trade-off when limited by the memory constraints applied to Table 1?
- Does the channel sparsity of the gradient truly achieve reduced computation in practice, that is on a GPU with CUDA, or is there only a potential reduction?
- Does the dynamic cache truly reduce peak memory usage, or only average memory usage? The analysis results of Figure 3 suggest that the peak usage remains high, at the beginning of each shift. Please confirm the effect of MECTA on the peak usage.

**Update following Response & Discussion**: The response addressed the questions and weaknesses identified by this review adequately, but not totally, and on the balance this deserves a recommendation that sides with acceptance. I have raised the score to borderline accept accordingly. The results from the response for simpler computational baselines justify the need for MECTA, and the further details provided by the response and revision make it more informative. MECTA is worth considering for acceptance because of its more dynamic and memory-efficient computation for adaptation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: Continual test-time model adaptation (CTA) recently draws researchers' attention since trained models may encounter dynamically-changing test-time environments. This work first revealed the limitations of prior arts in memory efficiency which could be a critical obstacle for applications of CTA to memory-limited edge devices and proposed a novel algorithm that can significantly reduce the memory overhead at test time. The proposed algorithm enables test-time adaptation using smaller batches, and sparse layer-and-channel caches without significantly losing accuracy. The authors empirically demonstrated the outstanding accuracy and memory efficiency of the proposed method.
strength_and_weaknesses: Strength:

1. The investigated problem is significant to real IoT scenarios. The authors reveal the overlooked low memory efficiency of traditional CTA methods, which is critical for applications on resource-constrained edge devices. Especially when considering the Internet of Things, the memory-limited tiny edge devices cannot afford 4Gb+ memory consumption for test-time adaptation as demonstrated by Fig 1.

2. The technical part is novel. The proposed method is motivated by Eq. (3) that the parameter-efficient training (only updating BN layers) still brings in large memory caches for back-propagation. Based on the motivation, the authors provide intuitive solutions that can effectively adapt models without large memory consumption in three dimensions.

    - For reducing batch sizes, the proposed method can stabilize the training using an adaptive memorization rate (beta) which is parameter-free and intuitive for the dynamic statistical shifts. The intuition is well supported by prior observations that the BN statistics differ by domain.

    - For reducing channels in the cache, the random dropping and on-demand training can avoid forgetting resembling implicit regularization when the momentum of SGD or Adam can make up the missing gradients.

    - The layer-wise training is intuitive as the adaptation typically is on demand of domain shift. 

3. The intuition of these methods are well supported by experiments. For instance, the use of BN-based distribution divergence as the indicator is supported by the experiment in Fig 3. The forgetting mitigation of MECTA is supported by the benchmarks on Tent.

4. I appreciate the authors designed the shift-accuracy evaluation which can reveal the significant impact of accumulating batches in the transition of domains. Except for the plausible results of MECTA, the experiment itself could remind other researchers to carefully handle the domain shift at test time instead of focusing on in-domain average accuracy.

Weakness:

1. The authors should discuss the connection between the proposed adaptation methods and existing parameter-efficient fine-tuning works. For example, the low-rank adaptation of transformers [A].

2. The authors only consider the ResNet and should discuss how the method can be extended to other model architectures.

[A] LoRA: Low-Rank Adaptation of Large Language Models. Edward J. Hu*, Yelong Shen*, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen

clarity,_quality,_novelty_and_reproducibility: Clarity: The work provides a clear motivation by visualizing the huge memory consumption in Figure 1 which is hard to adapt to small edge devices. Then the method development is clearly based on the bottleneck revealed by Eq (3) in terms of the batch size, the channel number, and the network depth.

Quality: The major claims including efficiency and adaptation accuracy have been well demonstrated by experiments. Specifically, the authors compare the model performance under the same cache constraint where MECTA significantly improves the accuracy of baselines. The benchmarks also show that MECTA mitigates catastrophic forgetting in Tent even without sample selection (like EATA). Without memory constraint, MECTA can greatly reduce memory consumption yielding comparable accuracy as the state-of-the-art baseline.

And the effects of each component in the algorithms are well explored. For example, layer-wise training complies with the non-uniform forgetting preference by layers (Fig. 3), and adaptive memorization can improve the per-domain accuracy and mitigate accuracy drops on shift. The new-designed shift accuracy precisely reveals how the accumulated memorization (in BN) affects the accuracy of domain shift. The ablation study in terms of accuracy-memory trade-off also helps understand how each component improves the frontiers.

Novelty: The paper considers a rather novel challenge: reduce memory consumption at test-time adaptation. The proposed method introduces a parameter-free memorization mechanism, layer-wise training strategy, and random cache pruning, which are novel in the scope of continual test-time adaptation. The authors also defend the technical novelty by comparing the technique to traditional memory footprint reduction, at the end of Page 5.

Reproducibility: The method is enclosed into one layer, as stated in Alg. 1. The major steps include computing distribution divergence, moving average, randomly dropping cache channels, and cache on demand. These steps are elementary and could be easily implemented. But I still appreciate it if the authors can publish codes (upon acceptance).

summary_of_the_review: The paper revealed and addressed an overlooked yet an important problem in continual test-time adaptation. Most claims are well supported by experiments and the methods are intuitive and well-explained. The experiments are detailed and clear. Thus, I recommend acceptance.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes MECTA in order to improve out-of-distribution model accuracy via computation-efficient online test-time gradient descents in a memory economic manner. The key idea behind MECTA is to reduce batch sizes, adopt an adaptive normalization layer to maintain stable and accurate predictions, and stop the back-propagation caching heuristically. The networks are first pruned to reduce the computation and memory overheads in optimization and the parameters are then recovered afterward to avoid forgetting. MECTA can be seamlessly plugged into state-of-the-art CTA algorithms at negligible overhead on computation and memory. On three datasets, CIFAR10, CIFAR100, and ImageNet, MECTA improves the accuracy by at least 8.5% with constrained memory and significantly reduces the memory cots of ResNet50 on ImageNet by at least 70% without sacrificing accuracy. 
strength_and_weaknesses: Strength:

1. The authors initiate a pilot study on the memory efficiency and reveal the main bottleneck is on the intermediate results cached for back-propagation, even if only few parameters need to be updated in the state-of-the-art computation-efficient solution (EATA).
2. The proposed MECTA is very well motivated. MECTA can drastically improve the memory efficiency of gradient-based CTA. 
3. The organization and writing are very clear. The authors did a good job in reviewing all the related work, and summarizing the corresponding advantages and disadvantages.
4. The proposed MECTA is backed by very impressive empirical results. Given limited memory constraints, MECTA improves the Tent and EATA by 8.5 − 73% accuracy on CIFAR10-C, CIFAR100-C, and ImageNet-C datasets. 

Weakness:

1. In addition to the current related work section, I highly recommend a discussion on model adaptation from big pre-trained models on device, and explain why it’s not a viable direction.
2. Can the authors report the computation complexity in terms of FLOPs?
3. I understand that all the three dimensions (Reduce B, Reduce C, Dynamic L) play quite important roles in MECTA, I also want to know which dimension is the dominant one and why.

clarity,_quality,_novelty_and_reproducibility: Clarity: Most parts of the paper are clearly written and explained. Minor concerns on clarity:
* The forget gate is not clearly stated. The authors may mention the term in Alg. 1.
* In Table 1, EATA is not always the best method on all datasets. On CIFAR100, Tent outperforms EATA. An explanation should be provided.
* In Page 9, the sentence before (B.1) does not fit into the content. The authors may explain this.

Quality: The studied problem is interesting and practical, none of the prior works investigated it before. This paper is technically solid. The algorithm design is delicate. Both efficiency and accuracy during continual test-time adaptation are well backed by compelling results.

Novelty: The initiated study on the memory efficiency of continual test-time adaptation is a novel problem. The proposed methodology also seems novel. 

Reproducibility: The algorithm pseudo codes are enclosed in the main body. Implementation and hyper-parameters details are provided in Appendix B.1. The datasets and baselines codes are all public available online and properly specified in the paper. 

summary_of_the_review: Overall, this paper is technically strong and novel. It solves an important problem in continual test-time adaptation with solid methodology and sound experiments.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The state-of-the-art adaptations improve out-of-distribution model accuracy via computation-efficient online test-time gradient descents but meanwhile cost about times of memory versus the inference, even if only a small portion of parameters are updated. To address this issue, the authors provide a novel solution called MECTA to drastically improve the memory efficiency of gradient-based CTA. MECTA can improve the memory efficiency of different CTA methods. The simple norm layer structure is ready to be plugged into various networks to replace batch-normalization. The MECTA Norm layer also enables the stop and restart of model adaptation without unused or absent caches for unwanted or on-demand back-propagation. To avoid forgetting due to removing parameters, pruning is conducted on cache data for back-propagation rather than forwarding and can greatly reduce memory consumption. Compelling results show that MECTA can maintain comparable performance to full back-propagation methods while significantly reducing the dynamic and maximal cache overheads. 
strength_and_weaknesses: Strength:

1. The paper provides sufficient insights and motivations on continual test-time adaptation with memory efficiency.

2. The proposed methodology seems effective. On ImageNet dataset trained by ResNet50, it significantly reduces memory costs by at least 70% without sacrificing accuracy. 

3. The proposed MECTA is efficient and can be seamlessly plugged into SOTA CTA algorithms at negligible overhead on computation and memory. 

Weakness:

1. How this method can be extended to other normalizations? Could you elaborate more on this with examples and experiments?

2. More discussions on the potential applications of MECTA, in reality, are expected to better highlight its significance.

3. Table 1 is a bit hard to interpret, maybe highlighting the dataset names would be better.

4. The choices for the backbone network used in the experiments should be explained. Experiments on more network backbones (in particular large networks) would be interesting. 

clarity,_quality,_novelty_and_reproducibility: Clarity: The paper is mostly well-written with a clear structure. The problem is well-motivated. The method is well explained.

Quality: The paper is of high quality, which can be reflected by good writing, solid problem motivation, simple but effective methodology formulation, and sound results.

Novelty: Both the problem and method are somewhat new.

Reproducibility: Given the pseudo codes in Alg. 1, and details on implementation and hyper-parameters, reproduction seems to be doable. 
summary_of_the_review: More could be discussed about the implications of their results, but overall, this is a solid paper tackling an important problem. The proposed approach is interesting and the technique is sound. The claims are well supported by impressive empirical results.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This paper studies the problem of lifelong/continual test-time adaptation and specifically looks at methods that perform an online gradient descent at test-time. Such methods tend to be memory-intensive. The paper proposes several strategies to yield significant memory savings.

In the initial reviews, this paper received highly positive and detailed reviews from 3 of the 4 reviewers. The 4th reviewer also gave a detailed review but expressed some concerns about the lack of comparisons with simple baselines. During the rebuttal period, the authors conducted and reported these experiments as well, after which the 4th reviewer also raised the score to 6.

There is a clear consensus among the reviewer for acceptance. The paper addresses an important problem at the intersection of lifelong/continual learning and test-time adaptation, proposes novel techniques, and has extensive experimental evaluation. Therefore, I recommend the paper for acceptance.
justification_for_why_not_higher_score: The reason a rating of Accept (poster) is justified. The paper is addressing an important problem and the reviewers have a consensus about its acceptance. However, as also pointed out in the reviews, the paper isn't the first to look at the problem of memory-efficient test-time adaptation; prior works such as TENT and EATA have also looked at this issue. Therefore, as considering the various other points made by the reviewers in their reviews and discussion, it was decided to not give a higher score than Accept (poster).
justification_for_why_not_lower_score: The paper definitely is above the bar for acceptance.
Decision: Accept: poster