MECTA: Memory-Economic Continual Test-Time Model Adaptation
Authors: Junyuan Hong
Lingjuan Lyu
Jiayu Zhou
Michael Spranger
Venue: ICLR 2023 poster (2023)
Keywords: continual test-time adaptation
memory efficiency

Abstract: Continual Test-time Adaptation (CTA) is a promising art to secure accuracy gains in continually-changing environments. The state-of-the-art adaptations improve out-of-distribution model accuracy via computation-efficient online test-time gradient descents but meanwhile cost about times of memory versus the inference, even if only a small portion of parameters are updated. Such high memory consumption of CTA substantially impedes wide applications of advanced CTA on memory-constrained devices. In this paper, we provide a novel solution, dubbed MECTA, to drastically improve the memory efficiency of gradient-based CTA. Our profiling shows that the major memory overhead comes from the intermediate cache for back-propagation, which scales by the batch size, channel, and layer number. Therefore, we propose to reduce batch sizes, adopt an adaptive normalization layer to maintain stable and accurate predictions, and stop the back-propagation caching heuristically. On the other hand, we prune the networks to reduce the computation and memory overheads in optimization and recover the parameters afterward to avoid forgetting. The proposed MECTA is efficient and can be seamlessly plugged into state-of-the-art CTA algorithms at negligible overhead on computation and memory. On three datasets, CIFAR10, CIFAR100, and ImageNet, MECTA improves the accuracy by at least 6% with constrained memory and significantly reduces the memory costs of ResNet50 on ImageNet by at least 70% with comparable accuracy. Our codes can be accessed at https://github.com/SonyAI/MECTA.

Published as a conference paper at ICLR 2023
MECTA: MEMORY-ECONOMIC CONTINUAL TEST- TIME MODEL ADAPTATION
Junyuan Hong" ; Lingjuan Jiayu Zhou', ; Spranger?
Michigan State University, "Sony Al [hongju12, jiayuz @msu. edu, [1 [Lingjuan, 1v, michael spranger.re..ee (sony com
ABSTRACT
Continual Test-time Adaptation (CTA) is a promising art secure accuracy gains in continually-changing environments. The state-of-the-art adaptations improve out- of-distribution model accuracy via computation-effiiien online test-time gradient descents but meanwhile cost about times of memory versus the inference, even if only a small portion of parameters are updated. Such high memory consumption of CTA substantially impedes wide applications of advanced CTA memory- constrained devices. In this paper, we provide a novel solution, dubbed MECTA, drastically improve the memory efficiency of gradient-based CTA. Our profiling shows that the major memory overhead comes from the intermediate cache for back- propagation, which scales by the batch size, channel, and layer number. Therefore, we propose to reduce batch sizes, adopt an adaptive normalization layer to maintain stable and accurate predictions, and stop the back-propagation caching heuristically. On the other hand, we prune the networks to reduce the computation and memory overheads in optimization and recover the parameters afterward avoid forgetting. The proposed MECTA is efficient and can be seamlessly plugged into state-of-the- art CTA algorithms at negligible overhead on computation and memory. On three datasets, CIFARIO. 10, CIFAR 100, and ImageNet, MECTA improves the accuracy by at least 6% with constrained memory and significantly reduces the memory costs of ResNet50 on ImageNet by at least 70% with comparable accuracy. Our codes can be accessed at https: / / thub. com/ SonyA /MECTA.
INTRODUCTION
Many machine-leariing applications require deploying well-trained deep neural networks from a large dataset to out-of-distribution (OOD) data and gynamicaly--hhangig environments, for example, unseen data variations (Dong et al., 2022; Liu et al., 2023) or corruptions caused by weather changes (Hendrycks & Dietterich, 2019; Koh et al., 9021). Hence, the recent efforts aim to tackle this emerging research challenge via continual test-time adaptation (CTA). The unsupervised, resource- constrained, and dynamic test-time environments in CTA make it challenging learning problem and call for self-supervised, efficient and stable solution. Decent examples include Tent (Wang et al., 2021) and EATA (Niu et al., 2022). Early in 2017, Li et al. found that updating natch-normalizatioo (BN) layers with all test-time data without any training greatly improved the model OOD) performance. Recently, Tent (Wang al., 2021) significantly improved the test-time performance by minimizing the prediction entropy in an efficient manner where only a few parameters are updated. More recently, EATA (Niu et al., 2022) improved sample efficiency and evade catastrophic forgetting of the in-distribution data.
While Tent and EATA had achieved impressive gains OOD accuracy via online optimization, such optimizations are accompanied by large memory consumption and are prohibitive in many real-world CTA applications. Since many devices are only designed for on-device inference rather than training. memory-limite devices, like small sensors, cannot afford CTA algorithms. In Fig. 1, we demonstrate that Tent/EATA adaptation of ResNet50 (He al., 2016) with a batch size of 64 (the default setting in Tent) costs more than 5 times of memory in model. backward as the standard inference on ImageNet-C (Hendrycks & Dietterich, 2019). The large peak memory consumption makes EATA or
Work was done during the internship Sony AI. Correspondence Lingjuan Lyu

---

Published as a conference paper at ICLR 2023
Memory-ecnnoicc continual adaptation via MECTA of ResNet50 w/ 16-szee batches
model. init
156 Mb
Forward Bodkward
Baseline Dense cache
Ours
optimizer,
+55 MD
Cache for informnco
model. forwelforwwrd
backwardd
Mb
MECTA Norm
[C]
(B)
loss, bockward
+11.6 Ml
optiatizer stop +0 Mb
MECTA Norm
(L)
Cecta inlermnce
-197
formand
Conv
Continual adaptation via Tent/EATA ResNet50 64 batches
Figure I: Demonstration of incremental memory footprints brought by each operation on ImageNet and illustration of the proposed MECTA method, which reduces the cache size of gradient-baad based adaptation. During forwarding, the MECTA Norm (B) stabilizes the normalization-ssttisse via shift-aware moving-averaa aveerae from small batches, (C) randomly drops caches by channel admitting sparse gradient descent without knowing gradient in advance, and (L) maintains caches only for layers on demand of training.
estimation
Tent impossible to be adopted on edge devices, for example, the popular board computrr,, Raspberry Pi with 1 Gb RAM, and old-generation of smartphones (Ignatov et al., 2018).
Observing the bottleneck on model. ba clacwware rd, a straightforward solution could be reducing batch sizes and model scales (including the number of channels and layers), but there are several obstacles to maintaining model performance simultaneously. First, a large batch size is essential for adaptation (Yang et al., 2022). Second, the amount of information extracted by deep and wide model is desired for modeling distributionnally-rooust semantic features (Hendrycks et al., 2020a).
In this paper, tackle the aforementioned challenges by proposing novel approach called Memory- Economic Continual Test-time Adaptation (MECTA). As illustrated in Fig. 1, our method is enclosed into a simple normalization layer. MECTA Norm, to reduce the three dimensions of intermediate caches: batch, channel and layer sizes. (B) MECTA Norm accumulates distribution knowledge from streaming mini-batches and is stable small batches and shifts between domains using shift-aware forget gate. (C) Resembling sparse-gradien descents, we introduce test-time pruning that stochastically removes channels of cached intermediate results without knowing gradient magnitudes. (L) The forget gate also guides the layer adaptation: if the layer distribution gap is sufficiently small, then the layer will be excluded from memory- intensive training. Our contributions are follows.
New Problem: We initiate the study on the memory efficiency of continual test-time adaptation, revealing the substantial obstacle in practice.
New Method: We propose novel method that improves the memory efficiency of different CTA methods. The simple norm layer structure is ready to be plugged into various networks to replace nichh-oormalizatonn. The MECTA Norm layer also enables us to stop and restart model adaptation without unused or absent caches for unwanted or on-demand back sack-propagation. Without forgetting due to removing parameters, our pruning is conducted on cache data for back-propagation rather than forwarding and can greatly reduce memory consumption.
Compelling Results: Our method maintains comparable performance full back-propagation methods while significantly reducing the dynamic and maximal cache overheads. Given limited memory constraints, our method improves the Tent and EATA by 8.5 73% accuracy on CIFARIO-C. CIFAR100-C, and ImageNet-C datasets.
2 RELATED WORKS
Test-time Adaptation (TTA) aims to improve model accuracy on Dut-of-Distribution (OOD) data by adapting models using unlabeled test data. In comparison, traditional learning algorithms train models robustly, e.g., distributionally robust neural networks (Sagawa et al., 2020) or adversarial training (Deng et al., 2023), in order to generalize to OOD data. Early examples for TTA include the test-time training (TTT) (Sun et al., 2020) and its variant (Liu et al., 2021) which jointly train a source model via both supervised and self-supervised objectives, and then adapt the self-supervised objectives at test time. Adaptive risk minimization (Zhang et al., 2021), contextual
via
2

---

Published as a conference paper at ICLR 2023
meta-learning, conditional neural process (Garnelo et al., 2018) train models that can be adapted using inferred context information from test data. Though plausible, these methods are in need of redesign of the training process. Thus, they are less applicable for many off-the-shelf models pre-trained on datasets that are too big to attain by a small device or are prohibited from unauthorized sharing. To address the problem, training-agnostic adaptation methods are proposed recently, through adapting aatch-normalization statistics (Nado et al., 2021: Schneider et al., 2020; Khurana et al., 2022), minimizing unsupervised entropy losses (Wang et al., 2021: Goyal et al., 2022), maximizing prediction consistency over multiple augmentations (Zhang et al., 2022), and classifier adjustment (Iwasawa & Matsuo, 2021). Though these methods are effective, their applicability limited to a constant test environment in contrast real scenarios.
Efficient Continual Test-time Adaptation (CTA) considers the test scenario with dynamically changing rather than static environments and calls for efficient adaptation in place and in time. Wang et al. initiated the continual setting on computer vision by augmented supervision, which however has large computation hurdle due to multiple times of inference on one sample. The BN-based approaches are efficient alternatives, which only need to update statistics (Nado et al., 2021; Schneider et al., 2020). Yet, it has been shown in recent research that removing covariate shift by BN is not enough for reaching the supervised performance (Wang et al., 2021; Niu et al., 2022). In comparison. Tent is more effective and efficient with one-time propppropngttion per sample (Wang et al., 2021). Later, Niu et al. (2022) improved the sample efficiency via selective proo-ppppgation. Despite the advance of CTA in computation efficiency, the memory efficiency is overlooked regardless its importance in on-device adaptation. Plausible directions are on-device sparse (continual) learning (Wang et al., 2022b; Mallya et al., 2018; Mallya Lazebnik, 2018) or fine-tuning (Jiang et al., 2022; Cai et al., 2020), but how to conduct memory-efficiee learning without labels is still unclear. In this paper, we inspect the gradient computation in parameter-efficien adaptation and disclose the main bottleneck for striking the balance between memory consumption of feature eaches and learning effectiveness. Based on the analysis, we propose a novel solution to reduce caches by batch, channel and layer reductions without sacrificing performance.
3 PROBLEM FORMULATION
Let P(x)) denote the data distribution sampled from the set of distributions P, and Po is the training distribution. We assume that a distributions in P
is either identical to or is biased from Po lot. A neural network model fo parameterized by 0 is pre-trained on the training set Po by 0g mingge P.t))( (=))
where c(x) is an oracle function for ground-truth labels and i is loss function defined over the sample x and model parameter 0. One example of the loss is the cross-entropy loss: ff (f()) c(x)) (()))))))(
- E, c(x); log fo(x); + log >, exp fo(x);; given logits from 0-governed model [x).. With the pre-trained model, Tent and EATA continually adapt the model via recursive updating.
0:- = Optimizezes,
1.-1). P
(1)
). where Optimize(;,") represents a generic optimization algorithm minimizing the first variable given 0 initialized by Q.- Without labels, the entropy function (ff(()))= =
for step E
resembles the cross-entropy loss with self-supervisions in Tent.
Parameter-efficient adaptation. As the efficiency is critical at test-time, we focus on the state-of- the-art efficient solution to the CTA: EATA and Tent, both of which adopt the one-step gradient descent the optimization strategy in Eq. (1). Thus, Optimizegee, (EH, 0c-1) = f n 20 EH given a learning rate 17. To efficiently train models, we constrain the parameter space in a subspace of the original one, denoted as O, θ C O. We follow Tent and EATA to make the parameters trainable in the aatch-normalization layers (BN layer).
Now, we give a brief introduction to the batch- natch-normalization layer (Ioffe & Szegedy, 2015). Let the input to the layer / be batch of features denoted as x' in the real space of dimension B x Cd x H's x WI, where B is the batch size, C¹ is the number of channels, H' and Wi are the height and width. We use [N] denote the set (1, ,N.. For (n, j. k) E [B] x (C*] x (H7) [W]] and small constant cg. the BN layer is defined as two sequential channel-wise operations,
2. A (normalization), d. = 422 + b/ (affine),
(2)
Vot+ea
3

---

Published as a conference paper at ICLR 2023
where n4, of are the mean and variance of wf in channel i, respectively. The output tensor is also called activation. For Tent and EATA, only the affine parameters, ? and 4, are trainable.
4 PROPOSED METHOD
In this section, we elaborate on the proposed method that improves the memory efficiency of gradient-based adaptation methods. First, straightforward derivations show that the intermediate representations ought to be stored for computing the gradients of affine layers and therefore forge huge memory overhead. Suppose the loss on the n-th sample is l. Based on Eq. (2), the gradient w.r.t. the i-th channel affine weight 7/4 is
ae Oy!
B ΣΣΣ
W
Oe, da bi.k "n,5.,.'
(3)
Therefore, to compute the gradient, each BN layer has to store the normalized representations z (cache) of the dimension B x C¹ x W x H at forwarding until 0l./0a is available. For an L- layer network, the inference-only memory consumption of affine layers is Rywd = (a..e B x C x W x H In comparison, the corresponding intermediate memory for back-propagation is accumulated by Rowd Σl= B C¹ x W H' Riwd- To reduce the memory overhead, straightforward idea is reduce B, C², and L by dropping corresponding entries in {2/M_ M==1* respectively. Since dropping entries in {2')L,, will vanish the corresponding gradients, the strategy will easily break down the learning if not carefully handled. Below, we will discuss the obstacles and solutions for accuracy-secur cache reduction by B, C², and L dimensions, respectively.
(Reduce B) Adaptive statistic estimation on dynamic distributions. As a sufficient number of samples essential for accurate statistic estimation (μ and o²) per BN layer, reducing samples in batch will bias the statistics for normalization. In standard or test-time training, exponential moving average (EMA) has been widely used to mitigate the bias by memorizing streamed batches (Yang et al., 2022; Chiley et al., 2019; Liao et al., 2016). To preserves the property of gradient on calibrated statistics, we follow Yang et al. and Schneider al. implement the EMA normalization test time. Let 6 be the composed tuple, [μ, o), for mean and variance in a BN layer. Φe denotes the running statistics iteration and @ is the one from the t-th batch. The EMA statistics are
Φ = (1 - 3) 3
(4)
where the parameter 3 E [0, 1] governs the memory length and therefore works as the forger gate. small 3 endorses the model with long-term memory, otherwise short-term memory.
Traditionally, 3 constant at running time, which however cannot accommodate the estimation to the dynamic distributions P.. Intuitively, when model is stably running in single domain, e.g., P. Rs P., 3 should be small to keep as many data points as possible to support accurate statistic estimation. In contrast, when the distribution is shifting, c.g., P. + Pt-1. 3 should be large for avoiding the mixture two distinct statistics. Complying with the intuition, we introduce a forgetting gate to calibrate the 3 adaptively, 3r = hi(dh-1, )z). where h(.,.) captures the distributional shifts. We consider non-parametric heuristic definition of h((;)) as follows.
(5) where ((,,-) is properly-defined distance function measuring distribution shifts. The KL divergence, ([0illl5) is defined as log o log of + 2 (o²² + (#1 p2) assuming two Gaussian distributions parameterized by Φ1 and d2,, respectively. The distance function is inspired by Li et al. (2017), where the authors showed that Φi and ø2 have larger KL divergence (based on a Gaussian assumption) if they are from distinct domains. In addition, 89 is estimated layer by layer, as the distribution of different layers will shift to different degrees. Intuitively, when the (I - 1)-th layers well aligned after calibrating d/. the deep layers should be aligned better.
B₁ = 1 - -D) D(01-1,0.) = Σ. KL(ó.-1,ll0u.)) + K ((
(Reduce C') Sparse gradients via stochastically-pruat -pruned caches. It is easy to see that dropping the caches the channel in will vanish the corresponding gradient the channel and may leave
4

---

Published as a conference paper at ICLR 2023
the corresponding affine parameter underfitted. Thus, trivially pruning channels would cast serious issues, especially when some channels are critical for OOD generalization (Sehwag et al., 2020). But, it is hard to predict which gradient is such important not to be pruned before it is computed. Therefore an efficient pruning strategy without depending on back-propagation is desired. For this purpose, we propose unconditioned pruning strategy by repeatedly generating stochastic mask M per iteration such that x 100% entries of the mask are zeros and the rest are ones, Given the input tensor to the afline layer, we mask the tensor by En,iJ,A Mizn,i.i.k for caches in Eq. (3).
Our pruning strategy has multi-fold merits. Since forwarding is not influenced, the prediction can be done still on the full size of the network preserving a high quality of semantic features. Meanwhile, pruning lowers memory usage significantly with much smaller intermediate caches and gradients. The recomputed masks per iteration resembles a progressive learning paradigm and the momentum technique in modern optimizers, like SGD or Adam, can impute the missing gradients. Moreover, the approach mitigates catastrophic forgetting since only a subset of affine weights are updated and the low-magnitude parameters are not updated. Specifically, given gradients gt, the model difference ||0, foll = |Σ, sell < O 99 I) will be reduced with some zeroed [[9.]]. which can be viewed as implicit anti norteforgetting regularization in EATA. Later, we empirically demonstrate the regularization effect in mitigating the forgetting in regularization -reet Tent.
(Dynamic L) Train layers demand. Most test-time adaptations will last for long time in single environment. For example, an auto-driving car will keep running in sunny weather for long daytime, producing thousands of images. Continually adapting the model to the same environment for an overly-long time will not improve models continually but waste resources. Thus, we propose to stop the pack-propagation if the optimization converges and we restart on demand of adaptation. The principle is that the adaptation is demanded when data distribution cerrrbutionuunddmmann fundamentally shifts. Recalling that we already measure the distributional shift by Eq. (5), we reuse the metric to guide the adaptation. Specifically, we use a threshold 3n to make the decision. If 3 I 6n. then the z' is cached for back-po prek--ropaggtion. Because of the layer-wise decision, any layer can halt training to save lot of memory early before the network is fully executed or the optimization of all layers converge. This technique can introduce dynamic memory allocation, benefiting multi-task mobile systems. For example, the extra memory freed by the training can be used for other apps the smartphone.
Finally, we summarize the proposed method in Algorithm I, where our method includes three hyper- parameters to trade off accuracy and memory. Notably, our method is fully enclosed in MECTA Norm layer, which can be easily embedded into a network, like the widely-used ResNet (He et al., 2016), enhance the memory efficiency at test-time adaptation.
Algorithm 1 Meeoory-Econone Continual Test-time Adaptation (MECTA)
Input: A model fa with L MECTA Norm layers, the total number of iterations T, pruning rate q. batch size B and layer cache threshold 8m.
1: for iteration E {1..... ................................ T) do
2: Initiate intermediate cache: Z =
3: for MECTA Norm layer / I.... do
4: Get layer input z':
5:
Compute the current batch statistics Ø6: Compute forget gate Bi. by Eq. (5) and the statistics of by Eq. (4); Stable statistics with reduced Compute at and z' by Eq. (2) using ø!:
6:
7:
8:
if 3 > 3h. then
Dynamic L b Reduce C
9:
Randomly remove q 100% of channels in of and cache Z U (z)];
10:
Output a' to the next layer;
11:
Compute loss and back back-propagate gradients with cache Z to update parameters;
Critical difference to prior work on memory footprint reduction. Previous attempts reduce footprint (Wang et al., 2022b; Yuan et al., 2021) focused on parameter sparsity and reduce the overhead of storing model parameters and gradients. However, the overhead of parameters or gradients is relatively small in comparison to the caches for backwarding, shown in Fig. 1. Instead, we focus on the large overheads of caches and our method is ready to work with the traditional parameter sparsity. For this purpose, channel pruning and on-demand layer training resembles the gradient sparsity (Garg & Khandekar, 2009) or coordinate descent (Wright, 2015), and the two tactics implicitly prune the gradients before pack-propag avoiding large caches. Other than gradient
5

---

Published as a conference paper at ICLR 2023
sparsity, gradient checkpointing (GC) Chen et al. (2016) is more general way to reduce memory footprint. GC only caches the imputs of segments of layers and recompute intermediate features on demand. The optimal memory reduction is about the 1/vLL of the original cost. Both GC and our method can greatly reduce the memory, but our method is favored because MECTA can reduce the memory on demand, MECTA is more computation efficient.
5 EXPERIMENTS
Datasets and pre-trained models. To evaluate the OOD generalization of models, we adopt three mmage-llassffication datasets: the CIFARIO-C, CIFAR100-C (Krizhevsky, 2009) and ImageNet- C (Deng et al., 2009) following previous arts (Niu et al., 2022). All the datasets are processed with 15 kinds of corruptions from 4 main categories (noise, blur, weather, and digital ones) with the highest severity level 5, which are widely used as the robustness benchmark for deep learning (Hendrycks & Dietterich, 2019). Without specifications, we use the ResNexx129-32 4d pre-trained by Aug- Mix (Hendrycks et al., 2020b) for CIFARIO-C and CIFARI00-C, and the ResNet50 pre-trained by Hendrycks et al. (2020a) for ImageNet-C, which has compelling robust performance the bench- mark learderboard (Croce et al., 2021).
Baselines. Our method is ready to replace match-normalization in adaptable networks for bet- ter OOD generalization. (1) CTA. Therefore, we plug the proposed MECTA into two backbone methods: Efficient Anti-forgetting Test-time Adaptation (EATA) (Niu et al., 2022) and Test-time entropy minimization (Tent) (Sun et al., 2020). We also include the simplest BN statistic adapta- tion (BN) (Schneider et al., 2020) which is the most memory- and computation-efit -efficiinn test-time adaptation only updating BN statistics without gradients. Tent improves the model accuracy via self-supervision and constrains the parameter updating to affine layers. EATA is the state-of-the-ar efficient solution that balances computation efficiency and accuracy well. We plug MECTA into them to show the potential improvements in accuracy, efficiency, and generality on methods with advanced principles. Though other methods are also applicable to the problem, they typically lack advantages in efficiency, thus out of our scope. (2) Memory reduction. Gradient checkpointing (GC) Chen et al. (2016) is general method for reducing memory costs in training. which trades in computation for memory and is applicable with any sequential structure. We apply GC to ResNet where we treat each block (including several conv-bn-relu layers) as the minimal unit and segment ResNet into vT. parts. Evaluation setup. We conduct experiments on the lifelong setting, where the batches of data come in data shall be stored for memorization. Similarly,
streaming manner. For privacy considerations, the types of corruption also arrive sequentially and are agnostic to the model. Thus, models have to be adapted online to gain higher accuracy on streaming data and environments. Such setup can simulate a challenging yet realistic scenario, admitting high practical values.
Evaluation metrics. We adopt two metrics to evaluate the effectiveness and memory efficiency of different methods. (1) Accuracy (%) is evaluated on all samples under the same corruption with the highest level 5 by default. (2) Cache size (Mb). Since the memory of model parameters and optimizers is always constant or linearly dependent on the size of intermediate variables, we focus on the tensor size of intermediate variables. Specifically, we sum up the memory consumption of the tensor for affine layers, i.e. the cache 2 in Eq. (3). For the ease of memory accountant, we eclude other memory costs that are the same among different algorithms.
More details regarding hyper-parameters and implementation can be found in the appendix.
5.1 BENCHMARKS ON OOD PERFORMANCE
Comparison under the same cache constraint. We evaluate the accuracy with constraint on the cache size. For this purpose, we let the gradient-free methods use a large batch size of 128 and a gradient-based method (Tent and EATA) select small batch size from [4, 8, 16, 32} such that the latter has a smaller maximal cache size than the former does. In Table 1, we report the per-domain accuracy in continual test-time adaptation in the column order, and we calculate the cache size on average and on maximum along the sequential adaptation. We have two main observations on MECTA. (1) MECTA improves accuracy under cache constraints. With the constraint, Tent and EATA have to adopt a small batch size and therefore perform poorly, even worse than gradient-free adaptation (BN). By reducing the cache dynamically. MECTA boosts the performance of EATA and Tent drastically by 6- 71.6% on all three datasets. EATA+MECTA shows the best performance in CIFAR10-C and ImageNet-C and comparable accuracy to Tent on CIFAR100-C. (2) MECTA is more computation- and memory-efficient than GC. Though GC lowers the memory costs,
6

---

Published as a conference paper at ICLR 2023
it meanwhile increases the computation costs by around 20% in terms of GFLOPs. In contrast, our method only marginally increases the computation cost by less than 0.2% and can bear larger batches.
Table I: Continual evaluation on three datasets regarding accuracy (%) and cache sizes (Mb). For fair comparison, batch sizes (BS) are chosen such that the corresponding cache sizes are lower than those of BN with a batch size of 128. Orig. denotes the original data. Blue cells highlight the highest accuracy on the same dataset, and the bold texts indicate the best accuracy given the same base algorithm. GFLOPs is the number of 10 Floating Point Operations for adapting one sample.
Noise
Blur
Weather
Digital
Acc.
Catbe
GFLOPs
Ale
BS [Ganea Shot. Impul. Defoe, Glsss Marion Zoom.
Snow
Frost
Fog
Bright Coan:
Elast.
Pixel.
JPBG
Orig.
Ang
Avg Max
CIEARIO-C
BN
128
81.5
83.2
79.9
92.4
80.7
91.1
92.1
87.9
87.7
85.7
92.7
89.8
85.6
87.6
83.0
94.2
87.2
134
134
1.1
Tent
68.8
46.3
16.4
12.8
11.
8.9
9.9
10.7
10.3
10.2
10.3
10.2
10.1
10.0
10.2
10.2
16.7
114
114
2,180
+GC
16
81.8
82.2
71.8
68.9
52.8
43.7
39.0
30.7
22.8
17.7
10.1
6.7
7.4
8.2
6.1
9.0
35.1
130
130
2.6
+MECTA
86.5
87.2
81.3
88.3
78.9
84.1
85.8
81.3
80.8
77.4
82.1
77.4
74.9
76.6
71.7
77.5
80.7
2.182
EATA
74.5
69.7
55.5
54.6
47.2
3
35.1
35.2
2
40.5
+GC
83.4
84.3
81.3
86.4
77.0
83.8
86.5
84.5
84.7
83.9
86.
87.9
80.9
85.5
79.3
87.9
84.1
130
130
2.6
+MECTA
31
86.6
88.3
84.4
89.2
82.4
87.4
89.3
86.5
87.6
86.4
89.6
88.0
86.1
87.7
83.1
90.3
87.0
102
130
2.182
CIFARIO-C
BN
128
57.6
59.0
56.6
T2.5
58.2
09.9
71.8
64.7
84.8
57.9
73.3
69.8
64.3
66.7
58.6
75.8
65.1
134
134
1.1
Tent
52.9
$3.9
46.7
50.5
31.2
29.7
23.7
14.3
10.0
6.7
5.9
3.4
3.9
3.6
3.5
3.5
21.5
114
114
2,180
AGC
56.8
61.4
69.6
65.3
66.9
54.9
46.8
+MECTA
58.8
61.2
58.2
73.2
60.7
73.4
65.8
66.7
60.1
73.6
65.5
59.1
68.2
2.182
EATA
65.3
63.8
64.9
59.1
53.9
66.8
70.1
2,180
+GC
16
57.3
60.5
58.5
69.9
5
68.9
69.8
63.7
64.4
59.4
71.6
67.9
62.8
67.1
58.2
74.4
64.5
130
130
2.6
+MECTA
31
58.7
60.7
57.8
72.7
59.4
70.9
73.6
65.8
60.8
60.4
74.7
70.7
66.0
68
60.6
77.7
66.6
75
130
2.182
ImageNNi-CC
BN
128
39.2
42.6
39.6
29.9
32.9
40.8
47.4
45.0 47.7
55.8 66.5 36.0
54.6
65.4
55.7
74.2
48.5
[411
411
4.1
TENT
33.8
16.5
0.8
0.4
0.3
0.4
0.4
0.4
0.4
0.3
0.4
0.2
0.3
0.4
0.3
0.4
3.5
355
355
8.183
4GC
43.3
46.1
10.3
+MECTA
48.6
50.9
48.5 5
35.7
38.
39.6
44.2
87.0
87.4
42.1
51.9
31.7
42.9
47.6
42.5
63.6
43.3
338
8.190
BATA
34.1
37
35.0
27.5
28.1 1
35.5
39.6
39.7
47.8
56.6
35.5
44.1
46.7 7
AGC
16
44.4
47.1
45.4
39.0
39.4
47.4
49.7
49.7
48.4
67.6
64.3
47.8
54.6
61.7
56.3
60.5
51.4
204
404
10.3
+MECTA
30
50.6
53.3
51.7
44.7
46.
52.2
50.1
53.4
53.0
02.0
68.9
62.9
60.4
67.1
61.7
73.6
56.7
542
397
8.190
Comparison using the same batch size. In Table
follow the CTA protocol of (Niu et al., 2022)
to compare the proposed method with other baselines, e.g., TTT (Sun et al., 2020). TTA (Ashukha et al., 2021), MEMO (Zhang et al., 2021) and CoTTA (Wang et al., 2022a). For a fair comparison, all methods only adapt BN layers if applicable, and use the stndardly-trained ResNet50, which is publicly available, for example, from the official PyTorch package. (1) MECTA memory-efficient with the same batch size. With only 30% of cache (or 52% of GC cache), MECTA maintains similar performance as the standard EATA. Notice that GC is less effective in cache reduction here than the reported results in (Chen et al., 2016). because the parameter-efficient adaption already drops a lot of caches for frozen convolutional layers. The cache cost of EATA+MECTA is pretty low which is only more than those of the BN and non-adapted source models, yet has a large gain (12%) accuracy meanwhile. (2) MECTA mitigates forgetting. Note Tent has been known to suffer from catastrophic forgetting in continual adaptation (Niu et al., 2022) and thus has decreasing trend the accuracy by sequential domains, and we find that our method can mitigate this issue. The inherent reason for the mitigation is quite intuitive: the MECTA will adaptively reduce the number of parameters to be updated, which is similar to the regularization of EATA on norm-bounded parameter distance. Also, we add the original dataset as the last domain in CTA to evaluate if the model forgets the original data. Both Tent and EATA have significantly lower accuracy on the original set with small batch size Table I, while Ten/EATAA-MCTA gains much higher original-set accuracy on all datasets.
5.2 QUALITATIVE STUDIES
In the following experiments, if not specified, we use ImageNet-C and robustly pre-trained ResNet50.
Effect of each component in ECTA on memory-accurac, trade-off. For brevity, we name the components as MECTA-X where X is B for adaptive forgetting, C for stochastic cache pruning and L for adaptive layer training. We define the accuracy averaged over all corruptions as the mean-corruption acc. An overall comparison with EATA and BN on mean-corruption acc is de- picted in Fig. 2a. For BN, we vary the batch size from (32, 64, 128, 256, 512). For EATA and EATA+MECTA-B we vary the batch size from (4, 8, 16, 32, 64) where 64 is the default setting from
7

---

Published as a conference paper at ICLR 2023
Table 2: Comparison to other baselines with batch size of 64 on the standardly-trainee ResNer50 on ImageNet-C. #fwd and #bwd denote the times of forward and backward over a batch on average. GN is the group norm and JT denotes the model is jointly trained via supervised cross-entropy and rotation prediction losses. The bold numbers indicate the highest accuracy (or smallest cache) and the underlined ones the second best result. Cache sizes with GC are in brackets.
Noise
Blor
Weather
Digitl
Cache
#fud Whwd
Alg.
Gauss.
Shor.
Impul. Defoc Glass Motion Zoom. Snow Frod Fog Bright. Contr. Eastt. Pisel. IPEG
Max (Mb)
ResNet50:
Reset model per perturlation
(GN-JT)
33.6
33.4
28.1
7.8
33.2
36.8
19.0
51.0
49.4
51.7
2460 x
BN
15.5
16.1
16.3
20.0
20.0
28.5
40.0
34.8
35.0
48.5
65.9
24.1
45.8
50.7
38.5
306
TTA
4.1
4.9
4.5
12.5
8.2
12.9
25.8
14.0
19.1 21.3
53.0
12.4
14.6
24.6
33.6
17.7
206
64
0
MEMO
7.5
8.7
9.0
19.7
13.0
20.7
27.6
25.3
28.8 32.1
61.0
11.0
23.8
33.0
37.5
23.9
2460 X
65
65
ResNet0:
Lifelnn odaptation
COTTA+GGC
203
228
206
22.0
31.7
424
47.2
38,9
44.5
48.6
2845 (1618)
33
Temt--GC)
34.1
31.7
19.3
12.2
6.9
0.8
0.7
0.6
0.7
9.5
2845 (1618)
Tent-MBCTA
24.5
29.5
28.3
22.0
23.5
27.4
37.2
28
27.1
36.8
50.7
15.5
38.0
40.2
34.7
30.9
847
EATAGAGC)
35.0
36.
36.8
338
34
47.3
53.2
51.1
45.6
99.7
68.0
44.2
57.2
60.4
54.7
48.0
2845
(1618)
0.56
EATA-MECTA
89.1
87.8
31.7
33.1
42.2
30.3
66.9
85.2
the original paper of EATA. For EATA+MECTA-BL, we let the batch size be 16 and vary the Bu, in {1, 6.25, 12.5, 25, 50, 100, 200, 400} x 10 4. For EATA+MECTA-BL we let the batch size be 16 and Pu. be 0.00125 and vary the pruning rate in {0.1, 0.3, 0.5, 0.7,
The simplest BN method reaches the best accuracy at the batch size of 128 with approximately 400 Mb cache, which is much lower than the best accuracy achieved by EATA. However, given the same cache size, EATA does not perform better than BN. Only when 1200 Mb cache memory is occupied, EATA significantly outperforms BN by 5%. The need for a large batch (corresponding to a large cache) by EATA comes from the poor performance when the batch size is small, e.g.. at a batch size of 8 or 16. MECTA-B treats EATA with a streaming normal- ization with adaptive forgetting and memorization leading to higher accuracy on small-batch adaptation. EATA+MECTA-1 achieves better or comparable accuracy using either the smallest memory or the largest memory. Meanwhile, MECTA-L and MECTA-C improve the trade-off frontier into a low-cost zone. Given cache size lower than 600 Mb, MECTA-B BL can beat the best accuracy of EATA. Last, with cache pruning, MECTA- BLC further improves the efficiency more impressively: best accuracy at an extremely small cache size (fewer than 100 Mb).
2200
2000
DATA- -MECTA BalA Bifli-SC
200
1020
2003
3000
4003
5300
5002
7000
Impul
Form 45
2.2
0.2
2003
2500
3000
3532
0000
2000
5003
5500
Figure 3: Dynamic cache sizes and layer-wise B using MECTA.
To understand the cache reduction, we study the memory/accuracy by three MECTA components. (L) Adaptive layer-training. In Fig. 3, we show that the cache size will dynamically change by iterations using EATA+MECTA (batch size of 32). Especially, the cache size will peak the beginning of distribution shifts and gradually vanishes. The 3 differs by layer and different layer has different sensitivity the distributional shifts, which motivate to set / and activate training liserl. In Fig. 2b, when 3n is small, all layers will be trained and therefore the cache size is larger than the efficient BN adaptation. Increasing in reduces the cache size average of the adaptation process. Though this meanwhile decreases the accuracy, the accuracy is still higher than the BN adaptation by more than 4% based on the same batch size.
802
500
ass
14 8
SM
SKAINECIA BL 507--ECTA-00
520
1002
1530
2000
2500
0.6
avetage cache size (Mb)
(a) Trf comparison.
(b) Adaptive training (L).
(c) Cache pruning (C).
Figure 2: Ablation study on the components of MECTA. The dash horizontal lines in (b) and (c) are the performance of BN with the best trade-off in (a) (of batch size 128).
8

---

Published as a conference paper at ICLR 2023
Table 3: Ablation study of MECTA-B regarding accuracy (%) and a batch size of 16. Bold texts indicate the best accuracy among ablations. See Table 7 for other backbone methods.
EMA
BECTA
Nobe
Dlur
Weather
Digital
Alp. = 0.1
unio 8
Gauss,
Shot.
Impul.
Defoc
Glass. Mution Zoom.
Snow
Frost
Fog
Bright
Contr.
Elast Piael. JPEG Orig.
Avg
44.4
47.1
45.4
39.0
39.4
47.4
49.7
49.7
48.4
57.6
64.3
47.8
54.5 61.7
56.3
69.5
61.4
EATA
/
49.2
52.0
50.6
43.8
45.1
62.5
65.5
64.4
53.5
62.3
69.6
62.1
60.0
67.4 61.6
74.4 16.5 62.6
50.0
68.1
45.7
68.2
66.6
64.8
68.2
70.3
62.6
61.1
08.1
76.0 67.3
Table 4: Evaluation of k-new K-old shift accuracy by EATA. Average accuracy (AA %) and worst accuracy (WA %) are reported for each target perturbation. Values in the brackets denote the difference between the current method and the base one. More combinations of (K, , k) are in Table 8
EMA
MECTA-B
Impul.
Motion
Fog
Flast
A 0.1 Auto P
WA
AA
WA
AA
35.5
34.6
37.0
36.8
50.1
49.8
46.3
47.8
X
35.4
(4.1) 30.6 (-4.0) 28.6
(-11.0)
20.1
(-16.7)
41.0 (-9.1) 26.9 (2239) 42.5 (455) 39.3 (-5.5)
34.4 (-LD 32.0 (-26) 28.6 6 (-84) 25.7 (-1LD) 43.6 (-6.5) 39.1 (-10.7) 42.6 (5.7) 41.2 (4666
(C) Channel pruning. Fig. 2c shows that the pruning linearly reduces the maximal cache size, which bounds the maximal memory requirement in the whole adaptation life-cycle. With such a reduction in memory, pruning only mildly sacrifices the accuracy by less than 1%
(B) Adaptive forgetting. In Fig. 2a, we observe that small batch size (corresponding to small cache size) hurts the accuracy of EATA significantly. Thus, it is crucial to ask how the proposed adaptive forgetling and memorization improve CTA accuracy in small batches. To answer this question, we reiterate the two keys of CTA: accurate modeling of data distribution from stream data and stability in dynamic environments. We show that our adaptive forget gate can achieve the two goals simultaneously without aeper-parameters compared to the traditional moving average method.
(B.1) Parameter-fres memorization improves the per-domain accuracy. In Table 3, we present the per-domain accuracy on the full perturbation set with batch size 16 using all perturbations. We include the exponential moving average (EMA) to show that memorizing streaming batches into moving average can improve the accuracy of small batch sizes. Despite the effectiveness of EMA, the hyperparameter / requires extra effort or extra labeled data in tuning. In contrast, MECTA-B is parameter-fie free and meanwhile outperforms the EMA on all corrupted and original data.
(B.2) Forgetting improves shift stability. Regarding the advantage of adaptive forgetting in Table 3, we conjecture that the adaptive 3, estimated by MECTA-B can benefit the accuracy when the data distribution shifts from one domain to another, namely, shift accuracy (stability). To quantify the shift accuracy, we suppose that a model first experiences K batches from one domain (old domain), and then receives ie batch from another domain (new domain). The shift accuracy is the correct ratio of model predictions on samples of the k-new batches. For example, we measure the shift accuracy by randomly sampling 49 batches from the old domain and retrieving one batch from the new domain without replacement until all data of the new domain have been evaluated. For a given kind of perturbation as a new domain, we evaluate the shift accuracy paired with all kinds of perturbation as the old domain and report the average-case accuracy (AA) and the worst-case accuracy (WA). Here, we use ResNet50 on ImageNet and a subset of perturbations for the case of intensive evaluations: Impul, Motion, Fog, and Elast. As summarized in Table 4, EMA causes large accuracy declines, compared to the eattheesttmmate BN statistics (without distribution memory). This observation implies that in contrast the large gains in continual evaluation, the distribution memory induced by EMA becomes an obstacle for models to efficiently adapt new domains. Instead, short-term memory is preferred. Without explicitly switching a hyper-parameter, our method automatically detects such domain shift and merits the accuracy on shift with memory from past domains.
CONCLUSION
Though the braseent-based Continual Test-time Adaptation (CTA) is an effective manner to improve running performance on test data, the high memory consumption becomes a considerable challenge for deploying the technique to resource-limited end devices, like smartphones. To our best knowledge, our work is the pilot study on the memory efficiency and reveal the main bottleneck is on the intermediate results cached for propagaton,,,, even if only few parameters need to be updated cerpuuattiooeeffiienttt efficient solution (EATA). Therefore, we propose the first cache- aware method that achieves higher accuracy with much lower cache memory.
in the state-of state-of-the the computation

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
In this paper, we work is not related to human subjects, practices to dataset releases, discrimina- tion/biaess/aarness concerns, adn also do not have legal compliance or research integrity issues. Our work is proposed to address the memory efficiency of model adaptation in continual distributional shifts. In this case, if the model trained out of good will, have good faith that our methods won't cause any ethic problem or negative social impacts.
REPRODUCIBILITY STATEMENT
The algorithm pscudo codes enclosed in the main body. We also provide details the implemen- tation, hyper-paramedte in Appendix B.I. The datasets and baselines codes are all public available online and specify them properly in our paper.
ACKNOWLEDGMENTS
This research funded by Sony AI. This material is based in part upon work supported by the National Science Foundation under Grant IIS 221222, II--1749940, ECCS- 2024270, Office of Naval Research N00014-20-1-238,, and National Institute Aging (NIA) RG174422449.
REFERENCES
Berkin Akin, Zeshan A. Chishti, and Alaa R. Alameldeen. Zcomp: Reducing dnn cross-layer memory footprint using vector extensions. In Proceedings of the 52nd Annual IEEF/ACM International Symposium on Microaccittcturre, pp. 126-138, Columbus OH USA, October 2019. ACM.
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. International Conference on Learning Representations, July 2021.
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. Tinytl: Reduce memory. not parameters for efficient on-device learning. Advances in Neural Information Processing Systems, 13:11225-11297, 2020.
Tianqi Chen, Bing Xu, Chiyuan Zhang, and Carlos Guestrin. Training deep nets with sublinear memory cost, April 2016.
Vitaliy Chiley. Ilya Sharapov, Atli Kosson, Urs Koster, Ryan Reece, Sofia Samaniego de la Fuente, Vishal Subbiah, and Michael James. Online normalization for training neural networks. In Advances in Neural Information Processing Systems, volume 32, Curran Associates, Inc., 2019.
Francesco Croce, Maksym Andriushchenko, Vikash Sehwag, Edoardo Debenedetti, Nicolas Flam- marion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robusthench: standardized adversarial robustness benchmark, October 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-244822 255, June 2009.
Yuyang Deng, Nidham Gazagnadou, Junyuan Hong, Mehrdad Mahdavi, and Lingjuan Lyu. On the hardness of robustness transfer: A perspective from rademacher complexity over symmetric difference hypothesis space. 2023.
Jiahua Dong, Lixu Wang, Zhen Fang, Gan Sun, Shichao Xu, Xiao Wang, and Qi Zhu. Federated classinncrmmentl learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10164-10173, 2022.
Rahul Garg and Rohit Khandekar. Gradient descent with sparsification: iterative algorithm for sparse recovery with restricted isometry property. In Proceedings of the 26th Annual International Conference Machine Learning, June 2009.
10

---

Published as a conference paper at ICLR 2023
Marta Garnelo, Dan Rosenbaum, Christopher Maddison, Tiago Ramalho, David Saxton, Murray Shanahan, Yee Whye Tch, Danilo Rezende, and SM Ali Eslami. Conditional neural processes. In International conference machine learning, pp. 1 1704-1713 1713. PMLR, 2018.
Sachin Goyal, Mingjie Sun, Aditi Raghunathan, and Zico Kolter. Test-time adaptation via conjugate pseudo-labels. July 2022.
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Computer Vision and Puttern Recognition, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference Learning Representations, March 2019.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, Dawn Song, Jacob Steinhardt, and Justin Gilmer. The many faces of robustness: A critical analysis of out-of-distribution generalization. June 2020a.
Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: simple data processing method improve robustness and uncertainty. In International Conference on Learning Representations, February 2020b.
Andrew G. Howard, Menglong Zhu, Bo Chen, Dmitry Kalenichenko, Weijun Wang, Tobias Weyand, Marco Andreetto, and Hartwig Adam. Mobilenets: Efficient convolutional neural networks for mobile vision applications, April 2017.
Gao Huang, Zhuang Liu, Laurens van der Maaten, and Kilian Q. Weinberger. Densely connected convolutional networks, January 2018.
Andrey Ignatov, Radu Timofte, William Chou, Ke Wang, Max Wu, Tim Hartley, and Luc Van Gool. Ai benchmark: Running deep neural networks on android smartphones. In Proceedings of the European Conference on Computer Vision (ECCV) Workshops, pp. 0-0, 2018.
Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv:1502.03166 [cs], March 2015.
Yusuke Iwasawa and Yutaka Matsuo. Test-time classifier adjustment module for model-agnostic domain generalization. In Advances in Neural Information Processing Systems, volume 34, pp. 2427 2440. Curran Associates, Inc., 2021.
Ziyu Jiang, Xuxi Chen, Xueqin Huang, Xianzhi Du, Denny Zhou, and Zhangyang Wang, Back razor: Memory-efficient transfer learning by self-sparsified backpropagation. In Advances in Neural Information Processing Systems, 2022.
Ansh Khurana, Sujoy Paul, Piyush Rai, Soma Biswas, and Gaurav Aggarwal. Sita: Single image test-time adaptation, September 2022.
Pang Wei Koh, Shiori Sagawa, Henrik Marklund, Sang Michael Xie, Marvin Zhang, Akshay Balsub- ramani, Weihua Hu, Michihiro Yasunaga, Richard Lanas Phillips, Irena Gao, Tony Lee, Etienne David, Ian Stavness, Wei Guo, Berton Earnshaw, Imran Haque, Sara M. Beery, Jure Leskovec, An- shul Kundaje, Emma Pierson, Sergey Levine, Chelsea Finn, and Percy Liang. Wilds: A benchmark of in- tn-the-wild distribution shifts. Proceedings of the 38th International Conference on Machine Learning, pp. 5637-5664. PMLR, July 2021.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Yanghao Li, Naiyan Wang, Jianping Shi, Jiaying Liu, and Xiaodi Hou. Revisiting batch normalization for practical domain adaptation. In International Conference on Learning Representations. arXiv, 2017.
Qianli Liao, Kenji Kawaguchi, and Tomaso Poggio. Streaming normalization: Towards simpler and more hologically-plausin normalizations for online and recurrent learning. October 201 6.
11

---

Published as a conference paper at ICLR 2023
Chenxi Liu, Lixu Wang, Lingjuan Lyu, Chen Sun, Xiao Wang, and Qi Zhu. Deja Continual model generalization for unseen domains. In International Conference on Learning Representations. 2023.
Yuejiang Liu, Parth Kothari, Bastien van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. Tit++: When does self--uper test-time training fail or thrive? In Advances in Neural Information Processing Systems, volume 34, pp. 21 21808-21 820. Curran Associates, Inc., 2021.
Arun Mallya and Svetlana Lazebnik. Packnet: Adding multiple tasks single network by iterative pruning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7765-7775 2018.
Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single network to multiple tasks by learning mask weights. In European Conference Computer Vision, volume 1 11208, pp. 72-88, Cham, 2018. Springer International Publishing.
Zachary Nado, Shreyas Padhy; D. Seulley, Alexander D Ammou, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-tine batch normalization for robustness under covariate shift. arXiv:2006. 10963 [cs, stat], January 2021.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International Conference on Machine Learning, 2022.
Shiori Sagawa, Pang Wei Koh, Tatsunori Hashimoto, and Percy Liang. Distributionally robust neural networks for group shifts: On the importance of regularization for worst-case generalization. In ICLR, April 2020.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. In Advances in Neural Information Processing Systems, October 2020.
Vikash Sehwag. Shiqi Wang. Prateek Mittal, and Suman Jana. Hydra: Pruning adversarially robust neural networks. In Advances in Neural Information Processing Systems, volume 33, pp. 19655- 19666. Curran Associates, Inc., 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition, April 2015.
Yu Sun, Xiaolong Wang, Zhuang Liu, John Miller, Alexei Efros, and Moritz Hardt. Test-time training with self-supervision for generalization under distribution shifts. In International conference on machine learning, pp. 9229-9224 PMLR, 2020.
Mingxing Tan and Quoc V. Le. Efficientnetv2: Smaller models and faster training, June 2021.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test- time adaptation by entropy minimization. International Conference on Learning Representations, 2021.
Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 7201-7211, 2022a.
Zifeng Wang. Zheng Zhan, Yifan Gong, Geng Yuan, Wei Niu, Tong Jian, Bin Ren, Stratis Ioannidis, Yanzhi Wang, and Jennifer Dy. Sparel: Sparse continual learning on the edge. In Advances in Neural Information Processing Systems, September 2022b.
Stephen J. Wright. Coordinate descent algorithms. Mathematical Programming, 151(1):3-34, June 2015.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks, April 2017.
12

---

Published as a conference paper at ICLR 2023
Tao Yang. Shenglong Zhou, Yuwang Wang, Yan Lu, and Nanning Zheng. Test-time batch normaliza- tion, May 2022.
Geng Yuan, Xiaolong Ma, Wei Niu, Zhengang Li, Zhenglun Kong, Ning Liu, Yifan Gong, Zheng Zhan, Chaoyang He, Qing Jin, Siyue Wang, Minghai Qin, Bin Ren, Yanzhi Wang, Sijia Liu, and Xue Lin. Mest: Accurate and fast memory-economic sparse training framework on the edge. In Advances in Neural Information Processing Systems, volume 34, pp. 08838-20850. Curran Associates, Inc., 2021.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. 2017.
[cs], June
Marvin Zhang, Henrik Marklund, Nikita Dhawan, Abhishek Gupta, Sergey Levine, and Chelsea Finn. Adaptive risk minimization: Learning adapt domain shift. In Advances in Neural Information Processing Systems, December 2021.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. NeurlPS 202/ Workshop DistShift, January 2022.
13

---

Published as a conference paper at ICLR 2023
A DISCUSSION
A.1 MEMORY FOOTPRINT FOR TEST-TIME ADAPTATION
When learning algorithm looks for deployment on edge devices, it is unavoidable to take the two folds into account: (1) whether the maximal memory consumption of the algorithm can be fitted into a mobile device; (2) whether the dynamic and instantly-free space of device memory (RAM) is sufficient for the adaptation. The current mobile devices generally have memory in GB levels. For example, current general mobile devices such Samsung S20 have 2GB or more memory, Raspberry Pi has 1GB or more memory. However, unlike the training or adaptation on a high-capacity GPU server where all the resources (memory and CPU time) be allocated for training, the memory on mobile devices may be temporarily and partially allocated for the operating system and other background applications. The situation could be even more severe for mobile devices. Thus, squeezing the memory footprint dynamically is crucial for edge devices.
To avoid losing focus of the paper, only consider the cache for gradient computation batch- normalization layers, which has been a substantially- memory footprint compared to those of the widely-studied model and gradient memory in the literature (Yuan al., 2021; Akin et al., 2019; Wang et al., 2022b). Admittedly, our work does not cover all memory consumption in the life-span of model adaptation. In Table 1, we only compute the memory costs (cache) of the paok-propggttion but not the forward operations, because the memory costs of the latter will be released immediately. The actual memory occupation in hardware, like NVIDIA GPU, will be enlarged due to the reservation for faster inference. The holistic solution that jointly optimizes the inference time and memory requires extra efforts on low-level computation and could be hardware-dependent. Thus, we leave the solution as open problem for future study.
B EXPERIMENTAL DETAILS AND SUPPLEMENTARY
B. IMPLEMENTATION DETAILS
Syper--arammters. All test-time adaptation objectives are optimized by stochastic gradient descent (SGD) with momentum of 0.9. Tent and EATA utilize a batch size of 64 with a learning rate of 0.005 (0.000 (0.00025) for CIFAR- 10 (CIFARIO) 100 and ImageNet). In our implementation, we use 0.0025 (0.0001) as learning rates to stabilize the training with smaller batch sizes. EATA uses 2,000 samples to estimate Fisher matrix for anti-forgetiing regularization. For MECTA, we set the threshold 3m for stopping layer training as 0.0025 for CIFAR 100, 0.00125 for CIFAR1O, and 0.00125 for ImageNet-C. The cache pruning rate is set to be 0.7 for all datasets.
We implement our algorithm using Torch 1.12.1, cudatoolkit 11.6 on NVIDIA Tesla T4 GPUs. The codes of baselines provided by the open sourced codes of EATA'. For gradient checkpointing, we use the official implementation from PyTorch, torch. ut ils. checkpoint. For each stage in the ResNet with m blocks, we will split the blocks sequentially into [m/2] segments. Therefore, ResNe150 will be split into segments, approximatedly equal to /50.
Measuring cache sizes. Without GC, we measure the cache size by summing up the tensor size of all features z in network. With GC, we estimate the cache size as two parts: one is the segment cache sizes and the other is the maximal cache for backwarding inside a segment. In ablation of B3 and evaluation of shift accuracy, we ignore the BN's in shortcut layers (or downsample layers) in variants of ResNet and use simplified implementation.
Measuring full footprint. In Fig. 1, we track the tensors that are cached in the GPU memory using the public tool?. The memory tracker will find all the PyThorch tensor variables in the garbage collector of Python. We also leverage tool, torch, cuda memory_cace cached, provided by Py PyTorch to estimate the maximal GPU memory costs including non-tensor variables.
thub.
14

---

Published as a conference paper at ICLR 2023
Implementation of stochastic cache. During forward, we will only store the remained values of 2 and the indexes of the remained channels (denoted as R). Later, we compute the gradient by
B dln 0%
W Σ
86. E 2k do, mad
i R E R.
0,
As the zero values do not need cache, the implementation can effectively reduce memory consumption with a small extra space for storing the index set R.
B.2 MORE EXPERIMENTAL RESULTS
MECTA in different network architectures. It known that larger and deeper models will merit the robustness of neural networks (Hendrycks et al., 2020a). In Fig. 4, we compare the EATA and EATA+MECTA on varying model depths. We adopt the protocol Table but reduce the batch size to 32 to accommodate the huge memory cost of deeper networks. The experiments are conducted with 4 perturbations. By increasing the depth of ResNet, the cache size increases steeply. Instead, MECTA reduces the cache consumption
52-
method
1400
mean-coruptton ass
1.0
EATA
1200
aVg cache size (M))
48
46
600
400
200
40
100
150
depth
low level and achieves even better accuracy. Beyond ResNet, we evaluate more network architectures in Table 5. For example. MobileNet (Howard
Figure 4: Evaluation on different ResNet models with varying depth. et al., 2017) is designed for edge devices with limited computation resources. Since our method only modify the batch- normalization layers, can be easily plugged into these networks. Except wide ResNet (WRN), our method outperforms EATA with much lower cache sizes and higher accuracy.
Table 5: Evaluation on different model architectures retrived from PyTorch pre-trained models.
Acc. (%)
Coche (Mb) Max
Architecture
MobileNe122 (Howard 2017)
EATA
26.8
854.8
854.8
EATA&MECTA
233.6
250.6
MobileNerV3 (Howard al. 2017)
EATA:MECTA
EATA
32.2
30.8
158.1
563.2
163.7
563.2
VGGI9+BN (Simonyan & Zisserman, 2015)
EADA+MECTA
35.7
1901.1
544.8
1901.1
564.9
WRN10IX2 Zagonuyko Komodakis, 2017)
EATA
54.7
2716.7
2716.7
Re<Ner101
EATA
41.2
1885.0
1585.6
EATA&MECTA
61.3
449.1
669.8
ResNet152
EATA+MECTA
52.9
2894.3
635.6 6
2894.3
813 6
ResNeX1101 8d (Xie et 2017)
EAIA
53.9
3802.1
3802,1
FATA&MECTA
1094.7
1186.1
DenseNer121 (Huang et al., 2018)
EATA
EATA+MECTA
45.9
42.2
2005/4
594.3
2005.4
597.6
EficieenNerV2-SS (Tan 2021)
EATA
1556.3
MAIA&
454.4
463.3
Does layer-sparse training help? In Fig. 5, we evaluate EATA with fewer trainable layers, which can reduce the cache size, following the protocol in Table 2. During adaptation, we keep & deepest layers to be trained and freeze other layers, which is denoted Lk in the figure. We also include the EATA+GC where we use gradient checkpointing for the trainable layers. We observe that reducing the trainable layers can significantly decrease the cache size which is even lower than MECTA. However, the corresponding accuracy is significantly decreased by 5% compared to EATA MECTA meanwhile. In comparison, though MECTA also uses the layer-sparse training, our method presents the best accuracy-memen memory trade-off in the experiment. The key difference is that our method sparsifies the training only on demand, specifically when layer well adapted without need for further training.
Does MECTA-B works with BN adaptation and Tent? We show that MECTA-B can generally works well with BN adaptation and Tent, Table Consistent on all three backbone methods, the MECTA-B can outperforms EMA and base methods without extra ssper-parameters. EMA and MECTA-B can salvage BN and Tent from pooer performance using small batch sizes.
15

---

Published as a conference paper at ICLR 2023
47.5
More shift-aceuracy evaluation. We consider more cases of (K, k) pairs in Table In all three trials, MECTA-B reduce the shift-accuracy drops w.r.t. the bascline. Given more new-domain samples, e.g.. k 5, the shift-accuracy using EMA and MECTA-B becomes higher than the base- line, implying the quick convergence of the adaptation.
mesn-crruption act (86)
L35
42.5
40.0
EATA+MECTA
EATA
35.0
15
EATA+GC
Cache size and by iteration. We extend Fig. a full
500
1000
1500
2000
2500
life-cycle version in Fig. 6. For evaluating 3, we run the
max cache (Mb)
experiment on ImageNet-C using ResNet50 and we use MECTA with EATA for adaptation. In more corruptions, we find the periodic fluctuation of 3 by the distributional shifts, which results in the dynamic cache sizes.
Figure 5: Compare MECTA to layer- sparse training using ResNet50 and batch size of 64. LI means that only the deepest layer is trained. Likewise, L45 denotes the deepest 45 layers.
Does MECTA works on even smaller batches? In Ta- ble 6, we extend Table I with more batch sizes. One interesting observation is that our method outperforms or comparable to other baselines given even smaller caches. For example, given batch size of 16, EATA+MECTA can outperform BN at the best batch size, when MECTA reduces the cache size to 71 Mb on average compared to the 134 Mb by BN.
Table Continual evaluation on three datasets with the highest severity level 5 regarding accuracy (%). For a fair comparison, batch sizes (BS) are chosen such that the corresponding cache sizes lower than those of BN with batch size of 128. Blue cells highlight the accuracy that is the highest among all methods, and the bold texts indicate the best accuracy given the same base algorithm.
Wenther
Digital
Acc.
Alg.
Gouss.
Shot,
Impul. Defoc.
Mation Zoom.
Frust
Bright.
Pisel.
JPEG
Orig.
Avg
Avg
CHEAR
32
60.2
82.0
78.7
91.2
79.2
89.9
91.0
86.0
86.6
84.5
91.9
88.8
84.2
86.8
81.8
93.2
86.1
34
BN
64
61.3
82.7
79.6
91.9
79.9
90.8
91.6
87.5
67.0
85.5
92.3
89.4
85.3
67.5
82.6
93.7
86.8
67
66
BN
128
81.5
88.2
79.9
92.4
80.7
91.1
92.1
87.9
87.7
85.7
92.7
89.8
85.6
87.6
88.0
94.2
87.2
134
134
TENT
8
68.8
46.3
16.4
12.8
11.8
8.9
9.9
10.7
10.3
10.2
10.3
10.2
10.1
10.0
10.2
10.2
16.7
114
114
270
86.0
87.5
85.6
86.6
85.5
86.6
83.6
87.0
83.0
85.7
80.7
87.2
85.5
457
~MECTA
16
86.6
86.3
79.1
82.8
73.3
76.3
76.9
72.6
71.5
G7.3
70.6
64.5
60.4
60.5
56.1
60.4
71.6
49
67
-MECTA
31
86.5
87.2
81.3
88.3
78.9
84.1
85.8
81.3
80.8
77.4
82.1
77.4
74.9
76.6
71.7
77.5
80.7
93
129
EATA
8
74.5
70.9
66.6
69.7
55.5
54.6
47.2
38
33
35.1
35.2
18.5
12.2
14.5
13.3
8.5
40.5
114
114
83.4
87
229
32
85.7
87.6
85
82.5
88.3
90.00
91
89.5
85.3
90.6
88.0
457
-METAA
16
86.1
86.4
81.0
86.9
80.3
84.9
87.4
83.8
85.3
85.5
88.3
86.3
83.2
86.4
82.5
88.6
85.2
53
68
MEBIA
31
86.6
88.3
84.4
89.2
82.4
87.4
89.3
86.5
87.G
86.4
89.6
88.0
80.1
87.7
83.1
90.3
87.0
102
130
CIAARIOO C
32
56.2
57.1
54.7
70.5
58.3
68.6
70.2
63.0
63.5
56.6
71.9
68.0
62.2
64.8
56.7
73.8
63.4
34
65.7
30.8
09.5
71.5
64.1
64.3
07.4
68.1
58.4
75.8
64.6
67
BN
128
57.6
59.0
56.6
72.5
58.2
69.9
71.8
64.7
64.8
57.9
73.5
69.8
64.3
66.7
58.6
75.8
65.1
134
134
TENT
53.0
53.9
46.8
50
31.2
29.8
23.8
14.4
10.1
6.7
5.9
3.5
4.0
3.7
3.6
3.5
21.5
114
114
16
56.8
61.4
59.6
68.5
56.1
65.3
66.9
59.7
60.0
54.9
65.0
57.0
54.2
56.2
46.8
58.6
59.2
229
229
12
58.5
63.0
61.8
71.6
60.3
69.7
71.8
65.1
66.3
60.9
72.4
68.
64.7
67.8
59.
74.3
66.0
457
457
-MECTA
52.3
67.0
-MECTA
31
58.8
61.2
58.2
73.2
60.7
71.4
73.4
65
66.
60.
73.6
68.4
65.5
67.3
59.1
75.1
66.2
77
130
52.1
64.4
33.2
65.4
51.7
63.9
64.8
59.3
58.8
54.0
66.6
63.3
56.4
61.4
52.9
70.1
58.3
114
16
57.5
60.5
58.5
69.9
6
68.9
69.8
63.7
64.4
09.4
71.6
67.9
62.8
67.1
58.2
74.4
64.5
229
229
32
58.4
62.4
60.9
72.
59.5
70.3
72.4
66.3
66.5
62
74.4
70.8
65.3
69.4
61.0
76.9
66.8
457
457
~MECTA
44
-MECTA
31
58.7
60.7
57.8
72.7
59.4
70.9
73.6
65.
64.5
60.4
74.7
70.7
66.0
68.4
60.6
77.5
66.6
75
130
ImageNes-C
RN
64
38.4
41.6
38.8
29.1
32.1
40.1
46.4
44.1
46.8
64.8
67.7
35.1
53.7
64.7
54.6
73.6
[47.6]
306
206
BN
125
39.2
42.6
39.6
29.9
82.9
40.8
47
45.0
47.7
66.8
G8.6
36.0
64.8
66.4
66.7
74.2
48.6
411
411
TENT
16
43.3
46.1
42.8
25.8
14.8
5.0
1.3
0.7
0.7
0.7
0.8
0.6
0.7
0.7
0.7
0.7
11.6
711
711
32
16.1
50.6
48.2
38.2
41.
42.5
36.6
36.2
38.8
48.8
27.5
30.5
34.3
26.9
34.8
38.6
2845
2815
48.6
49.3
46.6
3
34.8
87.5
26.7
29.2
35.7
14.0
20.6
21.9
16.1
19.3
31.0
194
212
-MECA
30
18.6
50.9
48.5
35.7
38.3
39.6
44
37
37.
42.
51.9
31.7
42.9
47.6
42.5
53.6
43.9
138
397
EATA
16
14.4
47.1
45.4
39.0
39.4
47.4
49.7
49.7
48.4
57.6
64.3
47.8
54.5
61.7
56.3
69.5
51.4
711
711
32
19.0
52.3
5L
44.4
45.2
52.3
5
54
52.7
61.3
67.6
52.7
5
65.3
60.4
72.3
55.9
2545
2645
<MECIA
50.7
45.3
50.6
53.8
60.6
67.6
61.8
56.6
-MECTA
46.1
52.2
52.9
67.1
73.6
342
16

---

Published as a conference paper at ICLR 2023
Table 7: Ablation study of MECTA-B on ImageNet-C with the highest severity level 5 regarding accuracy (%) and batch size of 16. Blue cells highlight the accuracy that the highest among all methods, and the bold texts indicate the best accuracy among ablations of EMA and MECTA-B,
EMA MECTA-B
Noise
Weather
Shot.
Defoc.
Glass.
Motion
Frost
Brght.
Contr.
Elast.
Pisel.
JPGG
x
x
33.7
36.5
34.0
24.6
27.2
35.0
40.5
39.1
42.3
40.4
62.7
30.7
47.6
59.1
48.9
68.1
42.5
39.8
43.3
40.3
30.3
48.3
56.6
36.0
55.4
x
43.3
42.8
25.8
14.8
0.7
0.8
0.7
Tear
/
x
49:2
53.3
51.6
38.8
S7.8
40.1
40.3
32.5
29.4
28.1
31.4
10.1
11.8
10.2
4.4
6.0
29.7
/
50.3
54.8
53.0
40.7
40.2
42.3
42.9
35.3
33.0
32
36.7
14.3
15.9
15.2
8.5
11.8
32.9
x
X
44.4
47.1
45.4
33.0
39.4
47.4
49.7
49.7
48.4
57.6
64.3
47.8
64.5
61.7
56.3
69.6
61.4
EAIA
50.0
54.3
Table Evaluation of k-new K-old shift accuracy by EATA. Average accuracy (AA %) and worst accuracy (WA %) are reported for each target perturbation. Values in the brackets denote the difference between the current method and the base method using batch statistics.
EMA
MECTA-B
Impul.
Motion
Elast
0.1
Auto 0
AA
WA
AA
WA
AA
WA
AA
WA
x
35.5
34.6
37.0
36.8
50.1
49.8
48.3
47.8
35.4 (40.1)
30.6 (-40)
26.6
20.1
(-16.)) 41.0 (9.1)
25.9
(-23.9) 42.8 (-55)
39.3 (-8.5)
(-10.7)
41.2
35.7
47.4
1
32.8 (-1.)
27.0 (6.4)
25.0
(-10.7)
17.3 (-18.)
38 48.4 (-11.)
23.9 (-25.0)
41.1 (6.3)
35.7 (-114)
34.1 (0-D)
32.3 (-I.))
27.4 (-8.))
23.1
(-12.5)
42.4 (-7.0)
36.5 (-12.4)
41.9 (-55)
40.0 (-.I)
x
95.4
34.6
36.9
36.5
50.3
49.3
48.5
47-7
45
38.4 (+3.0)
37.5
(42.9)
36.4 (L.5)
30.6 (-59))
53.3 (43.0)
50.2 (0.9)
52.0
(+3.5) 50.8 (63.1)
(40.3)
Impul, Motion
Bright
Orig.
350
sizs.
300
My!
5000
10000
15:000
20000
25000
itaration
Ganes Impul. Defoc, Glass, Motion Zoom, Snow Frost Fog Bright Centr.
Piael. jeEG Orig
horm
norm
a1x10-5
2.0
3.0
norm
0.5
5000
10000
15000
20000
25000
Figure 6: Dynamic cache size and 3 using MECTA on ImageNet-C.
17


Reviews:
[{"review_id": "rxHz7AWQYd7", "reviewer": "Reviewer_QbXv", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Test-time model adaptation updates model parameters during inference in order to reduce generalization error on shifted data. Continual test-time adaptation, the setting of this work, does so for varying shifts without knowledge of when the shift itself changes over time. The purpose of this work is to improve the efficiency of inference and adaptation to require less computation time and memory. In particular, the proposed memory economical continual test adaptation (MECTA) approach extends and tunes entropy minimization methods (Tent and EATA) so that their gradient updates do not need as much memory as the naive implementation of backpropagation during testing. This work highlights the memory usage of caching forward activations in particular, which is measured to be 5x the memory for inference with at ImageNet scale (ResNet-50, specifically). Efficiency is improved while maintaining accuracy by reducing batch size, extending normalization statistics to moving estimates rather than batch-wise estimates, dropping channels by a kind of test-time pruning or partial caching that is sparser than the standard backward pass, and gating updates by thresholding the change in layer-wise statistics. Any layer that is gated does not need its cache for updates, and so memory can be saved for the gated layer and the following layers. MECTA improves accuracy over Tent and EATA when memory is constrained (Table 1) while nearly achieving the same accuracy as EATA when memory is not limited (Table 2). These results are shown for the standard choices of datasets: ImageNet-C and CIFAR-10/100-C with common baseline models like ResNet-50.", "summary_of_the_review": "The deployment of test-time adaptation does require efficiency so that the updates made during inference do not delay predictions too much. Even more seriously, the adaptation computation needs to be feasible at all by fitting in device constraints such as memory, which is the constraint addressed by this work. Precise measurements of memory are provided, and the proposed MECTA offers several techniques that combine to reduce peak memory usage, at least in principle. The reduction of batch size and the gating of layer updates should reduce memory in practice also. On the other hand, the pruning/masking of gradient channels may or may not reduce memory depending on the implementation, so this requires clarification. While the experiments show the effect of the proposed techniques, they do not cover simple baselines, such as updating fewer layers or applying gradient checkpointing.\n\nThe amount of memory saved can be as much as ~70% of the original usage for a ResNet-50, but the lack of computational baselines makes it unclear how much this amount matters.\n\nQuestions:\n\n- How much memory is saved and accuracy maintained by simply updating different numbers of layers in the network, starting with the deepest and then adding more shallow layers? Please Consider only a single layer, say 10% of the layers, and 50% of the layers.\n- What is the memory/time trade-off of test-time adaptation with gradient checkpointing when compared to MECTA? In particular, what is this trade-off when limited by the memory constraints applied to Table 1?\n- Does the channel sparsity of the gradient truly achieve reduced computation in practice, that is on a GPU with CUDA, or is there only a potential reduction?\n- Does the dynamic cache truly reduce peak memory usage, or only average memory usage? The analysis results of Figure 3 suggest that the peak usage remains high, at the beginning of each shift. Please confirm the effect of MECTA on the peak usage.\n\n**Update following Response & Discussion**: The response addressed the questions and weaknesses identified by this review adequately, but not totally, and on the balance this deserves a recommendation that sides with acceptance. I have raised the score to borderline accept accordingly. The results from the response for simpler computational baselines justify the need for MECTA, and the further details provided by the response and revision make it more informative. MECTA is worth considering for acceptance because of its more dynamic and memory-efficient computation for adaptation.\n", "strength_and_weaknesses": "Strengths\n\n- MECTA does reduce the peak memory usage of Tent and EATA while still improving accuracy on shifted data in the continual adaptation setting. MECTA allows for 4x larger batches and consequently improves the accuracy of EATA by more than 10 points. The accuracy of Tent is also improved, but Tent was not designed for continual adaptation in the same way as EATA, so this improvement is less surprising and significant.\n- The stochastic dropping of channel-wise gradients (\"Sparse gradients via stochastically-pruned caches\") is a novel scheme to reduce gradient memory as far as I am aware. Dropping gradients instead of pruning channels has the advantage of preserving the forward computation and accuracy. (However, see weaknesses for a discussion of whether or not this can truly save memory given how deep networks are implemented.)\n- The layer-wise gating of updates is more granular than the input-wise gating of EATA, and so more computation can potentially be spared. In particular, MECTA can stop caching as soon as a layer is gated, while EATA requires an entropy prediction and therefore a full forward (and the corresponding memory) for each input to decide its gating.\n- MECTA has a regularizing side-effect on Tent that helps prevent forgetting, as shown by improved accuracy on _unshifted_ data after adaptation. This follows from its sparser updates and moving estimates.\n- The implementation of MECTA is encapsulated in a variation on the batch normalization layer, which makes it easy to adopt by swapping it into a network with batch normalization.\n- The experiments and in particular the analysis experiments in Figure 2 show that MECTA strictly dominates adaptation by BN for accuracy and memory usage. (However, without memory limits Tent and EATA can still do as well or better than MECTA, so this gain comes at the cost of specialization to this reduced computation setting.)\n\nWeaknesses\n\n- This work lacks simple computational baselines that would help prove the necessity and impact of MECTA's contributions. In particular, it does not investigate (1) updating fewer layers or (2) checkpointing gradients.\n  - Fewer layers: this work ignores the option of simply updating fewer layers during testing, by for instance only updating the last/deepest layers of the network. Only updating deeper layers removes the need for caching of all earlier layers.\n  - Gradient checkpointing: this work ignores the option of recomputation (a.k.a. rematerialization) by simply discarding forward caches and recomputing forward as needed for backward, and by doing so misses a simple baseline. This is now a common feature of deep learning frameworks, as is provided by the gradient checkpointing utility in PyTorch, for example.\n- A prior paper on continual adaptation is missing from the related work: CoTTA at CVPR'22. CoTTA is a contemporary method from the same time as EATA, and so likewise deserves inclusion in the related work on continual adaptation. In the submission, it is only mentioned and disqualified for its computational cost in the experiments (Section 5), which is not appropriate.\n- The proposed channel-sparse gradients may not save time or memory in practice. Most frameworks only support dense gradients for inputs and parameters, so zeroing out a particular channel may not alter the computation performed.\nThe analysis experiments or \"Qualitative Studies\" of Section 5.2 are done with a subset of the common corruptions. These fidings may or may not generalize to the full set, so it would be better to do them with the full set, for thoroughness and comparability with other results.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "EN9ehtVGZ8b", "reviewer": "Reviewer_wXpt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The state-of-the-art adaptations improve out-of-distribution model accuracy via computation-efficient online test-time gradient descents but meanwhile cost about times of memory versus the inference, even if only a small portion of parameters are updated. To address this issue, the authors provide a novel solution called MECTA to drastically improve the memory efficiency of gradient-based CTA. MECTA can improve the memory efficiency of different CTA methods. The simple norm layer structure is ready to be plugged into various networks to replace batch-normalization. The MECTA Norm layer also enables the stop and restart of model adaptation without unused or absent caches for unwanted or on-demand back-propagation. To avoid forgetting due to removing parameters, pruning is conducted on cache data for back-propagation rather than forwarding and can greatly reduce memory consumption. Compelling results show that MECTA can maintain comparable performance to full back-propagation methods while significantly reducing the dynamic and maximal cache overheads. ", "summary_of_the_review": "More could be discussed about the implications of their results, but overall, this is a solid paper tackling an important problem. The proposed approach is interesting and the technique is sound. The claims are well supported by impressive empirical results. ", "strength_and_weaknesses": "Strength:\n\n1. The paper provides sufficient insights and motivations on continual test-time adaptation with memory efficiency.\n\n2. The proposed methodology seems effective. On ImageNet dataset trained by ResNet50, it significantly reduces memory costs by at least 70% without sacrificing accuracy. \n\n3. The proposed MECTA is efficient and can be seamlessly plugged into SOTA CTA algorithms at negligible overhead on computation and memory. \n\nWeakness:\n\n1. How this method can be extended to other normalizations? Could you elaborate more on this with examples and experiments?\n\n2. More discussions on the potential applications of MECTA, in reality, are expected to better highlight its significance.\n\n3. Table 1 is a bit hard to interpret, maybe highlighting the dataset names would be better.\n\n4. The choices for the backbone network used in the experiments should be explained. Experiments on more network backbones (in particular large networks) would be interesting. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AOJuKddVCF", "reviewer": "Reviewer_veMt", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Continual test-time model adaptation (CTA) recently draws researchers' attention since trained models may encounter dynamically-changing test-time environments. This work first revealed the limitations of prior arts in memory efficiency which could be a critical obstacle for applications of CTA to memory-limited edge devices and proposed a novel algorithm that can significantly reduce the memory overhead at test time. The proposed algorithm enables test-time adaptation using smaller batches, and sparse layer-and-channel caches without significantly losing accuracy. The authors empirically demonstrated the outstanding accuracy and memory efficiency of the proposed method.", "summary_of_the_review": "The paper revealed and addressed an overlooked yet an important problem in continual test-time adaptation. Most claims are well supported by experiments and the methods are intuitive and well-explained. The experiments are detailed and clear. Thus, I recommend acceptance.", "strength_and_weaknesses": "Strength:\n\n1. The investigated problem is significant to real IoT scenarios. The authors reveal the overlooked low memory efficiency of traditional CTA methods, which is critical for applications on resource-constrained edge devices. Especially when considering the Internet of Things, the memory-limited tiny edge devices cannot afford 4Gb+ memory consumption for test-time adaptation as demonstrated by Fig 1.\n\n2. The technical part is novel. The proposed method is motivated by Eq. (3) that the parameter-efficient training (only updating BN layers) still brings in large memory caches for back-propagation. Based on the motivation, the authors provide intuitive solutions that can effectively adapt models without large memory consumption in three dimensions.\n\n    - For reducing batch sizes, the proposed method can stabilize the training using an adaptive memorization rate (beta) which is parameter-free and intuitive for the dynamic statistical shifts. The intuition is well supported by prior observations that the BN statistics differ by domain.\n\n    - For reducing channels in the cache, the random dropping and on-demand training can avoid forgetting resembling implicit regularization when the momentum of SGD or Adam can make up the missing gradients.\n\n    - The layer-wise training is intuitive as the adaptation typically is on demand of domain shift. \n\n3. The intuition of these methods are well supported by experiments. For instance, the use of BN-based distribution divergence as the indicator is supported by the experiment in Fig 3. The forgetting mitigation of MECTA is supported by the benchmarks on Tent.\n\n4. I appreciate the authors designed the shift-accuracy evaluation which can reveal the significant impact of accumulating batches in the transition of domains. Except for the plausible results of MECTA, the experiment itself could remind other researchers to carefully handle the domain shift at test time instead of focusing on in-domain average accuracy.\n\nWeakness:\n\n1. The authors should discuss the connection between the proposed adaptation methods and existing parameter-efficient fine-tuning works. For example, the low-rank adaptation of transformers [A].\n\n2. The authors only consider the ResNet and should discuss how the method can be extended to other model architectures.\n\n[A] LoRA: Low-Rank Adaptation of Large Language Models. Edward J. Hu*, Yelong Shen*, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "7BYMB1NJs_i", "reviewer": "Reviewer_pnAa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes MECTA in order to improve out-of-distribution model accuracy via computation-efficient online test-time gradient descents in a memory economic manner. The key idea behind MECTA is to reduce batch sizes, adopt an adaptive normalization layer to maintain stable and accurate predictions, and stop the back-propagation caching heuristically. The networks are first pruned to reduce the computation and memory overheads in optimization and the parameters are then recovered afterward to avoid forgetting. MECTA can be seamlessly plugged into state-of-the-art CTA algorithms at negligible overhead on computation and memory. On three datasets, CIFAR10, CIFAR100, and ImageNet, MECTA improves the accuracy by at least 8.5% with constrained memory and significantly reduces the memory cots of ResNet50 on ImageNet by at least 70% without sacrificing accuracy. ", "summary_of_the_review": "Overall, this paper is technically strong and novel. It solves an important problem in continual test-time adaptation with solid methodology and sound experiments. ", "strength_and_weaknesses": "Strength:\n\n1. The authors initiate a pilot study on the memory efficiency and reveal the main bottleneck is on the intermediate results cached for back-propagation, even if only few parameters need to be updated in the state-of-the-art computation-efficient solution (EATA).\n2. The proposed MECTA is very well motivated. MECTA can drastically improve the memory efficiency of gradient-based CTA. \n3. The organization and writing are very clear. The authors did a good job in reviewing all the related work, and summarizing the corresponding advantages and disadvantages.\n4. The proposed MECTA is backed by very impressive empirical results. Given limited memory constraints, MECTA improves the Tent and EATA by 8.5 − 73% accuracy on CIFAR10-C, CIFAR100-C, and ImageNet-C datasets. \n\nWeakness:\n\n1. In addition to the current related work section, I highly recommend a discussion on model adaptation from big pre-trained models on device, and explain why it’s not a viable direction.\n2. Can the authors report the computation complexity in terms of FLOPs?\n3. I understand that all the three dimensions (Reduce B, Reduce C, Dynamic L) play quite important roles in MECTA, I also want to know which dimension is the dominant one and why.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster