Title: RDumb: A simple approach that questions our progress in continual test-time adaptation
Venue: NeurIPS 2023
Accepted: yes
Keywords: test time adaptation

continual adaptation

benchmarking

imagenet-c

imagenet classification

robustness

continual learning

imagenet benchmark
Abstract: Test-Time Adaptation (TTA) allows to update pre-trained models to changing data distributions at deployment time. While early work tested these algorithms for individual fixed distribution shifts, recent work proposed and applied methods for continual adaptation over long timescales. To examine the reported progress in the field, we propose the Continually Changing Corruptions (CCC) benchmark to measure asymptotic performance of TTA techniques. We find that eventually all but one state-of-the-art methods collapse and perform worse than a non-adapting model, including models specifically proposed to be robust to performance collapse. In addition, we introduce a simple baseline, "RDumb", that periodically resets the model to its pretrained state. RDumb performs better or on par with the previously proposed state-of-the-art in all considered benchmarks.
Our results show that previous TTA approaches are neither effective at regularizing adaptation to avoid collapse nor able to outperform a simplistic resetting strategy.

RDumb: A simple approach that questions our progress in continual test-time adaptation

Abstract

Test-Time Adaptation (TTA) allows to update pre-trained models to changing data distributions at deployment time. While early work tested these algorithms for individual fixed distribution shifts, recent work proposed and applied methods for continual adaptation over long timescales. To examine the reported progress in the field, we propose the Continually Changing Corruptions (CCC) benchmark to measure asymptotic performance of TTA techniques. We find that eventually all but one state-of-the-art methods collapse and perform worse than a non-adapting model, including models specifically proposed to be robust to performance collapse. In addition, we introduce a simple baseline, 'RDumb', that periodically resets the model to its pretrained state. RDumb performs better or on par with the previously proposed state-of-the-art in all considered benchmarks. Our results show that previous TTA approaches are neither effective at regularizing adaptation to avoid collapse nor able to outperform a simplistic resetting strategy.

1 Introduction

Biological vision is remarkably robust at adapting to continually changing environments. Imagine cycling through the forest on a cloudy day and observing the world around you: You will encounter a wide variety of animals and objects, and be able to recognize them without effort. Even as the weather changes, rain sets in, or you start cycling faster, the human visual system effortlessly adapts and robustly estimates the surroundings [45]. Equipping machine vision with similar capabilities is a long-standing and unsolved challenge, with numerous applications in autonomous driving, medical imaging, and quality control, to name a few.

Techniques for improving the robustness to domain shifts of ImageNet-scale [37] classification models include pre-training of large models on diverse and/or large-scale datasets [26, 33, 50] and robustification of smaller models by specifically designed data augmentation [11, 14, 36]. While these techniques are applied during training time, recent work [8, 28, 29, 30, 35, 38, 47, 48] explored possibilities of further adapting models by Test-Time Adaptation (TTA). Such methods continuously update a given pretrained model exclusively using their input data, without having access to its labels. Test-time entropy minimization (Tent; 47) has become a foundation for state-of-the-art TTA methods. Given an input stream of images, Tent updates a pretrained classification model by minimizing the entropy of its outputs, thereby continuously increasing the model's confidence in its predictions for every input image.

Previous TTA work [8, 28, 29, 35, 38, 47, 54] evaluate their models on ImageNet-C [13] or smaller scale image classification benchmarks [18, 20]. ImageNet-C consists of 75 copies of the ImageNet validation set, wherein each copy is corrupted according to 15 different noises at 5 different severity

†

SelectNoisePairs

Conmputetrajectory

(Length L)

Random

Randomized

50k

ImageNet Val

Sampling

Crop + Flip

N

Apply

Noise

Nimages

Nimages

0 0 0

2.5

5.0

CCC

levels. When TTA models are evaluated on ImageNet-C, they are adapted on each noise and severity combination individually starting from their pretrained weights. Such a one-time adaptation approach is of little relevance when it comes to deploying TTA models in realistic scenarios. Instead, stable performance over a long run time after deployment is the desirable goal.

TTA methods are by design readily applicable to this setting and recently the field has started to move towards testing TTA models in continual adaptation settings [7, 30, 48]. Strikingly, this revealed that the dominant TTA approach Tent [47] decreases in accuracy over time, eventually being less accurate than a non-adapting, pretrained model [30, 48]. In this work, we refer to any model whose classification accuracy falls below that of a non-adapting, pretrained model, as having 'collapsed'.

This collapsing behaviour of Tent shows that it cannot be used in continual adaptation over long time scales without modifications. While previous benchmarking of TTA methods already managed to reveal the collapse of Tent, our work shows that in fact all current TTA methods collapse sooner or later, including methods with explicit built-in anti-collapse strategies .

Since current benchmarks have not been sufficient to detect collapse in several models, we introduce an image classification benchmark designed to thoroughly evaluate TTA models for their long-term behavior. Our benchmark, Continuously Changing Corruptions (CCC), tests models on their ability to adapt to image corruptions that are constantly changing, much like when fog turns to rain or day turns to night. CCC allows us to easily control different factors that could affect the ability of a given method to continuously adapt: the corruptions and their order, the difficulty of the images themselves, and the speed at which corruptions transition. Most importantly, the length of our benchmark is ten times longer than that of previous benchmarks, and more diverse by including all kinds of combinations of corruptions (see Figure 1a). Using CCC, we discover that seven recently published state-of-the-art TTA methods are less accurate than a non-adapting, pretrained model. While Tent was already shown to collapse [7, 30, 48], we show that this problem is not specific to

Tent, and that many other methods - including specifically designed continual adaptation methods collapse as well.

Finally, we propose ' RDumb ' 1 as a minimalist baseline mechanism that simply Resets the model to its pretrained weights at regular intervals. Previous work employs more sophisticated methods combining entropy minimization with various regularization approaches, yet we show that RDumb is superior on both existing benchmarks and ours (CCC). Our results call the progress made in continual TTA so far into question, and provide a richer set of benchmarks for realistic evaluation of future methods.

Our contributions are:

We introduce the continual adaptation benchmark CCC. We show that previous benchmarks are too short to meaningfully assess long-term continual adaptation behaviour, and are too uncontrolled to assess the short-term learning dynamics.

Using CCC, we show that the performances of all but one current TTA methods drop below a non-adapting, pre-trained baseline when trained over long timescales.

We propose ' RDumb ' as a baseline and show that it outperforms all previous methods with a minimalist resetting strategy.

2 CCC: Towards Infinite Testing with Continuously Changing Corruptions

Until recently, it was common to evaluate TTA methods only on datasets on individual domain shifts such as the corruptions of ImageNet-C [13]. However, the world is steadily changing and recently the community started moving towards continual adaptation, i.e., evaluating methods with respect to their ability to adapt to ongoing domain shifts [30, 42, 48].

The dominant method of evaluating continual adaptation on ImageNet scale is to concatenate the top severity datasets of the 15 ImageNet-C corruptions into one big dataset. We refer to the variant of this dataset introduced by [48] as Concatenated ImageNet-C (CIN-C). CIN-C was used to demonstrate the collapse of Tent and the stability of recent TTA methods by [7, 30, 48].

In Figure 1b, we evaluate a range of TTA methods on CIN-C and notice three potential problems: Firstly, ETA [30] appears to be stable and better than a non-adapting, pretrained baseline, but is revealed to collapse when tested on CCC. Additionally, while CoTTA[48] clearly goes down in performance, it is not yet clear whether it collapses or stabilizes above or below baseline performance. Fundamentally, CIN-C turns out to be too short to yield reliable, conclusive results. Secondly, assessing adaptation dynamics is further obscured by the considerable variations of the baseline performance among the different corruptions in CIN-C. This is not only a factor that affects adaptation itself (shown in [30, 47]), it also leads to substantial fluctuations in performance across multiple runs, making it difficult to obtain a clear and reliable assessment. Finally, CIN-C features exclusively abrupt transitions between different corruption types. In contrast, in the real world, domain changes may often be smooth and subtle with varying speeds: day to night, rain to sunshine, or the accumulation of dust on a camera. Therefore, it is important to also probe TTA methods on continual domain changes that are not tied to a specific point in time and thus constitute a relevant test for stable continual adaptation.

Here we propose a new benchmark, Continuously Changing Corruptions (CCC), to address these issues. CCC solves the issues of benchmark length, uncontrolled baseline difficulty, and transition smoothness in a simple and effective manner. Firstly, the length issue is remedied because the individual runs of CCC are constructed by a generation process which can generate very long datasets without reusing images. In this work we use runs of 7.5M images, which is 10 times as long as CIN-C. If required to compare methods in future work (where collapse is even slower), it is straightforward to generate even longer benchmarks within the CCC framework. Secondly, since both [30, 47] have shown that dataset difficulty is a confounder when studying adaptation, the difficulty of individual benchmark runs is kept stable. Additionally, we examine three different difficulty levels to ensure a comprehensive yet controlled evaluation. Finally, CCC exhibits smooth domain shifts: it applies two corruptions to each image. Over time, the severity of one corruption is smoothly increased while the

(a)

(b)

0.0

1.00.5

70

Noise

60

4.5 4.0 3.5 3.0 2.5 2.0 1.5

Noise

50

40

30

20

10

5.0

0.00.5 1.01.52.02.53.03.54.04.55.0

severity of the other is decreased, maintaining the desired difficulty. We also study three different speeds for applying this process. We will now outline the generation procedure of the dataset.

Continuously changing image corruptions To allow smooth transitions between corruptions, we introduce a more fine-grained severity level system to the ImageNet-C dataset. We interpolate the parameters of the original corruptions (integer-valued severities from 1 to 5) to finer grained severity levels from 0 to 5 in steps of 0.25. We apply two different ImageNet-C corruptions to each image, such that we can decrease the severity of one corruption while increasing the severity of another one. Hence, the corruptions of CCC are given by quadruples ( c 1 , s 1 , c 2 , s 2 ), where c 1 and c 2 are ImageNet-C corruption types and s 1 and s 2 are severity levels. When applying such a corruption, we first apply c 1 and then c 2 at their respective severities (see Figure 2a).

Calibration to desired baseline accuracy In order to control baseline accuracy, we need to know how difficult each combination of 2 noises and their respective severities is. To that end, we first select a subset of 5,000 images from the ImageNet validation set. For each corruption ( c 1 , s 1 , c 2 , s 2 ) , we corrupt all 5,000 images accordingly and evaluate the resulting images with a pre-trained ResNet-50 [10]. The resulting accuracy is what we refer to as baseline accuracy and what we use for controlling difficulty. In total, we evaluate more than 463 million corrupted images. Previous work, [12], measures normalized accuracy using AlexNet [19], which is less pertinent in present-day contexts. In addition, the accuracy of non-adapting Vision Transformers are stable on CCC as well (Figure 4).

Generating Benchmark Runs Having calibrated the corruptions pairs, we prepare benchmark runs with different baseline accuracies, transition speeds, and noise orderings. We pick 3 different baseline accuracies: 34%, 17%, and 2% (CCC-Easy, CCC-Medium, CCC-Hard respectively). For each one of the difficulties, we select a further 3 transition speeds: 1k, 2k, 5k. Lastly, for each difficulty and transition speed combination we use 3 different noise orderings, determined by 3 random seeds. To generate each run, we first select the initial corruption at the severity which according to our calibration is closest to the desired baseline accuracy. We then transition to the second corruption of the noise ordering by repeatedly either decreasing the severity of the first noise by 0.25 or increasing the severity of the second noise by 0.25 such that the baseline accuracy is as close to the target as possible (see Figure 2). In each step along each path, we sample 1k, 2k, or 5k images from the

ImageNet validation set depending on the desired transition speed. Each image is randomly cropped and flipped for increasing the diversity of the dataset, and then corrupted.

Once the path from the initial to the second corruption is finished, the process is repeated for transitioning to the third corruption and so on (for more details see Appendix B). In the end, we have 3 difficulties consisting of 9 benchmark runs each. CCC-Medium at a speed of 2k corresponds roughly to CIN-C's difficulty and transition speed.

3 RDumb: Turning your model off and on again

Continual test-time adaptation needs to successfully adapt models over arbitrarily long timescales during deployment. Resetting a model to its initial weights at fixed intervals fulfills this criterion by design, yet allows to benefit from adaptation over short time scales. Surprisingly, such an approach has never been tried before (see Appendix E for more discussion).

Regarding the choice of the adaptation loss, we build on the weighted entropy used in ETA [30]. For a stream of input images x 1 , x 2 , . . . , we compute class probabilities y t = f Θ t ( x t ) and optimize the loss function

which weights the entropy H ( y t ) = -y ⊤ t (log y t ) of each prediction using the similarity to averaged previously predicted class probabilities, y t = ( y 1 + · · · + y t ) /t , and a comparison to a fixed entropy threshold H 0 . cos( u , v ) refers to the cosine similarity between vectors u and v . At each step, (part of) the weights Θ t are updated using the Adam optimizer [17]. At every T -th step, Θ t is reset to the baseline weights Θ 0 . We use ϵ = 0 . 05 and H 0 = 0 . 4 × ln 10 3 following [30], and select T = 1000 based on the holdout noises in IN-C (see Section 6).

4 Experiment Setup

We benchmark RDumb alongside a range of recently published TTA models. For all models, we use a batch size of 64. In all models, the BatchNorm statistics are estimated on the fly, and the affine shift and scale parameters are optimized according to a model-specific strategy outlined below.

BatchNorm (BN) Adaptation [29, 38] estimates the BatchNorm statistics (mean and variance) separately for each batch at test time. The affine transformation parameters are not adapted.

Tent [46] optimizes the entropy objective on the test set in order to update the scale and shift parameters of BatchNorm (in addition to learning the statistics).

Robust Pseudo-Labeling (RPL) [35] uses a teacher-student approach in combination with a label noise resistant loss.

Conjugate Pseudo Labels (CPL) [8] use meta learning to learn the optimal adaptation objective function across a class of possible functions.

Soft Likelihood Ratio (SLR) [28] uses a loss function that is similar to entropy, but without vanishing gradients. Anti-Collapse Mechanism: An additional loss is used to encourage the model to have uniform predictions over the classes, and the last layer of the network is kept frozen.

Continual Test Time Adaptation (CoTTA) [48] uses a teacher student approach in combination with augmentations. Anti-Collapse Mechanism: Every iteration, 0.1% of the weights are reset back to their pretrained values.

Efficient Test Time Adaptation (EATA) [30] uses 2 weighing functions to weigh its outputs: the first based on their entropy (lower entropy outputs get a higher weight), the second based on diversity (outputs that are similar to seen before outputs are excluded). Anti-Collapse Mechanism: An L 2 regularizer loss is used to encourage the model's weights to stay close to their initial values.

EATA Without Weight Regularization (ETA) For completeness, we also test ETA, which is EATA but without the regularizer loss, proposed in [30].

RDumb is our proposed baseline to mitigate collapse via resetting. We reset every T = 1 , 000 steps, as determined by a hyperparameter search on the holdout set (Section 6).

(a)

(b)

Accuracy [%]

Accuracy [%]

60

35

Ours

Ours

50

ETA

30

EATA

EATA

40

25

BN

BN

20

30

CoTTA

Pretrained

15

ETA

20

Pretrained

10

SLR

10

SLR

5

CoTTA

RPL

TENT

TENT

0

CPL

RPL

CPL

0

250000

500000

750000

0.0

2.5

5.0

7.5

Observed Samples

Observed Samples

×106

Following the original implementations, Tent, ETA, EATA, and RDumb use SGD with a learning rate of 2 . 5 · 10 -4 . RPL uses SGD with a learning rate of 5 · 10 -4 . SLR uses the Adam optimizer with a learning rate of 6 · 10 -4 . CoTTA uses SGD with a learning rate of 0.01, and CPL uses SGD with a learning rate of 0.001.

5 Results

CCC reveals collapse during continual adaptation, unlike CIN-C. For three models that were evaluated, evaluation on CIN-C yielded inconclusive or inaccurate results in detecting collapse: CoTTA collapses on CCC, while CIN-C shows it to be on a downward trend, but end performance still outperformed the baseline (Figure 3a). Additionally, ETA shows no signs of collapse on CIN-C, while collapsing very clearly on CCC (Figure 3b, more precisely on CCC-Medium and CCC-Hard, see Appendix Figure 10). When tested using ViT backbones, EATA is better than the pretrained model on CIN-C (Figure 4a), but worse than the pretrained model on CCC (Figure 4b, Table 2,3). Lastly, SLR on CIN-C appears to be somewhat stable, but only at around 10% accuracy. CCC reveals this to be only partly true: on CCC-Hard, SLR is not stable and collapses to nearly chance accuracy.

(a)

(b)

Accuracy [%]

Accuracy [%]

55

65

Ours

60

Ours

50

55

45

50

Pretrained

45

40

EATA

EATA

Pretrained

40

35

35

0

250000

500000

750000

0.0

2.5

5.0

7.5

Observed Samples

Observed Samples

×106

In summary, models evaluated on CCC show clear limits, which are impossible to see on CIN-C because of the high difficulty variance between runs, and its short length.

RDumb is a strong baseline for continual adaptation. RDumb outperforms all previous methods on both established benchmarks (CIN-C, CIN-3DCC) as well as our continual adaptation benchmark, CCC (Table 1 and Figure 3). Concretely, we outperform EATA and increase accuracy by more than 11% on CIN-C (improving from 41.8% points to 46.5% points), and by almost 7% when averaged all evaluation datasets. While not able to outperform RDumb, we note that EATA is also a strong method for preventing collapse except for the counterexample in Table 2.

The results transfer to Imagenet-3D Common Corruptions. To further demonstrate the effectiveness of our method, we show results on Imagenet-3DCC [16], which features 12 types of corruptions, which take the geometry and distances between objects into account when applied to an image. Similarly to CIN-C, we test our models on 10 different permutations of concatenations of all the noises of IN-3DCC, which we call CIN-3DCC. As in the case of CIN-C and CCC, RDumb outperforms all previous methods (Table 1).

The results transfer to Vision Transformers. To further validate our claims, we test both EATA and our method with a Vision Transformer (ViT, 3) backbone (Figure 4). The difference in average accuracy between our method and EATA is larger when using a ViT, as compared to a ResNet-50: on CIN-C and CCC the gap is 10.9% points and 11.7% points respectively. Additionally, EATA's accuracy on CCC is below that of a pretrained, non-adapting model 2 . This collapse can only be seen by using CCC, and not when evaluating on CIN-C.

RDumb allows adaptation of a variety of architectures without tuning. We evaluate RDumb and EATA across a range of popular backbone architectures. Out of the nine architectures evaluated (see Table 2,3), RDumb outperformed EATA by an average margin of 4.5% points on seven of them, and worse by an average margin of only 0.25% points on the remaining two.

6 Analysis and Ablations

Optimal reset intervals. To determine the optimal reset interval, we run ETA with reset intervals T ∈ [125 , 250 , 500 , 1000 , 1500 , 2000] on CIN-C using the IN-C holdout noises. We concatenate the 4 holdout noises at severity 5 as our base test set. This base test set is repeated until the model sees 750k images, which is equal to the length of CIN-C. We do this for every permutation of the 4 holdout corruptions. On this holdout set, we find that the optimal T is equal to 1,000.

RDumb is less sensitive to hyperparameters. An added benefit to our method is that it is less sensitive to hyperparameters than EATA. We conduct a simple hyperparameter search of the E 0 parameter-the hyperparameter that controls how many outputs get filtered out because of their high entropy. Our method consistently outperforms EATA across every hyperparameter tested (Table 5), and for the highest value, 0 . 7 , EATA collapses to almost chance accuracy on all splits, while our method does not. In addition, RDumb's performance benefits from finetuning ( H 0 = { 0 . 2 , 0 . 3 } ), while EATA is not able to improve.

RDumb is effective because ETA reaches maximum adaptation performance fast. ETA is quick to adapt to new noises from scratch. On each of the holdout set noises and severities, ETA reaches its maximum accuracy after seeing only about 12,500 samples, which is about 200 adaptation steps (Figure 5a). After that, accuracy decays at a pace slower than its initial increase. Therefore, when resetting and readapting from scratch, only a few steps with substantially suboptimal predictions are encountered before performance is again close to optimal.

Comparing resetting to regularization. Previous works typically optimize two loss terms: one loss encourages adaptation on unseen data, another loss regularizes the model to prevent collapse. Having to optimize two losses should be harder than optimizing just one - we see evidence for this

(a)

(q)

100

Severity

50-

Normalized Accuracy

１２34５

Accuracy [%]

50

40

Noise

ETA (base model)

Saturate

ETA + Resetting (Rdumb)

Spatter

ETA + Regularization (EATA)

30-

Speckle Noise

0

0

1000

2000

0

1

2

3

4

Observed Samples

Adaptation Steps

in short term adaptation on CIN-C (Figure 5b): ETA and EATA optimize the same loss, but EATA additionally optimizes an anti-collapse loss. Consequently, ETA beats EATA by 2% points on CIN-C.

Collapse Analysis. We now investigate potential causes and effects of the observed collapse behavior. We propose a theoretical model, fully specified in Appendix A, which can explain both collapsing and non-collapsing runs. The model consists of a batch norm layer followed by a linear layer trained with the Tent objective. Within this model, we can present two scenarios. In the first, the model successfully adapts and plateaus at high accuracy (Figure 7a). In the second, we see early adaptation which is then followed by collapse (Figure 6a,7a). The properties of noise in the data influence whether we observe the case of successful or unsuccessful adaptation.

Interestingly, the model predicts that the magnitude of weights increases over the course of optimization; this signature of entropy minimization can be found in both the theoretical model and a real experiment using RDumb without resetting on CCC-Medium (Figure 6). Unfortunately, weight explosion happens only after model performance is already collapsed (Figure 8b). The effect is observable across all layers (Figure 6c,8).

7 Discussion and Related Work

Domain Adaptation. In practice, the data distribution at deployment is different from training, and hence the task of domain adaptation , i.e., the task of adapting models to different target distributions has received a lot of attention [6, 14, 21, 22, 36, 42, 47]. The methods on domain adaptation split into different categories based on what information is assumed to be available during adaptation. While some methods assume access to labeled data for the target distribution [27, 53], unsupervised domain adaptation methods assume that the model has access to labeled source data and unlabeled target data at adaptation time [5, 21, 22, 41]. Most useful for practical applications is the case of test-time adaptation , where the task is to adapt to the target data on the fly, without having access to the full target distribution, or the original training distribution [29, 30, 35, 38, 42, 47, 54].

In addition to the division made above, one can further distinguish what assumptions are made about how the target domain is changing. Many academic benchmarks focus on one-time distribution shifts. However, in practical applications, the target distribution can easily change perpetually over time, e.g., due to changing weather and lightness conditions, or due to sensor corruptions. Therefore, the latter setting of continual adaptation has been receiving increasing attention recently. The earliest example of adapting a classifier to an evolving target domain that we are aware of is [15], which learn a series of transformations to keep the data representation similar over time. [5, 44, 49] use an adversarial domain adaptation approach for this. [2] pointed out that two of these approaches can be prone to catastrophic forgetting [34]. To deal with this, different solutions have been proposed [1, 2, 24, 28, 30, 48].

Test-time adaptation methods have classically been applied in the setting of one-time domain change, but can be readily applied in the setting of continual adaptation, and some recent methods have been explicitly designed and tested with continual adaptation in mind [30, 47, 48]. Because TTA methods use only test data and don't alter the training procedure, they are particularly easy to apply and have

(a)

(c)

2

75

90

85

80

Toy model

accuracy

source

(clean)

三

FC

4

adaptationweights

Toy model

Accuracy [%]

20

40

Accuracyon

CCC Medium

[Nm|/1|

F 9°0

0.4

First BatchNorm

layer

[Nm|/10|

0.4

0.2

"Layer 1"

ResNet

65

70

(noisy)

target

2-

1

0

[ml|

0.2

0.0

[ml]

一

0.0

0

gradient steps

20000

40000

60000

0

gradient steps

20000

40000

60000

0

50000

100000

0

50000

100000

0

50000

100000

(b)

"Layer 2"

ResNet

"Layer 3"

ResNet

0.4 1

ResNet

"Layer 4"

BN

[Nm|/10|

0.4

[Nm|/110n| - [m|l

0.4

[Nm|/10n| - [m|l

0.2

0.2

β,Y

W

→ENT

0.2

0.0

0.0

0.0

0

50000

100000

0

50000

100000

0

50000

100000

gradient steps

gradient steps

gradientsteps

been shown to be superior to other domain adaptation approaches [6, 29, 36, 38], Therefore, we focus only on TTA methods, which we discussed in more detail in Section 4.

Continual Adaptation Benchmarks. While continually changing datasets are used in the continual learning literature, e.g. [4, 9, 25, 39, 40, 52], they have been used in TTA benchmarks only very recently. In contrast to all previous benchmarks, we want to evaluate how continual adaptation methods change over long periods of time, when the noise changes in a continuous manner. The longest datasets for TTA were made up of hundreds of thousands of labeled images in total, while we adapt to 7.5M images per run. Other datasets are comprised of short video clips [25, 39, 40] 10-20 seconds in length. Besides maximizing its length, we set out to create a dataset that is well calibrated and closely related to the well-known ImageNet-C dataset. Additionally, with our noise synthesis, we can guarantee a wide variety of noises in each one of our evaluation runs, we can control the speed at which the noise changes, and we can control the difficulty of the generated noise. Lastly, CCC accounts for different adaptation speeds, as demonstrated by [35] and [28]. They showed that training their methods on ImageNet-C for more than one epoch leads to better performance.

8 Conclusion

TTA techniques are increasingly applied to continual adaptation settings. Yet, we show that all current TTA techniques collapse in some continual adaptation settings, eventually performing worse than even non-adapting models. And while some methods are stable in some situations, they are still outperformed by our simplistic baseline 'RDumb', which avoids collapse by resetting the model to its pretrained state periodically. These observations were made possible by our new benchmark for continual adaptation (CCC), which was carefully designed for the precise assessment of long and short term adaptation behaviour of TTA methods and we envision it to be a helpful tool for the development of new, more stable adaptation methods.

Author contributions

Author order below is determined by contribution among the respective category. Conceptualization, RDumb: OP with input from all authors; Conceptualization, CCC: StS, MB, MK; Methodology: OP, StS; Software: OP; Data Curation: OP; Investigation: OP; Formal analysis: MK, StS, OP; Visualization: StS, OP with input from all authors; Writing, Original Draft: OP, StS, MK; Writing, Review & Editing: all authors; Supervision: MB, MK, StS.

References

Abnar, S., Berg, R. v. d., Ghiasi, G., Dehghani, M., Kalchbrenner, N., and Sedghi, H. (2021). Gradual domain adaptation in the wild: When intermediate distributions are absent. arXiv preprint arXiv:2106.06080 .

Bobu, A., Tzeng, E., Hoffman, J., and Darrell, T. (2018). Adapting to continuously shifting domains. Workshop Track - ICLR 2018 .

Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 .

Feng, F., Chan, R. H., Shi, X., Zhang, Y., and She, Q. (2019). Challenges in task incremental learning for assistive robotics. IEEE Access , 8:3434-3441.

Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V. (2016). Domain-adversarial training of neural networks. The journal of machine learning research , 17(1):2096-2030.

Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., and Brendel, W. (2018). Imagenet-trained cnns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint arXiv:1811.12231 .

Gong, T., Jeong, J., Kim, T., Kim, Y., Shin, J., and Lee, S.-J. (2022). Note: Robust continual test-time adaptation against temporal correlation. In Advances in Neural Information Processing Systems .

Goyal, S., Sun, M., Raghunathan, A., and Kolter, Z. (2022). Test-time adaptation via conjugate pseudo-labels. arXiv preprint arXiv:2207.09640 .

Han, J., Liang, X., Xu, H., Chen, K., Lanqing, H., Mao, J., Ye, C., Zhang, W., Li, Z., Liang, X., et al. (2021). Soda10m: A large-scale 2d self/semi-supervised object detection dataset for autonomous driving. In Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2) .

He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016 , pages 770-778. IEEE Computer Society.

Hendrycks, D., Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. (2020a). The many faces of robustness: A critical analysis of out-of-distribution generalization. ArXiv preprint , abs/2006.16241.

Hendrycks, D. and Dietterich, T. (2019a). Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint arXiv:1903.12261 .

Hendrycks, D. and Dietterich, T. G. (2019b). Benchmarking neural network robustness to common corruptions and perturbations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 . OpenReview.net.

Hendrycks, D., Mu, N., Cubuk, E. D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. (2020b). Augmix: A simple data processing method to improve robustness and uncertainty. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020 . OpenReview.net.

Hoffman, J., Darrell, T., and Saenko, K. (2014). Continuous manifold based adaptation for evolving visual domains. In Computer Vision and Pattern Recognition (CVPR) .

Kar, O. F., Yeo, T., Atanov, A., and Zamir, A. (2022). 3d common corruptions and data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18963-18974.

Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 .

Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images.

Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Bartlett, P. L., Pereira, F. C. N., Burges, C. J. C., Bottou, L., and Weinberger, K. Q., editors, Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States , pages 1106-1114.

LeCun, Y., Cortes, C., and Burges, C. (2010). Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , 2.

Lee, D.-H. (2013). Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In ICML Workshop : Challenges in Representation Learning (WREPL) .

Liang, J., Hu, D., Wang, Y., He, R., and Feng, J. (2021). Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence .

Liu, Z., Hu, H., Lin, Y ., Yao, Z., Xie, Z., Wei, Y ., Ning, J., Cao, Y ., Zhang, Z., Dong, L., Wei, F., and Guo, B. (2022). Swin transformer v2: Scaling up capacity and resolution. In International Conference on Computer Vision and Pattern Recognition (CVPR) .

Liu, Z., Miao, Z., Pan, X., Zhan, X., Lin, D., Yu, S. X., and Gong, B. (2020). Open compound domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12406-12415.

Lomonaco, V. and Maltoni, D. (2017). Core50: a new dataset and benchmark for continuous object recognition. In Levine, S., Vanhoucke, V., and Goldberg, K., editors, Proceedings of the 1st Annual Conference on Robot Learning , volume 78 of Proceedings of Machine Learning Research , pages 17-26. PMLR.

Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., and van der Maaten, L. (2018). Exploring the limits of weakly supervised pretraining. In Proceedings of the European Conference on Computer Vision (ECCV) .

Motiian, S., Jones, Q., Iranmanesh, S., and Doretto, G. (2017). Few-shot adversarial domain adaptation. Advances in neural information processing systems , 30.

Mummadi, C. K., Hutmacher, R., Rambach, K., Levinkov, E., Brox, T., and Metzen, J. H. (2021). Test-time adaptation to distribution shift by confidence maximization and input transformation. arXiv preprint arXiv:2106.14999 .

Nado, Z., Padhy, S., Sculley, D., D'Amour, A., Lakshminarayanan, B., and Snoek, J. (2020). Evaluating prediction-time batch normalization for robustness under covariate shift. ArXiv preprint , abs/2006.10963.

Niu, S., Wu, J., Zhang, Y., Chen, Y., Zheng, S., Zhao, P., and Tan, M. (2022). Efficient test-time model adaptation without forgetting. arXiv preprint arXiv:2204.02610 .

Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., et al. (2019). Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems , 32.

Prabhu, A., Torr, P. H., and Dokania, P. K. (2020). Gdumb: A simple approach that questions our progress in continual learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part II 16 , pages 524-540. Springer.

Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning , pages 8748-8763. PMLR.

Ratcliff, R. (1990). Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review , 97(2):285.

Rusak, E., Schneider, S., Pachitariu, G., Eck, L., Gehler, P. V ., Bringmann, O., Brendel, W., and Bethge, M. (2021). If your data distribution shifts, use self-learning. Transactions of Machine Learning Research .

Rusak, E., Schott, L., Zimmermann, R., Bitterwolf, J., Bringmann, O., Bethge, M., and Brendel, W. (2020). Increasing the robustness of dnns against image corruptions by playing the game of noise. ArXiv preprint , abs/2001.06057.

Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015). Imagenet large scale visual recognition challenge. International journal of computer vision (IJCV) .

Schneider, S., Rusak, E., Eck, L., Bringmann, O., Brendel, W., and Bethge, M. (2020). Improving robustness against common corruptions by covariate shift adaptation. In Advances in neural information processing systems .

Shi, X., Li, D., Zhao, P., Tian, Q., Tian, Y., Long, Q., Zhu, C., Song, J., Qiao, F., Song, L., Guo, Y., Wang, Z., Zhang, Y., Qin, B., Yang, W., Wang, F., Chan, R. H. M., and She, Q. (2020). Are we ready for service robots? the OpenLORIS-Scene datasets for lifelong SLAM. In 2020 International Conference on Robotics and Automation (ICRA) , pages 3139-3145.

Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaik, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., Vasudevan, V., Han, W., Ngiam, J., Zhao, H., Timofeev, A., Ettinger, S., Krivokon, M., Gao, A., Joshi, A., Zhang, Y., Shlens, J., Chen, Z., and Anguelov, D. (2020). Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) .

Sun, Y., Tzeng, E., Darrell, T., and Efros, A. A. (2019a). Unsupervised domain adaptation through self-supervision. ArXiv preprint , abs/1909.11825.

Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A. A., and Hardt, M. (2019b). Test-time training for out-of-distribution generalization. ArXiv preprint , abs/1909.13231.

Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., and Li, Y. (2022). Maxvit: Multi-axis vision transformer. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV , pages 459-479. Springer.

Tzeng, E., Hoffman, J., Saenko, K., and Darrell, T. (2017). Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 7167-7176.

Van de Ven, G. M. and Tolias, A. S. (2019). Three scenarios for continual learning. arXiv preprint arXiv:1904.07734 .

Wang, D., Shelhamer, E., Liu, S., Olshausen, B., and Darrell, T. (2020a). Fully test-time adaptation by entropy minimization. ArXiv preprint , abs/2006.10726.

Wang, D., Shelhamer, E., Liu, S., Olshausen, B., and Darrell, T. (2020b). Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006.10726 .

Wang, Q., Fink, O., Van Gool, L., and Dai, D. (2022). Continual test-time domain adaptation. arXiv preprint arXiv:2203.13591 .

Wulfmeier, M., Bewley, A., and Posner, I. (2018). Incremental adversarial domain adaptation for continually changing environments. In 2018 IEEE International conference on robotics and automation (ICRA) , pages 4489-4495. IEEE.

Xie, Q., Luong, M., Hovy, E. H., and Le, Q. V. (2020). Self-training with noisy student improves imagenet classification. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020 , pages 10684-10695. IEEE.

Xie, S., Girshick, R. B., Doll´ ar, P., Tu, Z., and He, K. (2017). Aggregated residual transformations for deep neural networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017 , pages 5987-5995. IEEE Computer Society.

Yu, F., Chen, H., Wang, X., Xian, W., Chen, Y., Liu, F., Madhavan, V., and Darrell, T. (2020). Bdd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) .

Yue, X., Zheng, Z., Zhang, S., Gao, Y., Darrell, T., Keutzer, K., and Vincentelli, A. S. (2021). Prototypical cross-domain self-supervised learning for few-shot unsupervised domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 13834-13844.

Zhang, M., Levine, S., and Finn, C. (2022). Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems , 35:38629-38642.

A 2D Example Experiments and Analysis

(a)

(b)

10

10

5

2

clean corrupt

data

data

5

F0

0

class 1

-0

0

class 2

5

2

-5

10 J

10 J

20

10

0

10

20

5.0

2.5

0.0

2.5

5.0

20

10

0

10

20

5.0

2.5

0.0

2.5

5.0

clean data cross entropy

corrupt data entropy

corrupt data accuracy

clean data cross entropy

corrupt data entropy

corrupt data accuracy

2.0

8

0.3

0.8

2.0

0.30

0.8

6

1.5

9

6

0.7

6

1.5

6

0.25

6

0.7

4

4

0.2

1

0.204 -

2-

+0

1.0

°0

2

0.1

0

- 0.6

0.5

2

0

1.0

0.5

2

0.15

0.10

2

0

0.6

0.5

06'0

0

0.25

0

6

5

0.90

0

0.25

0

clean data

5

5

0.85

0.20

0.85

0Z°0

corrupt data

4

0.15

4

0.15

entre

3

- 0Z°0

0.70

0.10

0.65

0.05

0.65

F 20°0

0

20000

40000

00009

0

20000

40000

60000

0

20000

40000

00009

0

20000

40000

60000

0

20000

40000

60000

0

20000

40000

60000

steps

steps

steps

steps

steps

steps

In order to better understand collapse, we constructed a simple 2D Gaussian binary classification example.

Data. The 2D datasets are constructed as follows: the data is sampled by drawing sampled from two Gaussian blobs with identical variance corresponding to the two classes N ( µ i , Σ) . For the corrupted case, the data is then rotated by an angle θ and combined with additional additive Gaussian noise N (0 , Σ corrupt ) . Finally, both in the clean and the corrupt data case, the data is rotated by an angle of -π 4 (which minimizes the effect of the batch normalization in the model).

̸

Note that for the clean data, we allocate most of the variance to the class dimension, σ 1 . On the corrupted data, we add noise primarily perpendicular to the class dimension ( ˜ σ 2 ), which is the main defining factor whether or not we observe collapsing behavior. θ c 1 = θ c 2 serves to break the symmetry between signal and noise in the data, which results in Tent starting to deviate towards the noise direction.

In our example, we parametrize this model as follows: In condition 1, µ i = ( ± 1 , 0) and Σ 2 = ( 1 0 0 0 . 03 ) , θ = -π 9 and Σ 2 corrupt = R T θ c ( 0 . 25 0 0 4 ) R θ c , where R θ c is the rotation matrix for an angle of

(a)

Accuracy [%]

40

Accuracy on

CCC Medium

[Nm|/|0|

F 9°0

0.4

layer

First BatchNorm

[Nm|/1|  [o|1

'0

ResNet

"Layer 1”

(q)

40

CCC Medium

Accuracy on

[Nm|/]|

0.4

F 9°0

First BatchNorm

layer

[n|/|  [α|

0.4

"Layer 1"

ResNet

20

0.2

0.2

20

1

0.2

0.2

三

[αl]

0

0°0

0.0

0 -

0.0 J

0.0

0

50000

100000

0

50000

100000

0

50000

100000

0

5000

10000

0

000S

10000

0

厂

5000

10000

[N|/1|  [l]

0.4

ResNet

"Layer 2”

[Nal]/|0|  |a|1

0.4

"Layer 3"

ResNet

[Nm|/1|  [n|]

0.4

ResNet

"Layer 4"

[Nm|]/[n|

0.4

"Layer 2"

ResNet

[N|/|  [|]

0.4

ResNet

"Layer 3"

[Nn|/|  [|]

0.4

ResNet

"Layer 4"

0.2

0.2

0.2

0.2

0.2

0.2

0.0

0.0

0.0

0.0

0.0

0.0

0

50000

100000

0

50000

100000

0

50000

100000

0

5000

10000

0

5000

10000

0

5000

10000

gradient steps

gradient steps

gradient steps

gradient steps

gradient steps

gradient steps

θ c = -π 6 . In condition 2, everything is the same except for Σ 2 corrupt = R T θ c ( 0 . 25 0 0 25 ) R θ c . In both conditions, we sample 300 000 data points which are equally distributed over both classes.

Model. The classification model consumes the dataset of shape N × 2 and consists of a batchnorm layer ( ϵ = 0 ) followed by a fully connected layer with two input channels, one output channels, no bias term and a sigmoid nonlinearity. Because our data is always centered, we do not learn the offset parameter of the affine adaptation in the batchnorm layer, but only the scale parameter.

Model training. The model is trained to minimize the binary cross-entropy on the clean data. We use batch gradient descent on the whole 300,000 sample dataset with a learning rate of 0.1 and no momentum. We decay the learning rate by a factor of 0.1 after 1000 and 2000 steps and stop training after 3000 steps.

Model Learning and Adaptation. We adapt the model on the corrupted data using Tent. More precisely, we optimize the scale parameter of the batch norm layer to minimize the entropy of the predictions using SGD with a learning rate of 0.01 and no momentum. We process the whole dataset in one batch and adapt for 60 000 steps.

Results. In the toy model, simple cases emerge where the loss does not result in collapse, or vice versa (Fig. 7a and Fig. 7b, respectively), mainly depending on the relation of signal and noise variances and directions.

The toy example furthermore predicts that the adapted parameters of a model should grow on the long run and indeed we were able to find exactly this effect when running ETA on a ResNet50 on CCC-Medium (Fig. 8), suggesting that our minimal setup successfully reproduces the relevant aspects of the large scale case. However, the weight explosion becomes apparent only after the collapse happens, hence weight regularization is not enough to avoid the collapse (Fig. 8(b)).

In the Fig. 7(a) example, we find the direction of high target domain performance and stay there; in this case entropy minimization is stable. In the Fig. 7(b) experiment, the domain shift adds more noise nearly orthogonal to the signal direction, which entropy minimization tries to use for making high confidence predictions: we still initially find the direction of high target domain performance, but traverse this region and continue into a direction of low entropy and low accuracy. This shows that even in a linear example, entropy minimization can show initial performance improvement and then collapse.

B Path Finding Algorithm

Algorithm 1 describes the pseudo code of the algorithm used to generate CCC. The algorithm is based on a set of Calibration Matrices, similar to the one shown in Figure 2. There exists a matrix m for every ( n a , n b ) pair, such that m [ i ][ j ] is equal to accuracy of a pretrained ResNet-50 on the

comination of noises ( n a , n b ) , and severities ( i/ 5 , j/ 5) . We will release the full set of matrices upon publication.

Additionally, Algorithm 2 uses a function MinValidPath ( s 1 , s 2 ) : this function returns the minimum path that starts at ( s 1 , s 2 ) and ends at (0 , s j ) for some s k . The cost of a path is simply the average of all entries along the path. The minimum path is defined as the path with a cost closest to b a in absolute terms. Lastly, a path is only valid if it starts with s 2 equal to 0, every transition either decreases s 1 by 0 . 25 , or increases s 2 by 0.25, and stops once s 1 is equal to 0 .

Algorithm 1 Algorithm used to generate each split of CCC

Algorithm 2 CalculatePath

The resulting dataset features transitions between two different noises like the ones seen in Figure 9 (a). We additionally plot the accuracy of a pretrained ResNet-50, alongisde the severities of the different noises in Figure 9 (b).

(a)

(b)

Snow: 0.0

Snow: 0.25

Snow: 0.5

Snow:0.5

Snow: 0.5

Snow: 0.5

Snow: 0.75

Snow: 0.75

Snow: 0.75

Snow: 0.75

Snow: 0.75

Snow: 0.75

Snow: 0.75

Snow: 0.75

Snow: 1.0

Snow: 1.0

Snow: 1.0

Snow: 1.25

Snow: 1.25

Snow: 1.5

SeveritiesandAccuraciesAlongTransitions

Severities

2

Snow: 1.75

Snow: 2.0

Snow: 2.25

Snow: 2.5

Snow: 2.75

D

Accuracy [%]

40

Snow: 3.0

Zoom Blur:0.5

Snow: 3.25

Snow: 3.25

Snow: 3.5

Snow: 3.75

20

30

60f

Mous

defocus_blur

impulse_noise

50

pixelate

Mous

10

o

100

200

300

400

500

Numberof NoiseCombinationsSeen

Snow: 4.0

Snow: 4.0

We additionally share the following metadata about the length of traversals in CCCEasy/Medium/Hard:

C CCC Plots

(a)

(b)

(c)

Accuracy [%]

Accuracy [%]

Accuracy [%]

50

Ours

EATA

40

Ours

20

40

ETA

BN

EATA

Pretrained

30

BN

15

30

20

SLR

20

Pretrained

10

EATA

10

0

CoTTA

10

D

SLR

CoTTA

TENT

ETA

RPL

D

5

Pretrained

TENT

RPL

CoTTA

CPL

ETA

0.0

2.5

5.0

7.5

0.0

2.5

5.0

7.5

0.0

2.5

5.0

7.5 CPL

Observed Samples

×106

Observed Samples

×106

Observed Samples

×106

D EATA Implementation and Ablations

Our implementation of EATA differs from the official implementation 3 . The reason for this is that the official implementation uses clean ImageNet validation images to calculate the Fisher vector matrix for its regularizer 4 . This stands in contradiction with the method, which should not have access to the training distribution at test time, as shown in the paper in Table 1.

Instead of using 2 , 000 ImageNet validation images, we calculate the Fisher matrix using the first 2 , 000 images in our data stream. We conduct a hyperparameter search on the weight regularizer tradeoff parameter β :

Using the optimal value, 100 led to worse results than the default value, 2000 , on CCC:

In the end, we used the original value of 2000, as that was optimal on the CCC dataset.

In addition, we conducted a hyperparameter search for EATA on a ViT backbone. As shown in 4, EATA performs worse than a pretrained, non adapting baseline in this setting. To that end, we tried to stabilize the model by increasing the value of β , the hyperparameter that controls the weight of the anti-forgetting regularizer. As with the previous experiment, the original value of 2000 is optimal.

E Novelty of Resetting

Our work is the first to propose resetting to solve collapse in TTA methods. Notably, while prior work [30, 47, 54] has briefly touched upon the concept of episodic resetting, the methodology and its application is significantly distinct and unrelated to collapse in TTA.

Tent [47] mentions episodic in the context of overfitting to a single sample in segmentation (similar to [54]'s overfitting to a single sample). Resetting here is unrelated to collapse, as the paper doesn't discuss collapse at all.

Although MEMO [54] uses resetting, it does so because it overfits to one image (and its augmentations) every step. MEMO doesn't discuss collapse or catastrophic forgetting. MEMO compares itself to a version of Tent that resets after every step (which they call Tent + episodic resetting), because MEMO without augmentations is similar to Tent + episodic resetting with a batch size of 1. (Note: MEMO is outperformed by BN/Tent/ETA when using the standard batch size of 64)

EATA [30] shows results for Tent + episodic resetting after every step in its tables, but provides no reasoning or discussion for doing this. Tent + episodic resetting is outperformed by regular Tent.

F CIFAR10 Experiments

We conduct CIFAR10 experiments to show the need for ImageNet scale benchmarks. Tent, without an anti-collapse mechanism, does not collapse on CIFAR10-C, even after seeing 100 million images Fig. 11a,b. Like ImageNet, CIFAR10-C's noises also exhibit high variance in difficulty Fig. 11c.

G Compute details

We conduct all experiments on Nvidia RTX 2080 TI GPUs with 12GB memory per device. All experiments except our study on larger models were conducted on a single GPU. For CoTTA experiments, we use data parallel training on 2 GPUs. A bulk of the compute spent for this work was on computing baseline accuracies on the calibration dataset, which contains 463M images.

H Software and Dataset Licenses

H.1 Datasets

ImageNet-C [12]: Creative Commons Attribution 4.0 International, https://zenodo.org/record/2235448

ImageNet-C [12], code for generating corruptions: Apache License 2.0 https://github.com/hendrycks/robustness

ImageNet-3D-CC [16]: CC-BY-NC 4.0 License https://github.com/EPFL-VILAB/3DCommonCorruptions

H.2 Models

PyTorch's [31] Backbones https://pytorch.org/vision/stable/models.html

Adaptive BN [29, 38]: Apache License 2.0, https://github.com/bethgelab/robustness

Tent [47]: MIT License, https://github.com/DequanWang/tent

RPL [35]: Apache License 2.0, https://github.com/bethgelab/robustness

CoTTA [48]: MIT License, https://github.com/qinenergy/cotta

CPL [8]: MIT License, https://github.com/locuslab/tta_conjugate

EATA [30]: MIT License https://github.com/mr-eggplant/EATA

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nTest-time adaptation techniques seek to adapt a model to new unlabelled samples without access to the original training data. However, TTA approaches are typically not evaluated for long runtimes. This work proposes a new benchmark Continuously Changing Corruptions (CCC), which is a stream of corrupted ImageNet images with gradual changes and varying degrees of difficulty. Previous approaches fail on (CCC) when compared to the pretrained model as well as a simple new baseline method dubbed RDumb. RDumb is shown to achieve high performance across several experimental settings.\n\nSTRENGTHS:\nStrengths:\n- The overall message of the paper is easy to follow and the experiments are illustrative of a serious shortcoming in the evaluation of TTA methods.\n- The baseline is simple and satisfies all the TTA assumptions, and should be considered as a baseline in future works in this area\n- Experiments are very extensive and thorough\n\nWEAKNESSES:\nWeaknesses:\n- There are a few issues related to clarity that should be addressed in the next version of the paper. See Questions\n- I would encourage the authors to add error bars to their tables, so that it is clear that the differences are statistically significant\n\nQUESTIONS:\nQuestions:\n1) The mathematical notation in Eq. 1 is a bit difficult to parse. In particular, it is unclear what is meant by $cos(y_t, \\bar{y}_t)$. Is this cosine similarity or cosine distance? I would encourage the authors to clarify this and also provide a clear description of the equation in the text.\n\n2) Can the authors precisely define \"collapse\"? This term is used but is never defined.\n\n3) In Figures 3 and 4, if RDumb is being reset to the pretrained model weights, then why do the blue and purple lines never intersect? \n\n4) It would be interesting to test CLIP image encoders [1] that are naturally robust and have high zero-shot accuracy on OOD samples. In future work, I would be curious to see how such models behave on the proposed CCC when TTA techniques are applied. \n\nReferences:\n- Learning Transferable Visual Models From Natural Language Supervision, Radford et al. 2021\n\nLIMITATIONS:\nLimitations are addressed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new benchmark for continual test-time adaptation (CTTA): CCC (Continually Changing Corruptions).\nExperiments for existing state-of-the-art approaches using their proposed benchmark demonstrate that even a non-adapting model performs better than existing approaches for this benchmark.\nFurther, they propose a simple approach as a baseline, named $\\textit{RDumb}$, which periodically resets the model to its pre-trained state, which performs better than the SOTA approaches.\n\nSTRENGTHS:\n1. This paper proposes a more challenging benchmark for CTTA. \n2. Combining a pair of corruptions is a novel idea.\n3. Experimental results demonstrate that several state-of-the-art approaches perform poorly on this benchmark.\n4. A simple periodic reset mechanism with weighted entropy (as in ETA) adaptation loss outperforms the SOTA CTTA approaches.\n\nWEAKNESSES:\n1. In CoTTA (Wang et al., CVPR 2022), there is a setting of gradually changing the corruptions (CIFAR10-to-CIFAR10C gradual). So the idea of gradually changing corruption is not entirely new, even though CCC has a combination of two corruptions.\n2. This paper does not analyze the reasons for the performance degradation of the SOTA TTA approaches, such as overfitting, error accumulation, catastrophic forgetting, or interference, and does not provide any insights or suggestions on mitigating or preventing it.\n3. Experimental results on CIFAR10-to-CIFAR10C, CIFAR100-to-CIFAR100C classification tasks, along with their CCC variants, are missing.\n\nMinor Comment: Line 62: use `` for starting quotes before RDumb\n\nQUESTIONS:\n1. In CoTTA (Wang et al., CVPR 2022), there is a setting of gradually changing the corruptions (CIFAR10-to-CIFAR10C gradual). CoTTA, as well as PETAL [1], reports performance improvement in this gradual setting, so it would be interesting to see how they would work on the CCC variant of CIFAR10-to-CIFAR10C. Do the authors have any intuition behind this?\n2. As the authors point out, the ImagenetC dataset is large and other benchmarks are much smaller; the dataset being smaller can also make the task more challenging, since there would be lesser data to learn from for a domain shift. It would be interesting to see how the performance is for CCC variants of CIFAR10-to-CIFAR10C, CIFAR100-to-CIFAR100C.\n3. Also, with regards to the semantic segmentation task, do the authors have some idea as to how this dataset/benchmark building mechanism would be applicable?\n\n\n*References*\n1. Brahma, Dhanajit, and Piyush Rai. \"A Probabilistic Framework for Lifelong Test-Time Adaptation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\nLIMITATIONS:\n1. Since the state-of-the-art approaches are being evaluated on a new benchmark dataset, it is very important to tune their hyperparameters optimally.\n2. The details regarding hyperparameter tuning for various SOTA approaches are missing.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
The paper newly introduces a new benchmark dataset for Continual Test-Time Adaptation (CTTA) named Continually Changing
Corruptions (CCC), and suggests a simple technique - repeatedly initialize the learned model weights during CTTA. CCC is composed of interpolated corruption data and its data scale varies at each timestep. They can control the difficulty of CCC data by changing the degree of interpolation of two different corruptions.

STRENGTHS:
The paper suggests a new large-scale continual test-time adaptation benchmark dataset, considering corruption interpolation, scale variation, and repetition.

WEAKNESSES:
The suggested method is too naive and not attractive in view of knowledge transfer. Periodically resetting the trainable weights to the initial state indicates simply discarding obtained knowledge and adaptive representation regardless of their importance and relevancy. Even though this remedy outperforms baselines, there is a lack of quantitative and qualitative analysis of why it happens and of the reason RDumb behaves differently from baselines. Additionally, the decision of resetting iteration is heuristic.

The suggested idea is surprising but I strongly recommend more rigorous analyses and validation of the suggested method and CCC dataset since they are counterintuitive in some sense.

QUESTIONS:
Why does RDumb consistently outperform a pre-trained model during TTA? Shouldn't its performance reach the pre-trained model's performance when re-setting the model weights to the initially pre-trained ones?

LIMITATIONS:
Please see the weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a new benchmark (dubbed CCC) for test-time adaptation, which generalizes previous corrupted imagenet benchmarks. Specifically, the CCC gradually draws a sequence of corruptions (e.g. gaussian noise or motion blur), and gradually interpolates between two consecutive corruptions, creating a stream without hard boundaries, akin to many realistic settings. By considering long sequences of corruption pairs, CCC yields a stream longer than previous benchmarks, which sheds new light on how previous methods fair in such settings. Indeed, it is shown that given a long enough stream, methods collapse to a performance worse than a non-adapted model. The authors then propose a new baseline, Rdumb, which resets to the default parameters of the pretrained model at regular, fixed intervals. Over both CCC and previous benchmarks, it is shown that Rdumb, despite its simplicity, works well.\n\nSTRENGTHS:\n1. The proposed benchmark is well-designed. I liked the use of calibration with a pretrained model to generate different stream difficulties, and the fact that this overall is stream generator can is flexible and can accommodate smooth transitions. \n2. The proposed baseline, Rdumb, is extremely simple and should be considered as a standard baseline for TTA. \n3. Overall the paper is well written and easy to follow.\n\nWEAKNESSES:\n1. The idea of resetting the model to its original pretrained state in order to combat accumulating degradation in TTA is not new (as correctly stated in the paper). Indeed, it was shown in [45] that resetting after each task boundary helps the network. For a fair comparison with COTTA, how would such method perform if the probability of resetting the weights was determined by a similar cross-validation technique as used in Section 6 ? \n2. COTTA also has some experiments with gradually changing corruptions (see Table 3). it would be good for this to be mentioned in the paper. \n\nminor comments : \n1. regarding figure 2, I would use the color red as the most severe corruptions, and yellow as the least severe. \n2. Optimal resetting interval : I think it should be Rdumb, and not Gdumb (lines 189-190)\n\nQUESTIONS:\n1. My understanding of TTA's use-cases is mostly for online settings; the deployed model receives data for which to output predictions, and this data may encounter distribution shifts. In such a setting, how would one determine the frequency of when to reset the model ?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}