RDumb: A simple approach that questions our progress in continual test-time adaptation
Authors: Ori Press
Steffen Schneider
Matthias Kuemmerer
Matthias Bethge
Venue: NeurIPS 2023 poster (2023)
Keywords: test time adaptation
continual adaptation
benchmarking
imagenet-c
imagenet classification
robustness
continual learning
imagenet benchmark
TL;DR: We rigourosly benchmark many test time adaptation methods and find a simple baseline approach to be superior.

Abstract: Test-Time Adaptation (TTA) allows to update pre-trained models to changing data distributions at deployment time. While early work tested these algorithms for individual fixed distribution shifts, recent work proposed and applied methods for continual adaptation over long timescales. To examine the reported progress in the field, we propose the Continually Changing Corruptions (CCC) benchmark to measure asymptotic performance of TTA techniques. We find that eventually all but one state-of-the-art methods collapse and perform worse than a non-adapting model, including models specifically proposed to be robust to performance collapse. In addition, we introduce a simple baseline, "RDumb", that periodically resets the model to its pretrained state. RDumb performs better or on par with the previously proposed state-of-the-art in all considered benchmarks.
Our results show that previous TTA approaches are neither effective at regularizing adaptation to avoid collapse nor able to outperform a simplistic resetting strategy.

RDumb: A simple approach that questions our progress in continual test-time adaptation
Ori Press
Steffen Schneider 1.2
Matthias Kümmerer't
Matthias Bethge
'University of Tübingen, Tübingen Al Center, Germany 2EPFL, Geneva, Switzerland
Abstract
Test-Time Adaptation (TTA) allows to update pre-trained models to changing data distributions at deployment time. While early work tested these algorithms for individual fixed distribution shifts, recent work proposed and applied methods for continual adaptation over long timescales. To examine the reported progress the field, we propose the Continually Changing Corruptions (CCC) benchmark measure asymptotic performance of TTA techniques. We find that eventually all but state-of-the-art methods collapse and perform worse than non-adapting model, including models specifically proposed to be robust to performance collapse. In addition, introduce simple baseline, "RDumb", that periodically resets the model to its pretrained state. RDumb performs better or on par with the previously proposed state-of-the-arr in all considered benchmarks. Our results show that previous TTA approaches are neither effective at regularizing adaptation to avoid collapse nor able to outperform a simplistic resetting strategy.
1 Introduction
Biological vision is remarkably robust at adapting to continually changing environments. Imagine cycling through the forest on a cloudy day and observing the world around you: You will encounter a wide variety of animals and objects, and be able to recognize them without effort. Even as the weather changes, rain sets in, or you start cycling faster, the human visual system effortlessly adapts and robustly estimates the surroundings [45]. Equipping machine vision with similar capabilities is a long-standing and unsolved challenge, with numerous applications in autonomous driving, medical imaging, and quality control, to name a few.
Techniques for improving the robustness to domain shifts of ImageNet-scale [37] classification
models include pre-training of large models on diverse and/or large-scale datasets [26, 33, 50] and
robustification of smaller models by specifically designed data augmentation [11, 14, 36). While
these techniques are applied during training time, recent work [8, 28, 29, 30, 35, 38, 47, 48] explored possibilities of further adapting models by Test-Time Adaptation (TTA). Such methods continuously update a given pretrained model exclusively using their input data, without having access to its labels. Test-time entropy minimization (Tent; 47) has become foundation for state-of-the Given an input stream of images, Tent updates a pretrained classification model by minimizing the entropy of its outputs, thereby continuously increasing the model's confidence in its predictions for every input image.
TTA methods.
Previous TTA work [8, 28, 29, 35, 38, 47, 54] evaluate their models on ImageNet-C [13] or smaller scale image classification benchmarks [18, 20J. ImageNet-C consists of 75 copies of the ImageNet validation set, wherein each copy is corrupted according to 15 different noises at 5 different severity
ori www.sseettheeeebbbccoo org. 'oint senior authors. Code: httpp://ggttti com/ wttp:/cccoooeeeeeeeeeeeee oripress/COC.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a)
CIN
CCC
(b)
Select Noise Pairs
Lomoute trajectory (Leggth
10umb
Accurco
750k imagets
COTTA
7.5
Comperan
Nois
image
Sendurtiedd
Acoracy (B6)
Aaply Noke
images
Nimages
Vimages
One transtion: NXL
75x
Seps
7500K Images
Figure 1: Continuously Changing Corruptions show limitations of existing TTA methods. (a) Comparison between ImageNet- Val, CIN-C and CCC. The proposed version CCC is 10x longer than CIN-C and could naturally be extended even further without repeating images. CCC consists of sequences of smooth transitions from one ImageNet-C noise another one. For each such pair, we construct trajectory continuously interpolating from one pure noise to the other pure noise such that baseline accuracy is kept constant. For each point along the trajectory, we sample a batch of Ik, 2k, (b) Due
or 5k images from ImageNet- Val, randomly crop and flip it and apply the noise combination. to its short length and high variability in difficulty, CIN-C (top) unable to reveal the collapse of methods such as ETA and CoTTA, while CCC (middle and bottom) can.
levels. When TTA models are evaluated on ImageNet-C, they are adapted on each noise and severity combination individually starting from their pretrained weights. Such one-time adaptation approach of little relevance when it comes to deploying TTA models in realistic scenarios. Instead, stable long run time after deployment is the desirable goal.
performance
TTA methods are by design readily applicable to this setting and recently the field has started move towards testing TTA models in continual adaptation settings [7, 30, 48]. Strikingly, this revealed that the dominant TTA approach Tent [47] decreases in accuracy over time, eventually being less accurate than a non-adapting, pretrained model [30, 48]. In this work, we refer to any model whose classification accuracy falls below that of a non-adapting, pretrained model, as having "collapsed".
This collapsing behaviour of Tent shows that it cannot be used in continual adaptation over long time scales without modifications. While previous benchmarking of TTA methods already managed to reveal the collapse of Tent, our work shows that in fact all current TTA methods collapse sooner or later, including methods with explicit built-in anti-collapse virategies.
Since current benchmarks have not been sufficient to detect collapse in several models, we introduce an image classification benchmark designed to thoroughly evaluate TTA models for their long-tent behavior. Our benchmark, Continuously Changing Corruptions (CCC), tests models on their ability adapt to image corruptions that are constantly changing, much like when fog turns rain or day turns to night. CCC allows us to easily control different factors that could affect the ability of given method continuously adapt: the corruptions and their order, the difficulty of the images themselves, and the speed at which corruptions transition. Most importantly, the length of our benchmark is ten times longer than that of previous benchmarks, and more diverse by including all kinds of combinations of corruptions (see Figure la). Using CCC, we discover that seven recently published cate-of-the-an TTA methods are less accurate than non-adapting, pretrained model. While Tent was already shown to collapse [7, 30, 48], show that this problem is not specific to

---

Tent, and that many other methods including specifically designed continual adaptation methods collapse as well.
Finally, we propose "RDumb" as a minimalist baseline mechanism that simply Resets the model to its pretrained weights regular intervals. Previous work employs more sophisticated methods combining entropy minimization with various regularization approaches, yet we show that RDumb is superior on both existing benchmarks and ours (CCC). Our results call the progress made in continual TTA so far into question, and provide a richer set of benchmarks for realistic evaluation of future methods.
Our contributions are:
We introduce the continual adaptation benchmark CCC. We show that previous benchmarks are too short to meaningfully assess long-term continual adaptation behaviour, and are too uncontrolled to assess the short-term learning dynamies.
Using CCC, we show that the performances of all but one current TTA methods drop below a non-adapting, pre-trained baseline when trained over long timescales.
We propose "RDumb" as a baseline and show that outperforms all previous methods with a minimalist resetting strategy.
2 CCC: Towards Infinite Testing with Continuously Changing Corruptions
Until recently, it was common to evaluate TTA methods only on datasets on individual domain shifts such as the corruptions of ImageNet-C [13]. However, the world is steadily changing and recently the community started moving towards continual adaptation, i.e., evaluating methods with respect to their ability adapt ongoing domain shifts [30, 42, 48].
The dominant method of evaluating continual adaptation on ImageNet scale is to concatenate the top severity datasets of the 15 ImageNet-C corruptions into one big dataset. We refer to the variant of this dataset introduced by [48] as Concatenated ImageNet-C (CIN-C). CIN-C was used to demonstrate the collapse of Tent and the stability of recent TTA methods by [7, 30, 48].
In Figure 1b, we evaluate range of TTA methods on CIN-C and notice three potential problems: Firstly, ETA [30] appears to be stable and better than non-adapting, pretrained baseline, but is revealed to collapse when tested on CCC. Additionally, while CoTTA[48] clearly goes down in performance, it is not yet clear whether it collapses or stabilizes above or below baseline performance. Fundamentally, CIN-C turns out to be too short to yield reliable, conclusive results. Secondly, assessing adaptation dynamics is further obscured by the considerable variations of the baseline performance among the different corruptions in CIN-C. This is not only factor that affects adaptation itself (shown in [30, 47J). it also leads substantial fluctuations in performance across multiple runs, making difficult to obtain a clear and reliable assessment. Finally, CIN-C features exclusively abrupt transitions between different corruption types. In contrast, in the real world, domain changes may often be smooth and subtle with varying speeds: day to night, rain to sunshine, or the accumulation of dust on a camera. Therefore, is important also probe TTA methods on continual domain changes that are not tied to a specific point in time and thus constitute a relevant test for stable continual adaptation.
Here we propose a new benchmark, Continuously Changing Corruptions (CCC), to address these issues. CCC solves the issues of benchmark length, uncontrolled baseline difficulty, and transition smoothness in a simple and effective manner. Firstly, the length issue is remedied because the individual runs of CCC without reusing images. In this work we use runs of 7.5M images, which is 10 times as long as CIN-C. If required to compare methods in future work (where collapse is even slower), it is straightforward to generate even longer benchmarks within the CCC framework. Secondly, since both [30, 47] have shown that dataset difficulty is a confounder when studying adaptation, the difficulty of individual benchmark runs is kept stable. Additionally, we examine three different difficulty levels to ensure a comprehensive yet controlled evaluation. Finally, CCC exhibits smooth domain shifts: it applies two corruptions to each image. Over time, the severity of one corruption is smoothly increased while the
constructed by a generation process which can generate very long datasets
'The name was inspired by GDumb [32].
3

---

Baseline Accuracy 10
Gaussian Noise
Gansiian Noise
95 30 23 2.0 1.5.
100
3.5 Motion Blur
4.0
45
Motion Blur
(a)
(b)
Figure 2: (a) Each corruption of CCC consists of applying two ImageNet-C corruptions at different severities. We extend the individual severities to more fine- fine-grained than in ImageNet-C. allowing for smoother noise changes, and exponentially more (noise, severity) combinations. The corners are enlarged for easier viewing, zoom in for greater detail. (b) Sample dataset sequences with a constant baseline accuracy. The sequences start from the left where Motion Blur is zeroed out, and end at the top with Gaussian noise zeroed out. The colors red, orange, and yellow correspond to trajectories in CCC-Easy, CCC-Medium and CCC-Hard, respectively.
severity of the other is decreased, maintaining the desired difficulty. We also study three different speeds for applying this process. We will now outline the generation procedure the dataset.
Continuously changing image corruptions To allow smooth transitions between corruptions, we introduce more fine-grat une-grained severity level system the ImageNet-C dataset. We interpolate the parameters of the original corruptions (integer-valued severities from I to 5) to finer grained severity levels from such that we can decrease the severity of one corruption while increasing the severity of another one. Hence, the corruptions ImageNet-C corruption types and 3 and $2 severity levels. When applying such corruption, first apply e₁ and then C2 at their respective severities (see Figure 2a).
steps of 0.25. We apply two different ImageNet-C corruptions to each image,
CCC are given by quadruples (C1, 81, 022 82), where ej and e2 are
Calibration to desired baseline accuracy In order to control baseline accuracy, need know how difficult each combination 2 noises and their respective severities is. To that end, we first select subset of 5,000 images from the ImageNet validation set. For each corruption (c1, $1, C22 82). corrupt all 5,000 images accordingly and evaluate the resulting images with pre-trained ResNet-50 [10]. The resulting accuracy is what refer baseline accuracy and what difficulty. total, we evaluate more than 463 million corrupted images. Previous work, [12], measures normalized accuracy using AlexNet [19], which is less pertinent in present-day contexts. In addition, the accuracy of non-adapting Vision Transformers are stable on CCC as well (Figure 4).
for controlling
Having calibrated the corruptions pairs, we prepare benchmark runs with different baseline accuracies, transition speeds, and noise orderings. We pick 3 different baseline accuracies: 34%, 17%, and 2% (CCC-Easy, CCC-Medium, CCC-Hard respectively). For each one of the difficulties, we select further 3 transition speeds: 1k, 2k, 5k. Lastly, for each difficulty and transition speed combination we use 3 different noise orderings, determined by 3 random seeds. To generate each run, we first select the initial corruption at the severity which according to our calibration is closest to the desired baseline accuracy. We then transition to the second corruption of the noise ordering by repeatedly either decreasing the severity of the first noise by 0.25 or increasing the severity of the second noise by 0.25 such that the baseline accuracy is as close the target as possible (see Figure 2). In each step along each path, sample 1k, 2k, or 5k images from the
Generating Benchmark Runs

---

ImageNet validation set depending on the desired transition speed. Each image is randomly cropped and flipped for increasing the diversity of the dataset, and then corrupted.
Once the path from the initial to the second corruption is finished, the process is repeated for transitioning the third corruption and have 3 difficulties consisting of 9 benchmark runs each. CCC-Medium at a speed of 2k corresponds roughly CIN-C's difficulty and transition speed.
(for more details Appendix B). In the end,
3 RDumb: Turning your model off and again
Continual test-time adaptation needs to successfully adapt models over arbitrarily long timescales during deployment. Resetting model to its initial weights at fixed intervals fulfills this criterion by design, yet allows to benefit from adaptation over short time scales. Surprisingly, such an approach has never been tried before (see Appendix for more discussion).
Regarding the choice of the adaptation loss, we build on the weighted entropy used in ETA [30]. For stream of input images X1, X2,..... we compute class probabilities y. -e,, (x+) and optimize the loss function
V = [1 cos(y,, F.-1) < c) A (H(y)) < Ho)l H(y:) exp (y(,) Ho)
(1)
which weights the entropy H (y:) ([yyy)-=u -y, (logy:) of each prediction using the similarity to averaged previously predicted class probabilities, y, = (y1 + ...... yt)/t, and comparison to fixed entropy threshold Ho. cos(u, v) refers to the cosine similarity between vectors and v. At each step, (part of) the weights O are updated using the Adam optimizer [17]. At every T-th step, O₂ is reset to the baseline weights Og. We use 0.05 and Ho 0.4 x In 10 following [30], and select T 1000 based on the holdout noises in IN-C (see Section 6).
Experiment Setup
We benchmark RDumb alongside a range of recently published TTA models. For all models, use a batch size of 64. In all models, the BatchNorm statistics are estimated on the fly, and the affine shift and scale parameters are optimized according a model-specific strategy outlined below.
BatchNorm (BN) Adaptation [2] estimates the BatchNorm statistics (mean and variance) separately for each batch at test time. The affine transformation parameters are not adapted.
Tent [46] optimizes the entropy objective on the test set in order to update the scale and shift parameters of BatchNorm (in addition to learming the statistics).
Robust Pseudo-Labeling (RPL) [35] uses a teacher-student approach in combination with a label noise resistant loss.
Conjugate Pseudo Labels (CPL) [8] use meta learning to learn the optimal adaptation objective function across a class of possible functions.
Soft Likelihood Ratio (SLR) [28] uses loss function that is similar to entropy, but without vanishing gradients. Anti-Collapse Mechanism: An additional loss is used to encourage the model to have uniform predictions over the classes, and the last layer of the network is kept frozen.
Continual Test Time Adaptation (CoTTA) [48] uses a teacher student approach in combination with augmentations. Colli-Cll Mechanism: Every iteration, 0.1% of the weights are reset back
to their pretrained values.
Efficient Test Time Adaptation (EATA) [30] uses 2 weighing functions to weigh its outputs: the first based on their entropy (lower entropy outputs get a higher weight), the second based on diversity (outputs that are similar to seen before outputs are excluded). Anti-Collape Mechanism: An L2 regularizer loss is used to encourage the model's weights to stay close to their initial values.
EATA Without Weight Regularization (ETA) For completeness, we also test ETA, which is EATA but without the regularizer loss, proposed in [30].
RDumb is our proposed baseline to mitigate collapse via resetting. We reset every T 000 steps, as determined by hyperparameter search the holdout set (Section 6).
5

---

Accuracy [%)]
Accuracy [%]]
Ours ETA EATA
Ours
30-
EATA
40
BN
BN CoTTA
Pretrained
ETA
20
Pretrained
SLR
SLR
5-
CoTTA
0-
TENT
CPL
TENT
RPL
250000
Observed Samples
Observed Samples
10F
(a)
(b)
Figure 3: Adaptation performance of all evaluated models depending the number of observed samples so far. (a) CIN-C. Model performances are averaged over the 10 runs of the benchmark. (b) CCC. Model performances averaged over the 27 runs of the three difficulty levels. See Appendix C, Figure 10 for separate plots for CCC Easy, Medium and Hard.
Following the original implementations, Tent, ETA, EATA, and RDumb use SGD with a learning rate of 2.5 10 RPL uses SGD with learning rate of 10 10-4 SLR uses the Adam optimizer with learning rate of 6 10 CoTTA uses SGD with learning rate of 0.01, and CPL uses SGD with learning rate of 0.001.
Results
CCC reveals collapse during continual adaptation, unlike CIN-C. For three models that were evaluated, evaluation on CIN-C yielded inconclusive or inaccurate results in detecting collapse: CoTTA collapses CCC, while CIN-C shows to be downward trend, but end performance still outperformed the baseline (Figure 3a). Additionally, ETA shows no signs of collapse on CIN-C, while collapsing very clearly on CCC (Figure 3b, more precisely CCC-Medium and CCC-Hard, see Appendix Figure 10). When tested using VIT backbones, EATA is better than the pretrained model on CIN-C (Figure 4a), but worse than the pretrained model on CCC (Figure 4b, Table 2,3). Lastly, SLR on CIN-C appears to be somewhat stable, but only at around 10% accuracy. CCC reveals this to be only partly true: CCC-Hard, SLR is not stable and collapses nearly chance accuracy.
Table Mean accuracy ResNet-50 models on CIN-C, CIN-3DCC and CCC. For each CCC split (Easy, Medium, and Hard), mean of 9 runs is taken. For the CIN-C and CIN-3DCC experiments, the accuracy reported is the mean of 10 different noise permutations. Grey indicates collapse.
Adaptation method
CIN-C
CIN-3DCC
CCC-Easy
CCC-Medium
CCC-Hard
Average
Pretrained [10]
18.0 0.0
31.5
34.1 = 0.22
17.3 +: 0.21
1.5 +: 0.02
20.5
BN [29, 38]
31.5 0.02
0.02
42.6 0.39
27.9 119:000 0.74
6.8 + 0.31
28.9
Tent [47]
15.6
24.4
3.9 0.58
1.110.17
0.51 4:
9.2
RPL [35]
2.66
3..0
7.5 0.83
2.3
0.67 067:0.1 0.14
12.5
SLR [28]
1.74
1.72
22.2 18.4
0.66 7.7
11.0
CPL [8]
3.3
5.7
0.41 0.06
0.22 0.03
0.14
1.9
CoTTA [48]
34.0 0.68
0.66
14.9 = 0.88
7.7 0.43
1.1
19.1
EATA [30]
41.8
43.6
48.2 + 0.6
35.4
8.7 +0.8
35.5
ETA [30]
43.8 0.33
42.7 + 0.33
41.4 = 0.95
L1
0.23 + 0.05
25.8
RDumb (ours)
46.5 0.15
45.2 1 0.15
49.3 = 0.88
38,9 1 1.4
9.6 16
37.9

---

Accuracy [%)]
Accuracy [%6]
m o Ma My
Anp o pM
M
Ours
Ours
15
Mnh MI
w/ May
Pretrained
EATA
EATA Pretrained
35
2500000
500000
7500000
0.0
2.5
5.0
7.5
Observed Samples
Observed Samples
X110
(a)
(b)
Figure 4: TTA using VIT backbone: (a) On CIN-C, EATA is better than the pretrained baseline (44.4% points vs 40. % points). (b) CCC-Medium, EATA worse than the pretrained baseline (38.5% points vs 42.0% points). RDumb (ours) is consistently better than both EATA and the baseline.
Table 2: Mean accuracy of different backbone architectures on CCC-Medium. Accuracy reported is an average across runs. Backbones used: [3, 10, 23, 43, 51], t: AugMix [14], f DeepAugment [11]. Grey indicates collapse.
RNX(101 #
MaxViT-T
SwinVIT-T
Method
RN18
RN34
RN50
RNS01
RN
ViT-B16
Pretrained
12.2
17.3
17.3
27.9
38,9
47.8
42,0
45.1
33.2
EATA
26.8
30.8
35.4
46.5
52.3
58.5
38.5
47.1
35.6
RDumb
32,5
37.2
38.9
47.0
51.9
58.4
50.2
49.9
36.5
In summary, models evaluated on CCC show clear limits, which are impossible to see on CIN-C because of the high difficulty variance between runs, and its short length.
RDumb is a strong baseline for continual adaptation. RDumb outperforms all previous methods on both established benchmarks (CIN-C, CIN-3DCC) as well as our continual adaptation benchmark, CCC (Table and Figure 3). Concretely, we outperform EATA and increase accuracy by more than 11 11% on CIN-C (improving from 41.8% points to 46.5% points), and by almost 7% when averaged all evaluation datasets. While not able to outperform RDumb, we note that EATA is also strong method for preventing collapse except for the counterexample in Table 2.
The results transfer Imagenet-3D Common Corruptions. To further demonstrate the effective- ness of our method, we show results on Imagenet-3DCC [16], which features 12 types of corruptions, which take the geometry and distances between objects into account when applied an image. Simi- larly to CIN-C, we test our models on 10 different permutations of concatenations of all the noises of IN-3DCC, which we call CIN-3DCC. As the case of CIN-C and CCC, RDumb outperforms all previous methods (Table 1).
The results transfer to Vision Transformers. To further validate claims, test both EATA and our method with a Vision Transformer (ViT, 3) backbone (Figure 4). The difference in average accuracy between
method and EATA is larger when using a VIT, compared ResNet-50: on CIN-C and CCC the gap is 10.9% points and 11.7% points respectively. Additionally, EATA's accuracy on CCC is below that of pretrained, non-adapting model This collapse can only be seen by using CCC, and not when evaluating on CIN-C.
"Inreessing the regularizer parameter value does not help stabilize the model, Appendix D.

---

Table 3: Mean accuracy of different backbone architectures CCC-Hard. Accuracy reported is average across 9 runs. Backbones used: [3, 10, 23, 43, 51], t: AugMix [14], f: DeepAugment [11]. Grey indicates collapse.
RNX1101 11
Method
RN18
RN34
RN50
RNS01
RN50 f$
VIT-B16
MaxViT-T
SwinViT-T
Pretrained
0.82
1.3
1.5
15.6
22.0
22.0
9.3
EATA
7.6
8.7
17.7
30.3
36.3
15.4
8.6
RDumb
8.3
10.7
9.6
14.7
29.9
35.6
23.8
22.0
8.0
RDumb allows adaptation of a variety of architectures without tuning. We evaluate RDumb and
EATA across range of popular backbone architectures. Out of the nine architectures evaluated (see Table 2,3), RDumb outperformed EATA by an average margin of 4.5% points on seven of them, and worse by an average margin of only 0.25% points the remaining two.
Analysis and Ablations
Optimal reset intervals. To determine the optimal reset interval, we run ETA with reset intervals T E [125, 250, 500, 1000, 1500, 2000] on CIN-C using the IN-C holdout noises. We concatenate the 4 holdout noises at severity as our base test set. This base test set repeated until the model sees 750k images, which is equal the length of CIN-C. We do this for every permutation of the holdout corruptions. On this holdout set, we find that the optimal T is equal to 1,000.
Table Accuracy of our method for different resetting times on CIN-C-Holdout
T (steps)
125
250
500
1000
1500
2000
Acc. [%]
42.1
44.4
46.0
46.7
46.5
46.4
RDumb less sensitive to hyperparameters. An added benefit to our method that is less sensitive to hyperparameters than EATA. We conduct simple hyperparameter search of the Eg parameter-the hyperparameter that controls how many outputs get filtered out because of their high entropy. Our method consistently outperforms EATA across every hyperparameter tested (Table 5), and for the highest value, 0.7, EATA collapses to almost chance accuracy on all splits, while our method does not. In addition, RDumb's performance benefits from finetuning (Ho {0.2, 0.3)), while EATA is not able to improve.
Table 5: Average accuracy on all of CCC splits on variety of H, values. For all other experiments this paper we use Ho 0.4 103, as [30].
Ho 10
0.1
0.2
0.3
0.4
0.5
0.6
0.7
EATA RDumb (ours)
27.8
27.9
29.9
30.8
28.7
28.0
0.33
31.6
32.9
33.1
32.6
30.7
25.7
16.8
RDumb is effective because ETA reaches maximum adaptation performance fast. ETA is quick to adapt to new noises from scratch. On each of the holdout set noises and severitics, ETA reaches its maximum accuracy after seeing only about 12,500 samples, which is about 200 adaptation steps (Figure 5a). After that, accuracy decays at a pace slower than its initial increase. Therefore, when resetting and readapting from scratch, only few steps with substantially suboptimal predictions encountered before performance is again close to optimal.
Comparing resetting to regularization. Previous works typically optimize two loss terms: one loss encourages adaptation on unseen data, another loss regularizes the model to prevent collapse. Having to optimize two losses should be harder than optimizing just one see evidence for this

---

(a)
(b)
Normalized Accuracy [96]
100
Severity
50
Accuracy [%)
50
40
Noise
ETA (base model)
Blur
Saturate Spatter
ETA Resett ne (Rdumb)
ETA Regularization (EATA)
Speckle
30
5 5x
1000
2000
Observed Samples
Adaptation Steps
Figure 5: (a) ETA'S normalized accuracy over time, the ImageNet-C holdout noises and each of their severities. For every noise in the holdout set, ETA reaches its maximum accuracy very quickly. (b) Rdumb shares ETA's property of fast adaptation, while regularization in EATA slows adaptation.
in short term adaptation on CIN-C (Figure 5b): ETA and EATA optimize the same loss, but EATA additionally optimizes an anti-collapse loss. Consequently, ETA beats EATA by 2% points on CIN-C.
Collapse Analysis. We now investigate potential causes and effects of the observed collapse behavior. We propose theoretical model, fully specified in Appendix A, which can explain both collapsing and non-collapsing runs. The model consists of a batch norm layer followed by a linear layer trained with the Tent objective. Within this model, can present two scenarios. In the first, the model successfully adapts and plateaus at high accuracy (Figure 7a). In the second, we see early adaptation which then followed by collapse (Figure 6a,7a). The properties of noise the data influence whether we observe the case of successful or unsuccessful adaptation.
Interestingly, the model predicts that the magnitude of weights increases the course of opti- mization; this signature of entropy minimization can be found in both the theoretical model and a real experiment using RDumb without resetting on CCC-Medium (Figure 6). Unfortunately. weight explosion happens only after model performance is already collapsed (Figure 8b). The effect is observable across all layers (Figure 6c,8).
7 Discussion and Related Work
Domain Adaptation. In practice, the data distribution at deployment is different from training, and hence the task of domain adaptation, i.e., the task of adapting models to different target distributions has received lot of attention [6. 14, 21, 22, 36, 42, 47]. The methods on domain adaptation split into different categories based on what information assumed to be available during adaptation. While some methods assume access labeled data for the target distribution [27,53], unsupervised domain adaptation methods assume that the model has access to labeled source data and unlabeled target data at adaptation time [5, 21, 22, 41J. Most useful for practical applications is the case of test-time adaptation, where the task is to adapt to the target data on the fly, without having access to the full target distribution, the original training distribution [29, 30, 35, 38, 42, 47, 54].
In addition to the division made above, one can further distinguish what assumptions are made about how the target domain is changing- Many academic benchmarks focus one-time distribution shifts. However, in practical applications, the target distribution can easily change perpetually over time, e.g., due to changing weather and lightness conditions, due to sensor corruptions. Therefore, the latter setting of continual adaptation has been receiving increasing attention recently. The earliest example of adapting a classifier to evolving target domain that we are aware of is [15], which learn a series of transformations to keep the data representation similar over time. [5, 44, 49] use an adversarial domain adaptation approach for this. [2] pointed out that two of these approaches can be prone to catastrophic forgetting [34]. To deal with this, different solutions have been proposed [1, 2, 24, 28, 30, 48].
Test-time adaptation methods have classically been applied in the setting of one-time domain change, but can be readily applied in the setting of continual adaptation, and some recent methods have been explicitly designed and tested with continual adaptation mind [30, 47, 48]. Because TTA methods only test data and don't 't alter the training procedure, they are particularly easy to apply and have

---

(8)
(c)
Tay model
model
Aocuracy on CCC Medium
First Narmh
Reshiet
Aoarwry (e)
abcur aoy
adaptation weights
a)
llv 1oy mal
0.6
Layer
Accussy (d
62-
60
gracient staps
greciient stops
50000
100000
2000
.00300
S0000
100000
(b)
Hastett
Wesfeet 'Layer
Rechart
Layer
'Layer
BN
Lin
[ee adlihel
ENT
50000
200000
gradieent 100000
50000
Figure 6: Analysis of entropy minimization collapse on synthetic and real data. Learning dynamics in terms of accuracy and weight magnitude are shown in (a) for two layer toy model consisting of batch norm and linear layer (b). Consistent with the theoretical analysis, we find that the adaptation weights in all layers increase over time continually (c), even long after the collapse as indicated by Accuracy on CCC-Medium has happened. Refer to Figure 7-8 and Appendix for additional properties of the toy model (a-b) and a zoomed-in view on (c).
been shown to be superior to other domain adaptation approaches [6, 29, 36, 38), Therefore, we focus only TTA methods, which discussed in more detail in Section 4.
Continual Adaptation Benchmarks. While continually changing datasets are used in the continual learning literature, e.g. [4, 9, 25, 39, 40, 52], they have been used in TTA benchmarks only very recently. In contrast to all previous benchmarks, we want to evaluate how continual adaptation methods change over long periods of time, when the noise changes in continuous manner. The longest datasets for TTA were made up of hundreds of thousands of labeled images in total, while we adapt 7.5M images per run. Other datasets are comprised of short video clips [25, 39, 40] 10-20 seconds length. Besides maximizing its length, we set out to create a dataset that is well calibrated and closely related to the well-known ImageNet-C dataset. Additionally, with our noise synthesis, can guarantee a wide variety of noises in each one of our evaluation runs, we can control the speed which the noise changes, and we can control the difficulty of the generated noise. Lastly, CCC accounts for different adaptation speeds, as demonstrated by [35] and [28]. They showed that training their methods ImageNet-C for more than one epoch leads better performance.
8 Conclusion
TTA techniques are increasingly applied to continual adaptation settings. Yet, we show that all current TTA techniques collapse in some continual adaptation settings, eventually performing worse than even non-adapting models. And while some methods are stable in some situations, they are still outperformed by our simplistic baseline "RDumb", which avoids collapse by resetting the model to its pretrained state periodically. These observations were made possible by our new benchmark for continual adaptation (CCC), which was carefully designed for the precise assessment of long and short term adaptation behaviour of TTA methods and we envision it to be helpful tool for the development of new, more stable adaptation methods.
10

---

Acknowledgements
We thank Evgenia Rusak, Çagatay Yildız, Shyamgopal Karthik, and Ofir Press for helpful discussions and feedback on the manuscript.
We thank the International Max Planck Research School for Intelligent Systems (IMPRS-IS) for supporting OP and StS; StS acknowledges his membership in the European Laboratory for Learning and Intelligent Systems (ELLIS) PhD program. StS was supported by a Google Research PhD Fellowship. MB is a member of the Machine Learning Cluster of Excellence, EXC number 2064/1 Project No 390727645 and acknowledges support by the German Research Foundation (DFG): SFB 1233, Robust Vision: Inference Principles and Neural Mechanisms, TP 4, Project No: 276693517. This work was supported by the Tübingen Al Center. The authors declare no conflicts of interests.
Author contributions
Author order below is determined by contribution among the respective category. Conceptualization, RDumb: OP with input from all authors; Conceptualization, CCC: Sts, MB, MK; Methodology: OP, StS; Software: OP: Data Curation: OP: Investigation: OP; Formal analysis: MK, Sts, OP:
Visualization: StS. OP with input from all authors: Writing, Original Draft: OP, StS, MK; Writing, Review & Editing: all authors; Supervision: MB, MK, StS.
References
[1] Abnar, S., Berg, R. v. d., Ghiasi, G., Dehghani, M., Kalchbrenner, N., and Sedghi, H. (2021). Gradual domain adaptation the wild: When intermediate distributions are absent. arXiv preprint vnXiv:2/6606608
[2] Bobu, A., Tzeng, E., Hoffman, J., and Darrell, T. (2018). Adapting to continuously shifting domains. Workshop Track ICLR 2018.
[3] Dosovitskiy, A., Beyer, L.., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T. De- hghani, M., Minderer, M., Heigold, G., Gelly, S., et al. (2020). An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929.
[4] Feng, F., Chan, R. H. Shi, X., Zhang, Y., and She, Q. (2019). Challenges in task incremental learning for assistive robotics. IEEE Access, 8:3434-3441.
[5] Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H., Laviolette, F., Marchand, M., and Lempitsky, V. (2016). nomain-adversariaa training of neural networks. The journal of machine learning research,
[6] Geirhos, R., Rubisch, P., Michaelis, C., Bethge, M., Wichmann, F. A., and Brendel, W. (2018). Imagenet-trained enns are biased towards texture; increasing shape bias improves accuracy and robustness. arXiv preprint
[7] Gong, T., Jeong, J., Kim, T., Kim, Y., Shin, J., and Lee, S.-J. (2022). Note: Robust continual test-time adaptation against temporal correlation. In Advances in Neural Information Processing Systems.
[8] Goyal, S., Sun, M., Raghunathan, A., and Kolter, Z. (2022). Test-time adaptation via conjugate pseudo-labels. arXiv preprint wxXx:22077096..4
[9] Han, J., Liang, X., Xu, H., Chen, K., Lanqing, H., Mao, J., Ye, C., Zhang, W., Li, Z., Liang, X., al. (2021). Sodal0m: A large-scale 2d supprseei- object detection dataset for autonomous driving, In Thirty- Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
[10] He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV. USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
11

---

[11] Hendrycks, D., Basart, S., Mu. N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. (2020a). The many faces of robustness: A critical analysis of out-of-distribution generalization. ArXiv preprint, abs/2006.16241.
[12] Hendryeks, D. and Dietterich, T. (2019a). Benchmarking neural network robustness common corruptions and perturbations. arXiv preprint arXiv:/903: 12261.
[13] Hendrycks, D. and Dietterich, T. G. (2019b). Benchmarking neural network robustness to com- mon corruptions and perturbations. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
[14] Hendrycks, D., Mu, N., Cubuk, D., Zoph, B., Gilmer, J., and Lakshminarayanan, B. (2020b). Augmix: A simple data processing method to improve robustness and uncertainty. In 8th Interna- tional Conference on Learning Representations, ICLR 2020, Addis Ababa. Ethiopia, April 26-30, 2020. OpenReview.net.
[15] Hoffman, J., Darrell, T., and Saenko, K. (2014). Continuous manifold based adaptation for evolving visual domains. In Computer Vision and Puttern Recognition (CVPR).
[16] Kar, O. F. Yeo, T., Atanov, A., and Zamir, A. (2022). 3d common corruptions and data augmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18963-18974.
[17] Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
[18] Krizhevsky, A., Hinton, G., et al. (2009). Learning multiple layers of features from tiny images.
[19] Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classification with deep convolutional neural networks. In Bartlett, P. L.,, Pereira, F. C. N., Burges, C. C., Bottou, L.,, and Weinberger, K. Q., editors, Advances Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of meeting held December 3-6, 2012, Lake Tahwe, Nevada, United States, pages 1106-1114.
[20] LeCun, Y., Cortes, C., and Burges, C. (2010). Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecmn cocrem.coo//xxa/mnniss... 2.
[21] Lee, D. (2013). Pscudo-label: The simple and efficient semi semi-supervised learning method for deep neural networks. In ICML Workshop Challenges in Representation Learning (WREPL).
[22] Liang, J., Hu, D.,. Wang, Y., He, R., and Feng, (2021). Source data-absent unsupervised domain adaptation through hypothesis transfer and labeling transfer. IEEE Transactions on Pattern Analysis and Machine Intelligence.
[23] Liu, Z., Hu, H., Lin, Y., Yao, Z., Xie, 7., Wei, Y., Ning, J., Cao, Y., Zhang, Z., Dong, L.,, Wei, F, and Guo, B. (2022). Swin transformer v2: Scaling up capacity and resolution. In International Conference on Computer Vision and Pattern Recognition (CVPR).
[24] Liu, 7., Miao, 7_, Pan, X., Zhan, X., Lin, D., Yu, S. X., and Gong, B. (2020). Open compound domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12406-12415.
[25] Lomonaco, V. and Maltoni. D. (2017). Core50: a new dataset and benchmark for continuous object recognition. In Levine, S., Vanhoucke, V., and Goldberg, K., editors. Proceedings of the 1st Annual Conference on Robot Learning, volume 78 of Proceedings of Machine Learning Research, pages 17-26. PMLR.
[26] Mahajan, D., Girshick, R., Ramanathan, V., He, K., Paluri, M., Li, Y., Bharambe, A., and van der Maaten, L. (2018). Exploring the limits of weakly supervised pretraining. In Proceddings of the European Conference on Computer Vision (ECCV).
[27] Motiaan, S., Jones, Q., Tranmanesh, S., and Doretto, G. (2017). Few-shot adversarial domain adaptation. Advances in neural information processing systems, 30.
12

---

[28] Mummadi, C. K., Hutmacher, R., Rambach, K., Levinkov, E., Brox, T., and Metzen, J. H. (2021). Test-time adaptation to distribution shift by confidence maximization and input transformation. arXiv preprint arXiv:2 14999.
[29] Nado, Z., Padhy, S., Sculley, D., D' Amour, A., Lakshminarayanan, B., and Snoek, (2020). Evaluating prediction-time batch normalization for robustness under covariate shift. ArXiv preprint, abs/2006. 10963.
[30] Niu, S., Wu, J., Zhang. Y., Chen, Y., Zheng, S., Zhao, P., and Tan, M. (2022). Efficient test-time model adaptation without forgetting. arXiv preprint arXiv:2204.02M) 026/0.
[31] Paszke, A., Gross, S., Massa, F. Lerer, A.,
Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L.., et al. (2019). Pytorch: An imperative style, pigh-performance deep learning library. Advances in neural information processing systems, 32.
[32] Prabhu, A., TOrI, P. H., and Dokania, P. K. (2020). Gdumb: A simple approach that questions our progress in continual learning. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part 16, pages 524-540. Springer.
[33] Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. (2021). Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pages 7748-8763, PMLR.
[34] Rateliff, R. (1990). Connectionist models of recognition memory: constraints imposed by learning and forgetting functions. Psychological review, 97(2))285.
[35] Rusak, E., Schneider, S., Pachitariu, G., Eck, L.., Gehler, V., Bringmann, O., Brendel, W., and Bethge, M. (2021). If your data distribution shifts, use self-learning. Transactions of Machine Learning Research.
(36] Rusak, E., Schott, L.,, Zimmermann, R., Bitterwolf, J., Bringmann, O., Bethge, M., and Brendel, W. (2020). Increasing the robustness of dnns against image corruptions by playing the game of noise. ArXiv preprint, abs/2001 100057..
[37] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., et al. (2015). Imagenet large scale visual recognition challenge. International journal of computer vision (IJCV).
[38] Schneider, S., Rusak, E., Eck, L., Bringmann, O., Brendel, W., and Bethge, M. (2020). Improv- ing robustness against common corruptions by covariate shift adaptation. In Advances in neural information processing systems.
[39] Shi, X., Li, D., Zhao, P., Tian, Q. Tian, Y., Long, Q.. Zhu, C., Song. J., Qiao, F., Song, L., Guo, Y., Wang, Z., Zhang, Y., Qin, B., Yang, W., Wang, F., Chan, R. H. M., and She, Q. (2020). Are we ready for service robots? the OpenLORIS Scene datasets for lifelong SLAM. 2020 International Conference on Robotics and Automation (ICRA), pages 3139-3145.
[40] Sun, P., Kretzschmar, H., Dotiwalla, X., Chouard, A., Patnaík, V., Tsui, P., Guo, J., Zhou, Y., Chai, Y., Caine, B., Vasudevan, V., Han, W., Ngiam, J., Zhao, H., Timofeev, A., Ettinger, S., Krivokon, M., Gao, A., Joshi, A., Zhang, Y., Shlens, J., Chen, Z., and Anguelov, D. (2020). Scalability in perception for autonomous driving: Waymo open dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[41] Sun, Y., Tzeng, E.,, Darrell, T., and Efros, A. A. (2019a). Unsupervised domain adaptation through self supe-uupevision. ArXiv preprint, 0tts/199..11888
[42] Sun, Y., Wang, X., Liu, Z., Miller, J., Efros, A. A., and Hardt, M. (2019b). Test-time training for out-of-distributiin generalization. ArXiv preprint, abs/1909, 13231.
[43] Tu, Z., Talebi, H., Zhang, H., Yang, F., Milanfar, P., Bovik, A., and Li, Y. (2022). Maxvit: Muiti-axis vision transformer. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIV, pages 459 459-479. Springer.
13

---

[44] Tzeng, E., Hoffman, J., Saenko, K., and Darrell, T. (2017). Adversarial discriminative domain adaptation. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 7167-7176.
[45] Van de Ven, G. M. and Tolias, A. S. (2019). Three scenarios for continual learning. arXiv preprint 19X4v:1904.07734.
[46] Wang, D., Shelhamer, E., Liu, S., Olshausen, B., and Darrell, T. (2020a). Fully test-time adaptation by entropy minimization. ArXiv preprint, abs/2006. 1ess2000..0026...
[47] Wang, D., Shelhamer, E., Liu, S., Olshausen, B., and Darrell, T. (2020b). Tent: Fully test-time adaptation by entropy minimization. arXiv preprint urXi:2206.11722
[48] Wang, Q., Fink. O.. Van Gool, L., and Dai, D. (2022). Continual test-time domain adaptation. arXiv preprint arXiv:2203. 13591.
[49] Wulfmeier, M., Bewley, A., and Posner, (2018). Incremental adversarial domain adaptation for continually changing enviromments. In 2018 IEEE International conference robotics and automation (ICRA). pages 4489 4495. IEEE.
[50] Xie, Q.. Luong, M., Hovy, E. H.,, and Le, Q. V. (2020). Self-training with noisy student improves imagenet classification. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 10684-10695. IEEE
[51] Xie, S., Girshick, R. B., Dollár, P., Tu, Z., and He, K. (2017). Aggregated residual transfor- mations for deep neural networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, Honolulu, HI, USA, July 21-26, 2017, pages 5987-5995. IEEE Computer Society.
[52] Yu, F., Chen, H., Wang. X., Xian, W., Chen, Y., Liu, F., Madhavan, V., and Darrell, T. (2020). 1dd100k: A diverse driving dataset for heterogeneous multitask learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).
[53] Yue, X., Zheng, Z.,, Zhang, S., Gao, Y., Darrell, T., Keutzer, K., and Vincentelli, A. S. (2021). Prototypical cross-domain self-supervised learning for few-shot unsupervised domain adaptation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13834-13844.
[54] Zhang, M., Levine, S., and Finn, C. (2022). Memo: Test time robustness via adaptation and augmentation. Advances in Neural Information Processing Systems, 35:38629-38642.
14

---

A 2D Example Experiments and Analysis
comrugt scuutdy
1.
13
108
300
0.5
0.00
dlean data co 5213
Figure 7: Theoretical analysis of adaptation under distribution shift. Collapsing non-collapsing behavior of entropy minimization can be reproduced with simple 2d Gaussian binary classification example, domain shift which slightly rotates the data and adds Gaussian noise, and model which consists of batch norm layer followed by logistic regression. Top: clean and corrupt data for two classes before (a) and after (b) batch norm. Middle: learning dynamics of entropy minimization in the 2d adaptation parameter space starting from initial parameters (yellow marker) over time. Bottom: accuracy, entropy, and size of adaptation weights over time.
In order to better understand collapse, constructed simple 2D Gaussian binary classification example.
The 2D datasets are constructed as follows: the data is sampled by drawing sampled from two Gaussian blobs with identical variance corresponding to the two classes M(,, E). For the corrupted case, the data is then rotated by an angle 0 and combined with additional additive Gaussian noise N(0 Ecorup)). Finally, both in the clean and the corrupt data case, the data is rotated by an angle of I (which minimizes the effect of the batch normalization in the model).
Data.
Y~ Ber((5.5)
(2)
X ~ N(N, E)
(3)
Xelean
R_
(4)
Xconupe N(Ro,, Xeleans R_ R,, -Ro,, ERg R_
(5)
with E
01
0
01 >> 02
(6)
02
,
0
Σ
02 2 σ1
(7)
02
cos (
sin 0 cos(O)
Rg
(8)
sin
Note that for the clean data, we allocate most of the variance to the class dimension, T1. On the corrupted data, we add noise primarily perpendicular the class dimension (ō2). which is the main defining factor whether or not we observe collapsing behavior. O₂, + 0c, serves to break the symmetry between signal and noise in the data, which results in Tent starting deviate towards the noise direction.
In example, parametrize this model follows: In condition 1, 14 (=1,0) and Σ² (b (.0z), ), 0 and Eeomupt RY, (0.25 9 Re., where Ra, is the rotation matrix for angle of
15

---

(a)
(b)
CDC Medur
lyper
CDC Meduan
laye
"Lyye !
1000
2000
Sin
libre
DM
LOVg
Longe
sestker
Rester
Recker
Resske
gracken Mtaa
gradentiter steze
gradianc SEACE
gradket
grockent stugs
Figure 8: Analysis of entropy minimization collapse on real data. (a) Consistent with the theoretical analysis in Fig. 7, we find that the adaptation weights in all layers increase over time continually, even long after the collapse as indicated by Accuracy on CCC-Medium has happened. (b) shows zoomed-in view where this increase not yet apparent, well after the collapse.
0.25 0 Ro.- In both 0- = - 6- In condition 2, everything is the same except for E = R d 25 conditions, we sample 300 000 data points which are equally distributed over both classes.
Model. The classification model consumes the dataset of shape N and consists of batchnorm layer (c 0) followed by a fully connected layer with two input channels, one output channels, no bias term and a sigmoid nonlinearity. Because our data is always centered, we do not learn the offset parameter of the affine adaptation in the batchnorm layer, but only the scale parameter.
Model training. The model is trained to minimize the binary cross-entropy the clean data. We use batch gradient descent on the whole 300,000 sample dataset with a learning rate 0. and no momentum. We decay the learning rate by factor of 0.1 after 1000 and 2000 steps and stop training after 3000 steps.
Model Learning and Adaptation. We adapt the model on the corrupted data using Tent. More precisely, we optimize the scale parameter of the batch norm layer minimize the entropy of the predictions using SGD with a learning rate of 0.01 and no momentum. We process the whole dataset one batch and adapt for 60 000 steps.
Results. In the toy model, simple cases emerge where the loss does not result in collapse, or vice versa (Fig. 7a and Fig. 7b, respectively), mainly depending on the relation of signal and noise variances and directions.
The toy example furthermore predicts that the adapted parameters model should grow on the long run and indeed we were able to find exactly this effect when running ETA on a ResNet50 on CCC-Medium (Fig. 8), suggesting that our minimal setup successfully reproduces the relevant aspects of the large scale case. However, the weight explosion becomes apparent only after the collapse happens, hence weight regularization is not enough to avoid the collapse (Fig. 8(b)).
In the Fig. 7(a) example, find the direction of high target domain performance and stay there; in this case entropy minimization is stable. In the Fig. 7(b) experiment, the domain shift adds more noise nearly orthogonal to the signal direction, which entropy minimization tries to use for making high confidence predictions: we still initially find the direction of high target domain performance, but traverse this region and continue into direction of low entropy and low accuracy. This shows that even in a linear example, entropy minimization can show initial performance improvement and then collapse.
B Path Finding Algorithm
Algorithm 1 describes the pseudo code of the algorithm used to generate CCC. The algorithm is based on set of Calibration Matrices, similar to the one shown in Figure 2. There exists matrix for every (Ma 125) pair, such that m|i][]] is equal to accuracy of pretrained ResNet-50 on the
16

---

comination of noises (R)) and severities (i/5.j/5). We will release the full set of matrices upon publication.
Additionally, Algorithm 2 uses a function MinValidPat((s; #2): this function returns the minimum path that starts (81, $2) and ends at (0,8z) for some 8g. The cost of path is simply the average of all entries along the path. The minimum path is defined as the path with a cost closest to 6, in absolute terms. Lastly, path is only valid if it starts with s2 equal to every transition either decreases 1 by 0.25, or increases 82 by 0.25, and stops once 81 is equal to 0.
Algorithm Algorithm used to generate each split of CCC
Require: ba,k, T
b Baseline accuracy, transition speed, and total split size. > Initialize the total images generated counter Initialize the first two corruptions. 3: path CaleulatePath(cu, c2, ba) & Calculate path along noise pair with an average accuracy closest ba.
I: ==
2: Cı Uniform(11 15))
4: loop
5:
82 + path ip]
6:
Subset Uniform(Inngeeeele
apply (1,, C2, $2) to Subset
8:
save Subset
9:
f 1-
10:
ift if T then return
end if
i = len(path) - then
13:
<-
14:
Uniform (11.orrm((11.111111 14})
15:
path CaleulatePath((e c2, ba) 0+0
Calculate path along the noise pair.
17:
else
18:
p p+1
b Move to the next severity combination
19:
end if
20:
end loop
Algorithm 2 CalculatePath
Require: C1, ba
The noises, and baseline accuracy
I: m +- CalibrationMattix(cs]]
2: MinPath, MinCost + Min Patha,, O, m, ba)
3: for 0.25,, 0.5, 0.75..... do
4:
Min ValidPaths(1,,, m, ba)
MinPath, MinCost <- MinValidPatl(S:: O, m,bb)
6: end for
7: return PinP
17

---

The resulting dataset features transitions between two different noises like the ones seen in Figure (a). We additionally plot the accuracy of a pretrained ResNet-50, alongisde the severities of the different noises in Figure! 9 (b).
Dec Bu
Bom
Dom Bac
So
Sew
Saveriiie
accurayy 198
8 E
storrroe
mo200 DAr
T00000, DLT
shop 200G
SC nO.S
Controst
5000
Deforgn
100
200
300
400
500
Number of Noise Comoination Seen
(b)
Figure 9: (a) Visualization of CCC-Medium's smooth transition between Zoom Blur Snow. Note: CCC additionally uses random flips and crops, which are not shown here. (b) As CCC transitions between noises, the severities of the first noise (red), and the second noise (blue) go up and down correspondingly, in order to keep the accuracy of a pretrained model stable.
We additionally share the following metadata about the length of traversals CCC- Easy/Mediun/Hard:
Table 6: CCC traversal length statistics, for each CCC split.
Min
Max
Mean
Median
CCC-Easy
11
36
22.8
23
CCC-Medium
21
41
33,9
34
CCC-Hard
41
41
41
41
18

---

C CCC Plots
Accusccy
Accursey
Accuracy [8]
2ATA
50
35
Fnnrained
20-
SUR
Oneeved Sempless
X007
Samples
(a)
(b)
(c)
Figure 10: Adaptation performance of all evaluated models using a ResNet-50 backbone. (a) CCC
Easy. (b) CCC Medium. (c) CCC Hard. For all subplots, model performances are averaged over the runs the respective difficulty level.
D EATA Implementation and Ablations
Our implementation of EATA differs from the official implementation' The reason for this is that the official implementation uses clean ImageNet validation images to calculate the Fisher vector matrix for its regularizer'. This stands in contradiction with the method, which should not have access to the training distribution at test time, as shown the paper in Table
Instead of using 2, 000 ImageNet validation images, we calculate the Fisher matrix using the first 000 images our data stream. We conduct hyperparameter search on the weight regularizer tradeoff parameter 3:
8
25
50
100
250
500
1000
1500
2000
Acc. [%]
46.5
46.9
47.1
46.7
46.1
45.6
44.8
44.0
Table 7: Accuracy of EATA on CIN-C holdout noises for different values of the weight regularizer loss.
Using the optimal value, 100 led to worse results than the default value, 2000, on CCC:
CIN-C
CCC-Easy
CCC-Medium CCC-Hard
CCC Avg
EATA-100
46.7
47.7
36.5
3.8
29.3
EATA-2000
41.8
48.2
35.4
8.7
30.8
Ours
46.5
49.3
38.9
9.6
32.6
Table 8: Accuracy of EATA on CIN-C holdout noises for different values of the Fisher alpha.
In the end, we used the original value of 2000, as that was optimal on the CCC dataset.
In addition, we conducted hyperparameter search for EATA on a ViT backbone. As shown in 4, EATA performs worse than pretrained, non adapting baseline this setting. To that end, we tried to stabilize the model by increasing the value of B3, the hyperparameter that controls the weight of the anti-forgetting regularizer. As with the previous experiment, the original value of 2000 is optimal.
version version f739h3668c
19

---

B
2000
3000
4000
Ace.
[%
38.5
27.8
16.5
Table 9: Accuracy of EATA CCC-Medium using VIT backbone, for different values of the regularizer, B.
E
Novelty of Resetting
Our work is the first to propose resetting to solve collapse in TTA methods. Notably, while prior work [30, 47, 54] has briefly touched upon the concept of episodic resetting, the methodology and its application significantly distinct and unrelated to collapse TTA.
Tent [47] mentions episodic in the context of overfitting to single sample in segmentation (similar to [54]'s overfitting to single sample). Resetting here is unrelated collapse, the paper doesn't t discuss collapse at all.
Although MEMO [54] uses resetting, it does because it overfits one image (and its augmenta- tions) every step. MEMO doesn't I discuss collapse or catastrophic forgetting. MEMO compares itself version of Tent that resets after every step (which they call Tent + episodic resetting). because MEMO without augmentations is similar to Tent episodic resetting with batch size of (Note: MEMO is outperformed by BN/Ten/ETA when using the standard batch size of 64)
EATA [30] shows results for Tent + episodic resetting after every step in its tables, but provides no reasoning or discussion for doing this. Tent episodic resetting outperformed by regular Tent.
F CIFAR10 Experiments
We conduct CIFAR10 experiments to show the need for ImageNet scale benchmarks. Tent, without an anti-collapse mechanism, does not collapse on CIFARIO-C, even after seeing 100 million images Fig. la.b. Like ImageNet, CIFARIO-C's noises also exhibit high variance in difficulty Fig. 11c.
(a) Accuracy Sevenity
(b)
Tent Accuracy on C10 C Gradual
(c) Pretraised Accuracy on CLD Severity Noses
keuna PM
19.0
78.5
70.0
250
30
Imagee Suen Milico
Figure 11: (a) When tested on infinite concatenation of severity 5 noises, Tent does not collapse even after seeing 100M CIFAR scale images. (b) Tent does not collapse to chance level when tested long term variant of CIFAR10- 10-C gradual. (c) CIFARIO-C exhibits great variations between individual corruptions, similar to ImageNet-C.
G Compute details
conduct experiments on Nvidia RTX 2080 TI GPUs with 12GB memory per device. All experiments except our study on larger models were conducted single GPU. For CoTTA experiments, we use data parallel training on GPUs. bulk the compute spent for this work was on computing baseline accuracies on the calibration dataset, which contains 463M images.
20

---

H
Software and Dataset Licenses
H.1 Datasets
ImageNet-C [12]: Creative Commons Attribution 4.0 International,
https://zenodo. htt///ecodd/222505
ImageNet-CC [12], code for generating corruptions: Apache License 2.0
ImageNet-3D-CC [16]: CC-BY-NC 4.0 License
H.2
Models
PyTorch's [31] Backbones
ittps://pytorch. http:///iisioo//tttaa///aaaaaaaa
Adaptive BN [29,38] 38]:
Apache License 2.0, https: hrtppp/r/ggttthh../////hgghhh//s
Tent [47]: MIT License,
https://gitthb..co
RPL [35]: Apache License 2.0,
COTTA [48]: MIT License,
CPL [8]: MIT License,
EATA [30]: MIT License
https://github. comp//nn---ggglaa///E/A
21


Reviews:
[{"review_id": "xf3mZVR9m0", "reviewer": "Reviewer_h5ar", "summary": "The paper proposes a new benchmark (dubbed CCC) for test-time adaptation, which generalizes previous corrupted imagenet benchmarks. Specifically, the CCC gradually draws a sequence of corruptions (e.g. gaussian noise or motion blur), and gradually interpolates between two consecutive corruptions, creating a stream without hard boundaries, akin to many realistic settings. By considering long sequences of corruption pairs, CCC yields a stream longer than previous benchmarks, which sheds new light on how previous methods fair in such settings. Indeed, it is shown that given a long enough stream, methods collapse to a performance worse than a non-adapted model. The authors then propose a new baseline, Rdumb, which resets to the default parameters of the pretrained model at regular, fixed intervals. Over both CCC and previous benchmarks, it is shown that Rdumb, despite its simplicity, works well. ", "questions": "1. My understanding of TTA's use-cases is mostly for online settings; the deployed model receives data for which to output predictions, and this data may encounter distribution shifts. In such a setting, how would one determine the frequency of when to reset the model ? \n\n", "limitations": "N/A", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The proposed benchmark is well-designed. I liked the use of calibration with a pretrained model to generate different stream difficulties, and the fact that this overall is stream generator can is flexible and can accommodate smooth transitions. \n2. The proposed baseline, Rdumb, is extremely simple and should be considered as a standard baseline for TTA. \n3. Overall the paper is well written and easy to follow. ", "weaknesses": "1. The idea of resetting the model to its original pretrained state in order to combat accumulating degradation in TTA is not new (as correctly stated in the paper). Indeed, it was shown in [45] that resetting after each task boundary helps the network. For a fair comparison with COTTA, how would such method perform if the probability of resetting the weights was determined by a similar cross-validation technique as used in Section 6 ? \n2. COTTA also has some experiments with gradually changing corruptions (see Table 3). it would be good for this to be mentioned in the paper. \n\nminor comments : \n1. regarding figure 2, I would use the color red as the most severe corruptions, and yellow as the least severe. \n2. Optimal resetting interval : I think it should be Rdumb, and not Gdumb (lines 189-190)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review. We’re happy you found our dataset interesting, and our paper easy to read. We want to address the weaknesses and questions below.\n\n> For a fair comparison with COTTA, how would such method perform if the probability of resetting the weights was determined by a similar cross-validation technique as used in Section 6?\n\nThis is a good suggestion. To recap, CoTTA has a standard parameter of p=0.001 (on ImageNet), which we used in the experiments so far. We now applied our full evaluation protocol to CoTTA, and checked different thresholds on the holdout set:\n\n| resetting param (p) | 0.001 (default) | 0.005 | 0.01 | 0.05 | 0.1 | RDumb |\n|--------------------|----------------|-------|------|------|-----|-------|\n| CIN-C Holdout Avg. Acc    | 35.73          | 36.77 | 38.04| 37.86| 37.41| 46.7  |\n\n\nWe find an optimal parameter of p=0.01 which we apply to our test set:\n\n|              \t| CCC-Easy* | CCC-Medium | CCC-Hard |\n|------------------|-------------------|------------|----------|\n| Pre-Trained  \t| 34.04         \t| 17.3   \t| 1.5  \t|\n| Cotta (p=0.001)  | 14.9          \t| 7.7    \t| 1.1  \t|\n| Cotta (p=0.01)   | 27.8          \t| 15.6   \t| 1.1  \t|\n| RDumb        \t| 49.15         \t| 38.9   \t| 9.6  \t|\n\n*\\*Note: All runs except for one are finished on CCC-Easy, therefore we take out the unfinished run from the metrics of all models to have a fair comparison. We will of course update this table with the finished run as soon as possible.*\n\nWhile the results seem to improve slightly, note that CoTTA is still worse than a pretrained net on all benchmark datasets. We suggest to include this analysis in the supplementary material, and use the published hyperparameters (which is also present in the CoTTA codebase), as the message remains consistent.\n\nDoes this clarify your concern? Thanks again for this good suggestion.\n\n> COTTA also has some experiments with gradually changing corruptions (see Table 3). it would be good for this to be mentioned in the paper.\n\nYes you are right, thanks for pointing this out, we now call this out in the discussion of previous work.\n\nFor completeness, however, we would like to point out three important differences and contributions CCC makes over this setting:\n\nFirst, the setting used in the COTTA paper is way easier than CCC and fails to show a collapse to chance level. We ran an experiment on the concatenated CIFAR10-C dataset for more than 100 million images (vs. 150k images in the CoTTA paper) with no sign of collapse (Figure 4, Rebuttal PDF). We also run a similar experiments for 25 million images on the gradual CIFAR10 task (Figure 5, Rebuttal PDF). While the fluctuations in performance increase, the setup still fails to produce the collapsing behavior to chance level we observe at ImageNet-scale with CCC. This highlights the difference in difficulty to our “real world sized” CCC dataset.\n\nSecond, “gradual” here means a transition from clean to corrupted images. CCC corruptions have a much larger combinatorial space due to the simultaneous application of multiple corruptions.\n\nThird, the C10-C tasks are less-well controlled, which is similar to the issues we discuss for concatenated ImageNet-C. You can see this in Figure 6 in the rebuttal PDF, where performance fluctuates greatly between noises. This makes the dataset not suitable for controlled evaluation of models in a scientific setting.\n\nThe point you raise will help to position CCC even better within the existing literature, thanks for the suggestion. \n\n> My understanding of TTA's use-cases is mostly for online settings; the deployed model receives data for which to output predictions, and this data may encounter distribution shifts. In such a setting, how would one determine the frequency of when to reset the model ?\n\nResetting is not conceptually different from any other hyperparameter in TTA models, your concern would hence also apply to the hyperparameters in CoTTA or EATA. \nHowever, in contrast to previous work, the resetting parameter has the advantage of being easily interpretable. As it tends to 0, the model gets more conservative, and approaches the performance of batch norm adaptation. As it tends to infinity (no resetting), the model approaches the performance of a (collapsing) TTA algorithm.\nWe show (cf. Table 2) that the model performance on CCC is fairly robust to the exact setting. In real-world scenarios, the parameter could be easily set by cross-validation (like any other hyperparameter in TTA) on a suitable development set (which are now very easy to generate for different difficulty levels and noise settings thanks to CCC).\n\nWe hope this addresses your concern, and we are happy to answer any questions that might still remain."}, {"review_id": "x4hRzYTCtx", "reviewer": "Reviewer_BezD", "summary": "The paper newly introduces a new benchmark dataset for Continual Test-Time Adaptation (CTTA) named Continually Changing\nCorruptions (CCC), and suggests a simple technique - repeatedly initialize the learned model weights during CTTA. CCC is composed of interpolated corruption data and its data scale varies at each timestep. They can control the difficulty of CCC data by changing the degree of interpolation of two different corruptions.", "questions": "Why does RDumb consistently outperform a pre-trained model during TTA? Shouldn't its performance reach the pre-trained model's performance when re-setting the model weights to the initially pre-trained ones?", "limitations": "Please see the weakness.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The paper suggests a new large-scale continual test-time adaptation benchmark dataset, considering corruption interpolation, scale variation, and repetition.", "weaknesses": "The suggested method is too naive and not attractive in view of knowledge transfer. Periodically resetting the trainable weights to the initial state indicates simply discarding obtained knowledge and adaptive representation regardless of their importance and relevancy. Even though this remedy outperforms baselines, there is a lack of quantitative and qualitative analysis of why it happens and of the reason RDumb behaves differently from baselines. Additionally, the decision of resetting iteration is heuristic.\n\nThe suggested idea is surprising but I strongly recommend more rigorous analyses and validation of the suggested method and CCC dataset since they are counterintuitive in some sense.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your review! We agree with your assessment that our results are quite surprising. We hope to clarify your concerns and address the remaining weaknesses below:\n\n> “Even though this remedy outperforms baselines”\n\nThis sounds like a possible misunderstanding. While RDumb indeed outperforms baselines (pre-training, batchnorm), it also outperforms (or matches) *all published state-of-the art test time adaptation methods*, which includes CoTTA and EATA, while being conceptually simpler and easier to analyze. Due to its technical simplicity, our paper argues that RDumb should serve as a baseline in follow up work, and questions the way we assess TTA methods.\n\n> The suggested method is too naive\n\nTechnical complexity by itself is not a criterion for acceptance at NeurIPS: the principle of Occam's razor remains pertinent. Our results suggest that an extremely simple method performs just as well or *better* than more complex ones.\n\nBut furthermore, besides the RDumb method, there is a lot of technical depth in the construction of CCC: Unlike previous datasets which simply stack a few existing data points, we propose a well-designed setup which is orders of magnitude bigger than current evaluation setups, in number of noises (210 vs 15), number of severities (441 vs 5).\n\nIt enables the benchmarking of continual methods for at least 10 times as long as current benchmarks, without repeating images (and longer, if required in the future). The complexity of the benchmark allows for evaluating methods on a controlled baseline accuracy, which wasn’t possible in previous work.\n\nWith CCC, we pose a now challenging evaluation setting which highlights the shortcomings of current TTA methods. This evaluating setting is arguably the first true “continual test-time adaptation setting”, as previous benchmarks miss important hallmarks of continual adaptation (as we argue).\n\nThis is a key contribution of this work, and its depth is easy to miss, as a lot of the work of constructing the benchmark is hidden in the supplementary material.\n\nWe’re happy to discuss this point further in case we missed what you meant.\n\n> … not attractive in view of knowledge transfer. Periodically resetting the trainable weights to the initial state indicates simply discarding obtained knowledge and adaptive representation regardless of their importance and relevance.\n\nWe respectfully disagree.\n\nThe fact that such a method outperforms others suggests that *none* of the previously used methods are effective at knowledge transfer, and that is a very valuable insight (as you point out) that was missed in previous work. The title of our paper exactly reflects this line of thought.\n\nWhile other methods are allowed to benefit from many images seen, they are still not as effective as a method that resets itself every 1,000 steps.\n\nThis is a very surprising result given claims in previous papers [1,2]. Don’t you agree? Happy to discuss further.\n\n> lack of quantitative and qualitative analysis of why it happens\n\nThis is a fair point. We plan to add the following analysis that will further strengthen the paper:\n\nWe constructed a simple 2d Gaussian binary classification example, a domain shift which slightly rotates the data and adds Gaussian noise, and a model which consists of a batch norm layer followed by a logistic regression. When running entropy minimization on the adaptation layer of the batch norm, simple cases emerge where the loss does or does not result in collapse, mainly depending on the relation of signal and noise variances and directions (Fig. 1-2, Rebuttal PDF).\n\nThe toy example furthermore predicts that the adapted parameters of a model should grow on the long run and indeed we were able to find exactly this effect when running ETA on a ResNet50 on CCC-Medium (Fig. 3, rebuttal PDF), suggesting that our minimal setup successfully reproduces the relevant aspects of the large scale case. However, the weight explosion becomes apparent only after the collapse happens, hence weight regularization is not enough to avoid the collapse (see Fig 3b).\n\nThank you for raising this issue, we think that this analysis adds to the paper and we will include it in the camera ready version. We would be happy to hear your thoughts on this experiment.\n\n> the decision of resetting iteration is heuristic.\n\nThis is not the case, please see Section 6. The reset interval is obtained by cross-validation on the holdout noises, which to our knowledge is the known best practice in the field. However, we are happy to re-evaluate with another selection strategy, if you have suggestions (but doubt it would change the core message of our paper).\n\nAs an aside, we would argue that a resetting interval is a more interpretable hyperparameter than a resetting fraction (in CoTTA) or regularizer weights (in EATA). \n\nWe’d be happy to discuss this point further with you.\n\n> Q: Why does RDumb consistently outperform a pre-trained model during TTA? Shouldn't its performance reach the pre-trained model's performance when re-setting the model weights to the initially pre-trained ones?\n\nYou’re right that the pretrained model and RDumb have similar accuracies everytime RDumb is reset. In Figure 2, each point is an average of 781 batches (approx. 50k images, i.e. the size of the ImageNet validation set). We do this because otherwise there would be too many points on the graph. RDumb is able to learn quickly and therefore surpasses the pretrained model within those 781 batches, as can also be seen in Figure 5a. We will make this clearer in the next draft, thank you for bringing this to our attention.\n\nWe hope this addresses your concerns, and would be happy to discuss further.\n\n[1] Wang, Qin, et al. \"Continual test-time domain adaptation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2022.\n\n[2] Niu, Shuaicheng, et al. \"Efficient test-time model adaptation without forgetting.\" International conference on machine learning. PMLR, 2022.\n"}, {"review_id": "UJfY1G37Bn", "reviewer": "Reviewer_xRL5", "summary": "This paper proposes a new benchmark for continual test-time adaptation (CTTA): CCC (Continually Changing Corruptions).\nExperiments for existing state-of-the-art approaches using their proposed benchmark demonstrate that even a non-adapting model performs better than existing approaches for this benchmark.\nFurther, they propose a simple approach as a baseline, named $\\textit{RDumb}$, which periodically resets the model to its pre-trained state, which performs better than the SOTA approaches.", "questions": "1. In CoTTA (Wang et al., CVPR 2022), there is a setting of gradually changing the corruptions (CIFAR10-to-CIFAR10C gradual). CoTTA, as well as PETAL [1], reports performance improvement in this gradual setting, so it would be interesting to see how they would work on the CCC variant of CIFAR10-to-CIFAR10C. Do the authors have any intuition behind this?\n2. As the authors point out, the ImagenetC dataset is large and other benchmarks are much smaller; the dataset being smaller can also make the task more challenging, since there would be lesser data to learn from for a domain shift. It would be interesting to see how the performance is for CCC variants of CIFAR10-to-CIFAR10C, CIFAR100-to-CIFAR100C.\n3. Also, with regards to the semantic segmentation task, do the authors have some idea as to how this dataset/benchmark building mechanism would be applicable?\n\n\n*References*\n1. Brahma, Dhanajit, and Piyush Rai. \"A Probabilistic Framework for Lifelong Test-Time Adaptation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.", "limitations": "1. Since the state-of-the-art approaches are being evaluated on a new benchmark dataset, it is very important to tune their hyperparameters optimally.\n2. The details regarding hyperparameter tuning for various SOTA approaches are missing.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. This paper proposes a more challenging benchmark for CTTA. \n2. Combining a pair of corruptions is a novel idea.\n3. Experimental results demonstrate that several state-of-the-art approaches perform poorly on this benchmark.\n4. A simple periodic reset mechanism with weighted entropy (as in ETA) adaptation loss outperforms the SOTA CTTA approaches.", "weaknesses": "1. In CoTTA (Wang et al., CVPR 2022), there is a setting of gradually changing the corruptions (CIFAR10-to-CIFAR10C gradual). So the idea of gradually changing corruption is not entirely new, even though CCC has a combination of two corruptions.\n2. This paper does not analyze the reasons for the performance degradation of the SOTA TTA approaches, such as overfitting, error accumulation, catastrophic forgetting, or interference, and does not provide any insights or suggestions on mitigating or preventing it.\n3. Experimental results on CIFAR10-to-CIFAR10C, CIFAR100-to-CIFAR100C classification tasks, along with their CCC variants, are missing.\n\nMinor Comment: Line 62: use `` for starting quotes before RDumb", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review. We’re glad you found the dataset novel and challenging, and our experimental results meaningful! We addressed all mentioned weaknesses below and replied to your questions.\n\n> In CoTTA, there is a setting of gradually changing the corruptions [...]. So the idea of gradually changing corruption is not entirely new, even though CCC has a combination of two corruptions.\n\nWe would like to point out three important differences and contributions CCC makes over this setting:\n\nFirst, the setting used in the COTTA paper is way easier than CCC and fails to show a collapse to chance level. We ran an experiment on the concatenated CIFAR10-C dataset for more than 100 million images with no sign of collapse (Fig. 4, Rebuttal PDF). We also run a similar experiments for 25 million images on the gradual CIFAR10 task (Fig. 5). While the fluctuations in performance increase, the setup still fails to produce the collapsing behavior to chance level we observe at ImageNet-scale with CCC. This highlights the difference in difficulty to our “real world sized” CCC dataset.\n\nSecond, “gradual” here means a transition from clean to corrupted images. CCC corruptions have a much larger combinatorial space due to the simultaneous application of multiple corruptions.\n\nThird, the C10-C tasks are less-well controlled, which is similar to the issues we discuss for concatenated ImageNet-C. You can see this in Figure 6 in the rebuttal PDF, where performance fluctuates greatly between noises. This makes the dataset not suitable for controlled evaluation of models in a scientific setting.\n\nWe are happy to discuss this further and agree with you that we should point out these features of CCC more prominently.\n\n> This paper does not analyze the reasons for the performance degradation of the SOTA TTA approaches, such as overfitting, error accumulation, catastrophic forgetting, or interference, and does not provide any insights or suggestions on mitigating or preventing it.\n\nThis is a fair point, and we added an analysis now.\n\nWe constructed a simple 2d Gaussian binary classification example, a domain shift which slightly rotates the data and adds Gaussian noise, and a model which consists of a batch norm layer followed by a logistic regression. When running entropy minimization on the adaptation layer of the batch norm, simple cases emerge where the loss does or does not result in collapse, mainly depending on the relation of signal and noise variances and directions (Fig. 1-2).\n\nThe toy example furthermore predicts that the adapted parameters of a model should grow on the long run and indeed we were able to find exactly this effect when running ETA on a ResNet50 on CCC-Medium (Fig. 3, rebuttal PDF), suggesting that our minimal setup successfully reproduces the relevant aspects of the large scale case. However, the weight explosion becomes apparent only after the collapse happens, hence weight regularization is not enough to avoid the collapse (Fig 3b).\n\nThank you for raising this issue, we think that this analysis adds to the paper and will include it in the camera ready version. We would be happy to hear your thoughts on this.\n\n> Experimental results on CIFAR10-to-CIFAR10C, CIFAR100-to-CIFAR100C classification tasks, along with their CCC variants, are missing.\n\nContinual test time adaptations make claims about the performance of real-world scale vision models during deployment time. CIFAR bears none of the desiderata for a dataset to be predictive of model performance in this setup, as we discussed above.\n\nHence, we refrained from running CIFAR scale experiments, the minimum scale for meaningful evaluation should be ImageNet-C, variants of it, or CCC. These experiments are sufficiently quick to run on typical research hardware.\n\nHowever, if we missed a good argument on how CIFAR experiments are useful in providing a signal for building continual adaptation models, we are happy to discuss further. In particular, if CIFAR scale experiments would meaningfully change your assessment of our work, we could discuss a suitable setup for hyperparameter search on all models for evaluation (as CoTTA, EATA, etc. lack details on the CIFAR evaluation protocol and how hyperparameter search was performed).\n\n> Also, with regards to the semantic segmentation task, do the authors have some idea as to how this dataset/benchmark building mechanism would be applicable?\n\nWhile possible with our construction mechanism, building a “Segmentation-CCC” is well beyond of scope for the current work. However, we think that any model should pass the CCC test first before scaled up to even more involved computer vision problems.\n\nOtherwise, we agree that this could be an interesting future work, the limiting factor is the huge compute budget needed to calculate the calibration set.\n\n> Since the state-of-the-art approaches are being evaluated on a new benchmark dataset, it is very important to tune their hyperparameters optimally.\n\nWe agree, and we performed hyperparameter searches on both EATA (Appendix C) and CoTTA (see reply to reviewer h5ar).For CoTTA, we ran a hyperparameter search on CoTTA using the same protocol we used for RDumb (Section 6), and found that CoTTA still collapses on every level of CCC.\nPlease note that the only additional hyperparam we consider is the resetting interval. All other parameters in RDumb are taken from previous work.\nIf you want us to run any other hyperparam search, we would be more than happy to do so, but we doubt it will change our key results (as we saw in CoTTA and EATA).\n\n> The details regarding hyperparameter tuning for various SOTA approaches are missing.\n\nPlease see our reply above, and also Section 6 and App. C in the paper. We are happy to add additional information.\nWe fully agree with your sentiment of the importance of hyperparameter tuning in TTA, but feel that the practice in our paper matches or exceeds the practiced standards in the field.\nWe are happy to discuss further, though.\n"}, {"review_id": "u8uLzoLtst", "reviewer": "Reviewer_Gw5H", "summary": "Test-time adaptation techniques seek to adapt a model to new unlabelled samples without access to the original training data. However, TTA approaches are typically not evaluated for long runtimes. This work proposes a new benchmark Continuously Changing Corruptions (CCC), which is a stream of corrupted ImageNet images with gradual changes and varying degrees of difficulty. Previous approaches fail on (CCC) when compared to the pretrained model as well as a simple new baseline method dubbed RDumb. RDumb is shown to achieve high performance across several experimental settings. ", "questions": "Questions:\n1) The mathematical notation in Eq. 1 is a bit difficult to parse. In particular, it is unclear what is meant by $cos(y_t, \\bar{y}_t)$. Is this cosine similarity or cosine distance? I would encourage the authors to clarify this and also provide a clear description of the equation in the text.\n\n2) Can the authors precisely define \"collapse\"? This term is used but is never defined.\n\n3) In Figures 3 and 4, if RDumb is being reset to the pretrained model weights, then why do the blue and purple lines never intersect? \n\n4) It would be interesting to test CLIP image encoders [1] that are naturally robust and have high zero-shot accuracy on OOD samples. In future work, I would be curious to see how such models behave on the proposed CCC when TTA techniques are applied. \n\nReferences:\n- Learning Transferable Visual Models From Natural Language Supervision, Radford et al. 2021", "limitations": "Limitations are addressed.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "Strengths:\n- The overall message of the paper is easy to follow and the experiments are illustrative of a serious shortcoming in the evaluation of TTA methods.\n- The baseline is simple and satisfies all the TTA assumptions, and should be considered as a baseline in future works in this area\n- Experiments are very extensive and thorough", "weaknesses": "Weaknesses:\n- There are a few issues related to clarity that should be addressed in the next version of the paper. See Questions\n- I would encourage the authors to add error bars to their tables, so that it is clear that the differences are statistically significant\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your review.  We’re happy you found our paper to be easy to follow, and the experiments to be extensive. We want to address your questions:\n> I would encourage the authors to add error bars to their tables, so that it is clear that the differences are statistically significant\n\nGood point. We updated the paper with a new version of the table:\n\n| Adaptation method    \t| CIN-C      \t| CIN-3DCC   \t| CCC-Easy   \t| CCC-Medium \t| CCC-Hard   \t| Average     \t|\n|--------------------------|----------------|----------------|----------------|----------------|----------------|-----------------|\n| Pretrained (He et al.) | 18.0 $\\pm$ 0.0 | 31.5 $\\pm$ 0.0 | 34.1 $\\pm$ 0.22 | 17.3 $\\pm$ 0.21 | 1.5 $\\pm$ 0.02 | 20.5        \t|\n| BN (Schneider et al.) | 31.5 $\\pm$ 0.02 | 35.7 $\\pm$ 0.02 | 42.6 $\\pm$ 0.39 | 27.9 $\\pm$ 0.74 | 6.8 $\\pm$ 0.31 | 28.9        \t|\n| Tent (Wang et al.) | 15.6 $\\pm$ 3.5 | 24.4 $\\pm$ 3.5 | 3.9 $\\pm$ 0.58 | 1.4 $\\pm$ 0.17 | 0.51 $\\pm$ 0.07 | 9.2   |\n| RPL (Rusak et al.) | 21.8 $\\pm$ 3.6 | 30.0 $\\pm$ 3.6 | 7.5 $\\pm$ 0.83 | 2.7 $\\pm$ 0.36 | 0.67 $\\pm$ 0.14 | 12.5 |\n| SLR (Mummadi et al.) | 12.4 $\\pm$ 7.7 | 12.2 $\\pm$ 7.7 | 22.2 $\\pm$ 18.4 | 7.7 $\\pm$ 9.0 | 0.66 $\\pm$ 0.57 | 11.0 |\n| CPL (Goyal et al.) | 3.0 $\\pm$ 3.3 | 5.7 $\\pm$ 3.3 | 0.41 $\\pm$ 0.06 | 0.22 $\\pm$ 0.03 | 0.14 $\\pm$ 0.01 | 1.9 |\n| CoTTA (Wang et al.) | 34.0 $\\pm$ 0.68 | 37.6 $\\pm$ 0.68 | 14.9 $\\pm$ 0.88 | 7.7 $\\pm$ 0.43 | 1.1 $\\pm$ 0.16 | 19.1 |\n| EATA (Niu et al.) | 41.8 $\\pm$ 0.98 | 43.6 $\\pm$ 0.98 | 48.2 $\\pm$ 0.6 | 35.4 $\\pm$ 1.0 | 8.7 $\\pm$ 0.8 | 35.5 |\n| ETA (Niu et al.) | 43.8 $\\pm$ 0.33 | 42.7 $\\pm$ 0.33 | 41.4 $\\pm$ 0.95 | 1.1 $\\pm$ 0.43 | 0.23 $\\pm$ 0.05 | 25.8 |\n| RDumb (ours) | **46.5 $\\pm$ 0.15** | **45.2 $\\pm$ 0.15** | **49.3 $\\pm$ 0.88** | **38.9 $\\pm$ 1.4** | **9.6 $\\pm$ 1.6** | **37.9** |\n\n\n\n> The mathematical notation in Eq. 1 is a bit difficult to parse. In particular, it is unclear what is meant by $\\cos(y_t, \\hat{y}_t)$. Is this cosine similarity or cosine distance? I would encourage the authors to clarify this and also provide a clear description of the equation in the text.\n\nWe apologize for not making this clear - what was meant is cosine similarity. We made this clearer in the text.\n\n>Can the authors precisely define \"collapse\"? This term is used but is never defined.\n\nWe consider a model as collapsed when a model classifies images worse than a pretrained, non-adapting model. Thanks for raising this clarity issue, we will make the term more clear in the next version. \n\n\n>In Figures 3 and 4, if RDumb is being reset to the pretrained model weights, then why do the blue and purple lines never intersect?\n\nEach point on the graph represents an average of 781 iterations (around 50k images seen). This is done so as to not have to plot millions of points. Although RDumb resets, it very quickly becomes better than a pretrained model (see also Figure 5a), which is why the lines don’t intersect. \n\n>It would be interesting to test CLIP image encoders [1] that are naturally robust and have high zero-shot accuracy on OOD samples. In future work, I would be curious to see how such models behave on the proposed CCC when TTA techniques are applied.\n\nWe think so too, which is why we think CCC is an important benchmark for future work.\n\nWe hope that our response clears up any questions you had, and if it doesn't, we’d be glad to add further clarifications. \n\n"}]

Author rebuttal:
We thank the reviewers for their constructive feedback. We are glad to hear that reviewers found our paper easy to follow, our proposed dataset interesting, and our method to be simple and effective.

We commented on all reported weaknesses and addressed the reviewers' questions in the individual responses below.

Concerning new results to be added to the paper, we report new results for [additional hyperparameter tuning on CoTTA](https://openreview.net/forum?id=VfP6VTVsHc&noteId=zMGLcCwMDu), an [updated Table 1](https://openreview.net/forum?id=VfP6VTVsHc&noteId=rZV3x9TDSH) with error bars, and extensive new analysis of the collapsing behavior, for which we attach additional figures in the PDF here.

We hope that our replies clarify all reviewer concerns, and are happy to engage in further discussion and experiments.

Meta-review: This work on test-time adaptation contributes a new benchmark (CCC) for continual adaptation, which mixes corruptions in a continuously changing fashion, and a simple method (RDumb), which periodically resets the model and can serve as a baseline for more sophisticated methods. Four reviewers with expertise in adaptation, continual learning, and personalization vote for acceptance (h5ar : 7, Gw5H: 7, xRL5: 5) and rejection (BezD: 3). The authors provide a rebuttal with experiments, and all reviewers respond to the authors except for BezD, as noted by the AC and mentioned by an author comment. The main argument for acceptance is that the RDumb method and CCC benchmark clearly question how well continual adaptation methods continue to adapt, and the question is justified by the thorough experiments. The main argument for rejection are lack of analysis or insight for why continual adaptation degrades (xRL5, BezD), lack of novelty w.r.t. continual adaptation (h5ar, xRL5), and insufficient development of how to reset (BezD, h5ar). The rebuttal addresses each of these negatives. While further analysis—of the dataset or adaptation degradation—and more sophisticated resetting would improve this work, the AC sides with acceptance, because the empirical content is already informative (even surprising!) and rigorous. As such this work can guide the next steps for continual adaptation and gauge progress in this direction.

Note: reviewer BezD engaged in AC-reviewer discussion to confirm the vote for rejection. They emphasize 1. the limitation of resetting as a method, 2. lack of analysis that explains how resetting succeeds while existing methods fail, and 3. the need to further validate the CCC benchmark in its ordering, scaling, and repeating of corruptions. Re: 1., the limitation of resetting is real, in particular the inability to forward transfer knowledge (say for cyclic shifts like seasons or repeated shifts like weather), the experiments on resetting show it to be a valid baseline. Re: 2. and 3., more analysis is always better (and specifically expanding the rebuttal experiments to cover ETA/EATA would show more generality), but the AC and the majority of reviewers see the current experiments as sufficient. The AC advises the authors to reflect on point 1. and consider addressing this in the discussion or conclusion, and the AC thanks BezD for their reasoned and specific points for further improvement.

The AC encourages the authors to incorporate the rebuttal material into the supplement for completeness, in particular the analysis of a toy example and the CIFAR-10/100-C results, as certain readers will find these informative.
Decision: Accept (poster)