SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Ori Press<sup>1*</sup> Steffen Schneider<sup>1,2</sup> Matthias Kümmerer<sup>1†</sup> Matthias Bethge<sup>1†</sup>
<sup>1</sup>University of Tübingen, Tübingen AI Center, Germany  
<sup>2</sup>EPFL, Geneva, Switzerland
Test-Time Adaptation (TTA) allows to update pre-trained models to changing data distributions at deployment time. While early work tested these algorithms for individual fixed distribution shifts, recent work proposed and applied methods for continual adaptation over long timescales. To examine the reported progress in the field, we propose the Continually Changing Corruptions (CCC) benchmark to measure asymptotic performance of TTA techniques. We find that eventually all but one state-of-the-art methods collapse and perform worse than a non-adapting model, including models specifically proposed to be robust to performance collapse. In addition, we introduce a simple baseline, "RDumb", that periodically resets the model to its pretrained state. RDumb performs better or on par with the previously proposed state-of-the-art in all considered benchmarks. Our results show that previous TTA approaches are neither effective at regularizing adaptation to avoid collapse nor able to outperform a simplistic resetting strategy.
Biological vision is remarkably robust at adapting to continually changing environments. Imagine cycling through the forest on a cloudy day and observing the world around you: You will encounter a wide variety of animals and objects, and be able to recognize them without effort. Even as the weather changes, rain sets in, or you start cycling faster, the human visual system effortlessly adapts and robustly estimates the surroundings [45]. Equipping machine vision with similar capabilities is a long-standing and unsolved challenge, with numerous applications in autonomous driving, medical imaging, and quality control, to name a few.
Techniques for improving the robustness to domain shifts of ImageNet-scale [37] classification models include pre-training of large models on diverse and/or large-scale datasets [26, 33, 50] and robustification of smaller models by specifically designed data augmentation [11, 14, 36]. While these techniques are applied during training time, recent work [8, 28, 29, 30, 35, 38, 47, 48] explored possibilities of further adapting models by Test-Time Adaptation (TTA). Such methods continuously update a given pretrained model exclusively using their input data, without having access to its labels. Test-time entropy minimization (Tent; 47) has become a foundation for state-of-the-art TTA methods. Given an input stream of images, Tent updates a pretrained classification model by minimizing the entropy of its outputs, thereby continuously increasing the model's confidence in its predictions for every input image.
Previous TTA work [8, 28, 29, 35, 38, 47, 54] evaluate their models on ImageNet-C [13] or smaller scale image classification benchmarks [18, 20]. ImageNet-C consists of 75 copies of the ImageNet validation set, wherein each copy is corrupted according to 15 different noises at 5 different severity
*ori.press@bethgelab.org. †Joint senior authors. Code: https://github.com/oripress/CCC.
Figure 1: Continuously Changing Corruptions show limitations of existing TTA methods. (a) Comparison between ImageNet-Val, CIN-C and CCC. The proposed version of CCC is  $10 \times$  longer than CIN-C and could naturally be extended even further without repeating images. CCC consists of sequences of smooth transitions from one ImageNet-C noise to another one. For each such pair, we construct a trajectory continuously interpolating from one pure noise to the other pure noise such that baseline accuracy is kept constant. For each point along the trajectory, we sample a batch of 1k, 2k, or 5k images from ImageNet-Val, randomly crop and flip it and apply the noise combination. (b) Due to its short length and high variability in difficulty, CIN-C (top) is unable to reveal the collapse of methods such as ETA and CoTTA, while CCC (middle and bottom) can.
levels. When TTA models are evaluated on ImageNet-C, they are adapted on each noise and severity combination individually starting from their pretrained weights. Such a one-time adaptation approach is of little relevance when it comes to deploying TTA models in realistic scenarios. Instead, stable performance over a long run time after deployment is the desirable goal.
TTA methods are by design readily applicable to this setting and recently the field has started to move towards testing TTA models in continual adaptation settings [7, 30, 48]. Strikingly, this revealed that the dominant TTA approach Tent [47] decreases in accuracy over time, eventually being less accurate than a non-adapting, pretrained model [30, 48]. In this work, we refer to any model whose classification accuracy falls below that of a non-adapting, pretrained model, as having "collapsed".
This collapsing behaviour of Tent shows that it cannot be used in continual adaptation over long time scales without modifications. While previous benchmarking of TTA methods already managed to reveal the collapse of Tent, our work shows that in fact all current TTA methods collapse sooner or later, including methods with explicit built-in anti-collapse strategies.
Since current benchmarks have not been sufficient to detect collapse in several models, we introduce an image classification benchmark designed to thoroughly evaluate TTA models for their long-term behavior. Our benchmark, *Continuously Changing Corruptions* (CCC), tests models on their ability to adapt to image corruptions that are constantly changing, much like when fog turns to rain or day turns to night. CCC allows us to easily control different factors that could affect the ability of a given method to continuously adapt: the corruptions and their order, the difficulty of the images themselves, and the speed at which corruptions transition. Most importantly, the length of our benchmark is ten times longer than that of previous benchmarks, and more diverse by including all kinds of combinations of corruptions (see Figure 1a). Using CCC, we discover that seven recently published state-of-the-art TTA methods are less accurate than a non-adapting, pretrained model. While Tent was already shown to collapse [7, 30, 48], we show that this problem is not specific to
Tent, and that many other methods – including specifically designed continual adaptation methods – collapse as well.
Finally, we propose "RDumb" as a minimalist baseline mechanism that simply Resets the model to its pretrained weights at regular intervals. Previous work employs more sophisticated methods combining entropy minimization with various regularization approaches, yet we show that RDumb is superior on both existing benchmarks and ours (CCC). Our results call the progress made in continual TTA so far into question, and provide a richer set of benchmarks for realistic evaluation of future methods.
- We introduce the continual adaptation benchmark CCC. We show that previous benchmarks are too short to meaningfully assess long-term continual adaptation behaviour, and are too uncontrolled to assess the short-term learning dynamics.
- Using CCC, we show that the performances of all but one current TTA methods drop below a non-adapting, pre-trained baseline when trained over long timescales.
- We propose "RDumb" as a baseline and show that it outperforms all previous methods with a minimalist resetting strategy.
2 CCC: Towards Infinite Testing with Continuously Changing Corruptions
Until recently, it was common to evaluate TTA methods only on datasets on individual domain shifts such as the corruptions of ImageNet-C [13]. However, the world is steadily changing and recently the community started moving towards continual adaptation, i.e., evaluating methods with respect to their ability to adapt to ongoing domain shifts [30, 42, 48].
The dominant method of evaluating continual adaptation on ImageNet scale is to concatenate the top severity datasets of the 15 ImageNet-C corruptions into one big dataset. We refer to the variant of this dataset introduced by [48] as Concatenated ImageNet-C (CIN-C). CIN-C was used to demonstrate the collapse of Tent and the stability of recent TTA methods by [7, 30, 48].
In Figure 1b, we evaluate a range of TTA methods on CIN-C and notice three potential problems: Firstly, ETA [30] appears to be stable and better than a non-adapting, pretrained baseline, but is revealed to collapse when tested on CCC. Additionally, while CoTTA[48] clearly goes down in performance, it is not yet clear whether it collapses or stabilizes above or below baseline performance. Fundamentally, CIN-C turns out to be too short to yield reliable, conclusive results. Secondly, assessing adaptation dynamics is further obscured by the considerable variations of the baseline performance among the different corruptions in CIN-C. This is not only a factor that affects adaptation itself (shown in [30, 47]), it also leads to substantial fluctuations in performance across multiple runs, making it difficult to obtain a clear and reliable assessment. Finally, CIN-C features exclusively abrupt transitions between different corruption types. In contrast, in the real world, domain changes may often be smooth and subtle with varying speeds: day to night, rain to sunshine, or the accumulation of dust on a camera. Therefore, it is important to also probe TTA methods on continual domain changes that are not tied to a specific point in time and thus constitute a relevant test for stable continual adaptation.
Here we propose a new benchmark, *Continuously Changing Corruptions* (CCC), to address these issues. CCC solves the issues of benchmark length, uncontrolled baseline difficulty, and transition smoothness in a simple and effective manner. Firstly, the length issue is remedied because the individual runs of CCC are constructed by a generation process which can generate very long datasets without reusing images. In this work we use runs of 7.5M images, which is 10 times as long as CIN-C. If required to compare methods in future work (where collapse is even slower), it is straightforward to generate even longer benchmarks within the CCC framework. Secondly, since both [30, 47] have shown that dataset difficulty is a confounder when studying adaptation, the difficulty of individual benchmark runs is kept stable. Additionally, we examine three different difficulty levels to ensure a comprehensive yet controlled evaluation. Finally, CCC exhibits smooth domain shifts: it applies two corruptions to each image. Over time, the severity of one corruption is smoothly increased while the
The name was inspired by GDumb [32].
(b)
Figure 2: (a) Each corruption of CCC consists of applying two ImageNet-C corruptions at different severities. We extend the individual severities to be more fine-grained than in ImageNet-C, allowing for smoother noise changes, and exponentially more (noise, severity) combinations. The corners are enlarged for easier viewing, zoom in for greater detail. (b) Sample dataset sequences with a constant baseline accuracy. The sequences start from the left where Motion Blur is zeroed out, and end at the top with Gaussian noise zeroed out. The colors red, orange, and yellow correspond to trajectories in CCC-Easy, CCC-Medium and CCC-Hard, respectively.
severity of the other is decreased, maintaining the desired difficulty. We also study three different speeds for applying this process. We will now outline the generation procedure of the dataset.
Continuously changing image corruptions To allow smooth transitions between corruptions, we introduce a more fine-grained severity level system to the ImageNet-C dataset. We interpolate the parameters of the original corruptions (integer-valued severities from 1 to 5) to finer grained severity levels from 0 to 5 in steps of 0.25. We apply two different ImageNet-C corruptions to each image, such that we can decrease the severity of one corruption while increasing the severity of another one. Hence, the corruptions of CCC are given by quadruples  $(c_{1}, s_{1}, c_{2}, s_{2})$ , where  $c_{1}$  and  $c_{2}$  are ImageNet-C corruption types and  $s_{1}$  and  $s_{2}$  are severity levels. When applying such a corruption, we first apply  $c_{1}$  and then  $c_{2}$  at their respective severities (see Figure 2a).
Calibration to desired baseline accuracy In order to control baseline accuracy, we need to know how difficult each combination of 2 noises and their respective severities is. To that end, we first select a subset of 5,000 images from the ImageNet validation set. For each corruption  $(c_{1}, s_{1}, c_{2}, s_{2})$ , we corrupt all 5,000 images accordingly and evaluate the resulting images with a pre-trained ResNet-50 [10]. The resulting accuracy is what we refer to as baseline accuracy and what we use for controlling difficulty. In total, we evaluate more than 463 million corrupted images. Previous work, [12], measures normalized accuracy using AlexNet [19], which is less pertinent in present-day contexts. In addition, the accuracy of non-adapting Vision Transformers are stable on CCC as well (Figure 4).
Generating Benchmark Runs Having calibrated the corruptions pairs, we prepare benchmark runs with different baseline accuracies, transition speeds, and noise orderings. We pick 3 different baseline accuracies:  $34\%$ ,  $17\%$ , and  $2\%$  (CCC-Easy, CCC-Medium, CCC-Hard respectively). For each one of the difficulties, we select a further 3 transition speeds: 1k, 2k, 5k. Lastly, for each difficulty and transition speed combination we use 3 different noise orderings, determined by 3 random seeds. To generate each run, we first select the initial corruption at the severity which according to our calibration is closest to the desired baseline accuracy. We then transition to the second corruption of the noise ordering by repeatedly either decreasing the severity of the first noise by 0.25 or increasing the severity of the second noise by 0.25 such that the baseline accuracy is as close to the target as possible (see Figure 2). In each step along each path, we sample 1k, 2k, or 5k images from the
ImageNet validation set depending on the desired transition speed. Each image is randomly cropped and flipped for increasing the diversity of the dataset, and then corrupted.
Once the path from the initial to the second corruption is finished, the process is repeated for transitioning to the third corruption and so on (for more details see Appendix B). In the end, we have 3 difficulties consisting of 9 benchmark runs each. CCC-Medium at a speed of  $2\mathrm{k}$  corresponds roughly to CIN-C's difficulty and transition speed.
3 RDumb: Turning your model off and on again
Continual test-time adaptation needs to successfully adapt models over arbitrarily long timescales during deployment. Resetting a model to its initial weights at fixed intervals fulfills this criterion by design, yet allows to benefit from adaptation over short time scales. Surprisingly, such an approach has never been tried before (see Appendix E for more discussion).
Regarding the choice of the adaptation loss, we build on the weighted entropy used in ETA [30]. For a stream of input images  $\mathbf{x}_1,\mathbf{x}_2,\ldots$ , we compute class probabilities  $\mathbf{y}_t = f_{\Theta_t}(\mathbf{x}_t)$  and optimize the loss function
$$
L \left(\mathbf {y} _ {t}; \bar {\mathbf {y}} _ {t - 1}\right) = \left(\frac {\mathbb {1} \left[ \left(| \cos \left(\mathbf {y} _ {t} , \bar {\mathbf {y}} _ {t - 1}\right) | <   \epsilon\right) \wedge (H \left(\mathbf {y} _ {t}\right) <   H _ {0}) \right]}{\exp \left(H \left(\mathbf {y} _ {t}\right) - H _ {0}\right)}\right) H \left(\mathbf {y} _ {t}\right) \tag {1}
$$
which weights the entropy  $H(\mathbf{y}_t) = -\mathbf{y}_t^\top (\log \mathbf{y}_t)$  of each prediction using the similarity to averaged previously predicted class probabilities,  $\overline{\mathbf{y}}_t = (\mathbf{y}_1 + \dots +\mathbf{y}_t) / t$ , and a comparison to a fixed entropy threshold  $H_{0}$ .  $\cos (\mathbf{u},\mathbf{v})$  refers to the cosine similarity between vectors  $\mathbf{u}$  and  $\mathbf{v}$ . At each step, (part of) the weights  $\Theta_t$  are updated using the Adam optimizer [17]. At every  $T$ -th step,  $\Theta_t$  is reset to the baseline weights  $\Theta_0$ . We use  $\epsilon = 0.05$  and  $H_0 = 0.4\times \ln 10^3$  following [30], and select  $T = 1000$  based on the holdout noises in IN-C (see Section 6).
We benchmark RDumb alongside a range of recently published TTA models. For all models, we use a batch size of 64. In all models, the BatchNorm statistics are estimated on the fly, and the affine shift and scale parameters are optimized according to a model-specific strategy outlined below.
- BatchNorm (BN) Adaptation [29, 38] estimates the BatchNorm statistics (mean and variance) separately for each batch at test time. The affine transformation parameters are not adapted.
- Tent [46] optimizes the entropy objective on the test set in order to update the scale and shift parameters of BatchNorm (in addition to learning the statistics).
- Robust Pseudo-Labeling (RPL) [35] uses a teacher-student approach in combination with a label noise resistant loss.
- Conjugate Pseudo Labels (CPL) [8] use meta learning to learn the optimal adaptation objective function across a class of possible functions.
- Soft Likelihood Ratio (SLR) [28] uses a loss function that is similar to entropy, but without vanishing gradients. Anti-Collapse Mechanism: An additional loss is used to encourage the model to have uniform predictions over the classes, and the last layer of the network is kept frozen.
- Continual Test Time Adaptation (CoTTA) [48] uses a teacher student approach in combination with augmentations. Anti-Collapse Mechanism: Every iteration,  $0.1\%$  of the weights are reset back to their pretrained values.
- Efficient Test Time Adaptation (EATA) [30] uses 2 weighing functions to weigh its outputs: the first based on their entropy (lower entropy outputs get a higher weight), the second based on diversity (outputs that are similar to seen before outputs are excluded). Anti-Collapse Mechanism: An  $L_{2}$  regularizer loss is used to encourage the model's weights to stay close to their initial values.
- EATA Without Weight Regularization (ETA) For completeness, we also test ETA, which is EATA but without the regularizer loss, proposed in [30].
- RDumb is our proposed baseline to mitigate collapse via resetting. We reset every  $T = 1,000$  steps, as determined by a hyperparameter search on the holdout set (Section 6).
(b)
Figure 3: Adaptation performance of all evaluated models depending on the number of observed samples so far. (a) CIN-C. Model performances are averaged over the 10 runs of the benchmark. (b) CCC. Model performances are averaged over the 27 runs of the three difficulty levels. See Appendix C, Figure 10 for separate plots for CCC Easy, Medium and Hard.
Following the original implementations, Tent, ETA, EATA, and RDumb use SGD with a learning rate of  $2.5 \cdot 10^{-4}$ . RPL uses SGD with a learning rate of  $5 \cdot 10^{-4}$ . SLR uses the Adam optimizer with a learning rate of  $6 \cdot 10^{-4}$ . CoTTA uses SGD with a learning rate of 0.01, and CPL uses SGD with a learning rate of 0.001.
CCC reveals collapse during continual adaptation, unlike CIN-C. For three models that were evaluated, evaluation on CIN-C yielded inconclusive or inaccurate results in detecting collapse: CoTTA collapses on CCC, while CIN-C shows it to be on a downward trend, but end performance still outperformed the baseline (Figure 3a). Additionally, ETA shows no signs of collapse on CIN-C, while collapsing very clearly on CCC (Figure 3b, more precisely on CCC-Medium and CCC-Hard, see Appendix Figure 10). When tested using ViT backbones, EATA is better than the pretrained model on CIN-C (Figure 4a), but worse than the pretrained model on CCC (Figure 4b, Table 2,3). Lastly, SLR on CIN-C appears to be somewhat stable, but only at around  $10\%$  accuracy. CCC reveals this to be only partly true: on CCC-Hard, SLR is not stable and collapses to nearly chance accuracy.
Table 1: Mean accuracy of ResNet-50 models on CIN-C, CIN-3DCC and CCC. For each CCC split (Easy, Medium, and Hard), a mean of 9 runs is taken. For the CIN-C and CIN-3DCC experiments, the accuracy reported is the mean of 10 different noise permutations. Grey indicates collapse.
Adaptation method	CIN-C	CIN-3DCC	CCC-Easy	CCC-Medium	CCC-Hard	Average
Pretrained [10]	18.0 ± 0.0	31.5 ± 0.0	34.1 ± 0.22	17.3 ± 0.21	1.5 ± 0.02	20.5
BN [29, 38]	31.5 ± 0.02	35.7 ± 0.02	42.6 ± 0.39	27.9 ± 0.74	6.8 ± 0.31	28.9
Tent [47]	15.6 ± 3.5	24.4 ± 3.5	3.9 ± 0.58	1.4 ± 0.17	0.51 ± 0.07	9.2
RPL [35]	21.8 ± 3.6	30.0 ± 3.6	7.5 ± 0.83	2.7 ± 0.36	0.67 ± 0.14	12.5
SLR [28]	12.4 ± 7.7	12.2 ± 7.7	22.2 ± 18.4	7.7 ± 9.0	0.66 ± 0.57	11.0
CPL [8]	3.0 ± 3.3	5.7 ± 3.3	0.41 ± 0.06	0.22 ± 0.03	0.14 ± 0.01	1.9
CoTTA [48]	34.0 ± 0.68	37.6 ± 0.68	14.9 ± 0.88	7.7 ± 0.43	1.1 ± 0.16	19.1
EATA [30]	41.8 ± 0.98	43.6 ± 0.98	48.2 ± 0.6	35.4 ± 1.0	8.7 ± 0.8	35.5
ETA [30]	43.8 ± 0.33	42.7 ± 0.33	41.4 ± 0.95	1.1 ± 0.43	0.23 ± 0.05	25.8
RDumb (ours)	46.5 ± 0.15	45.2 ± 0.15	49.3 ± 0.88	38.9 ± 1.4	9.6 ± 1.6	37.9
(b)
Figure 4: TTA using a ViT backbone: (a) On CIN-C, EATA is better than the pretrained baseline  $(44.4\%$  points vs  $40.1\%$  points). (b) On CCC-Medium, EATA is worse than the pretrained baseline  $(38.5\%$  points vs  $42.0\%$  points). RDumb (ours) is consistently better than both EATA and the baseline.
Table 2: Mean accuracy of different backbone architectures on CCC-Medium. Accuracy reported is an average across 9 runs. Backbones used: [3, 10, 23, 43, 51], †: AugMix [14], ‡: DeepAugment [11]. Grey indicates collapse.
Method	RN18	RN34	RNSO	RN50†	RN50‡	RNX101†‡	VITB16	MaxYiT	SwinYiT
Pretrained	12.2	17.3	17.3	27.9	38.9	47.8	42.0	45.1	33.2
EATA	26.8	30.8	35.4	46.5	52.3	58.5	38.5	47.1	35.6
RDumb	32.5	37.2	38.9	47.0	51.9	58.4	50.2	49.9	36.5
In summary, models evaluated on CCC show clear limits, which are impossible to see on CIN-C because of the high difficulty variance between runs, and its short length.
RDumb is a strong baseline for continual adaptation. RDumb outperforms all previous methods on both established benchmarks (CIN-C, CIN-3DCC) as well as our continual adaptation benchmark, CCC (Table 1 and Figure 3). Concretely, we outperform EATA and increase accuracy by more than  $11\%$  on CIN-C (improving from  $41.8\%$  points to  $46.5\%$  points), and by almost  $7\%$  when averaged all evaluation datasets. While not able to outperform RDumb, we note that EATA is also a strong method for preventing collapse except for the counterexample in Table 2.
The results transfer to Imagenet-3D Common Corruptions. To further demonstrate the effectiveness of our method, we show results on Imagenet-3DCC [16], which features 12 types of corruptions, which take the geometry and distances between objects into account when applied to an image. Similarly to CIN-C, we test our models on 10 different permutations of concatenations of all the noises of IN-3DCC, which we call CIN-3DCC. As in the case of CIN-C and CCC, RDumb outperforms all previous methods (Table 1).
The results transfer to Vision Transformers. To further validate our claims, we test both EATA and our method with a Vision Transformer (ViT, 3) backbone (Figure 4). The difference in average accuracy between our method and EATA is larger when using a ViT, as compared to a ResNet-50: on CIN-C and CCC the gap is  $10.9\%$  points and  $11.7\%$  points respectively. Additionally, EATA's accuracy on CCC is below that of a pretrained, non-adapting model<sup>2</sup>. This collapse can only be seen by using CCC, and not when evaluating on CIN-C.
2Increasing the regularizer parameter value does not help stabilize the model, see Appendix D.
Table 3: Mean accuracy of different backbone architectures on CCC-Hard. Accuracy reported is an average across 9 runs. Backbones used: [3, 10, 23, 43, 51], †: AugMix [14], ‡: DeepAugment [11]. Grey indicates collapse.
Method	RN18	RN34	RNSO	RN50†	RN50†‡	RNX101†‡	ViTB16	MaxVIT-T	SwinVIT-T
Pretrained	0.82	1.3	1.5	5.6	24.3	15.6	22.0	22.0	9.3
EATA	6.4	7.6	8.7	17.7	30.3	36.3	8.6	15.4	8.6
RDumb	8.3	10.7	9.6	14.7	29.9	35.6	23.8	22.0	8.0