SlowGuess/NeurIPS_2023_OCR huggingface dataset:
C CCC Plots
(c)
Figure 10: Adaptation performance of all evaluated models using a ResNet-50 backbone. (a) CCC Easy. (b) CCC Medium. (c) CCC Hard. For all subplots, model performances are averaged over the 9 runs of the respective difficulty level.
D EATA Implementation and Ablations
Our implementation of EATA differs from the official implementation<sup>3</sup>. The reason for this is that the official implementation uses clean ImageNet validation images to calculate the Fisher vector matrix for its regularizer<sup>4</sup>. This stands in contradiction with the method, which should not have access to the training distribution at test time, as shown in the paper in Table 1.
Instead of using 2,000 ImageNet validation images, we calculate the Fisher matrix using the first 2,000 images in our data stream. We conduct a hyperparameter search on the weight regularizer tradeoff parameter  $\beta$ :
β	25	50	100	250	500	1000	1500	2000
Acc. [%]	46.5	46.9	47.1	46.7	46.1	45.6	44.8	44.0
Using the optimal value, 100 led to worse results than the default value, 2000, on CCC:
Table 7: Accuracy of EATA on CIN-C holdout noises for different values of the weight regularizer loss.
CIN-C	CCC-Easy	CCC-Medium	CCC-Hard	CCC Avg
EATA-100	46.7	47.7	36.5	3.8	29.3
EATA-2000	41.8	48.2	35.4	8.7	30.8
Ours	46.5	49.3	38.9	9.6	32.6
Table 8: Accuracy of EATA on CIN-C holdout noises for different values of the Fisher alpha.
In the end, we used the original value of 2000, as that was optimal on the CCC dataset.
In addition, we conducted a hyperparameter search for EATA on a ViT backbone. As shown in 4, EATA performs worse than a pretrained, non-adapting baseline in this setting. To that end, we tried to stabilize the model by increasing the value of  $\beta$ , the hyperparameter that controls the weight of the anti-forgetting regularizer. As with the previous experiment, the original value of 2000 is optimal.
$^{3}$ https://github.com/mr-eggplant/EATA, version f739b3668c  
 $^{4}$ https://github.com/mr-eggplant/EATA/blob/f739b3668c/main.py#L144
β	2000	3000	4000
Acc. [%]	38.5	27.8	16.5
Table 9: Accuracy of EATA on CCC-Medium using a ViT backbone, for different values of the regularizer,  $\beta$ .
E Novelty of Resetting
Our work is the first to propose resetting to solve collapse in TTA methods. Notably, while prior work [30, 47, 54] has briefly touched upon the concept of episodic resetting, the methodology and its application is significantly distinct and unrelated to collapse in TTA.
- Tent [47] mentions episodic in the context of overfitting to a single sample in segmentation (similar to [54]'s overfitting to a single sample). Resetting here is unrelated to collapse, as the paper doesn't discuss collapse at all.
- Although MEMO [54] uses resetting, it does so because it overfits to one image (and its augmentations) every step. MEMO doesn't discuss collapse or catastrophic forgetting. MEMO compares itself to a version of Tent that resets after every step (which they call Tent + episodic resetting), because MEMO without augmentations is similar to Tent + episodic resetting with a batch size of 1. (Note: MEMO is outperformed by BN/Tent/ETA when using the standard batch size of 64)
- EATA [30] shows results for Tent + episodic resetting after every step in its tables, but provides no reasoning or discussion for doing this. Tent + episodic resetting is outperformed by regular Tent.
F CIFAR10 Experiments
We conduct CIFAR10 experiments to show the need for ImageNet scale benchmarks. Tent, without an anti-collapse mechanism, does not collapse on CIFAR10-C, even after seeing 100 million images Fig. 11a,b. Like ImageNet, CIFAR10-C's noises also exhibit high variance in difficulty Fig. 11c.
Figure 11: (a) When tested on an infinite concatenation of severity 5 noises, Tent does not collapse even after seeing 100M CIFAR scale images. (b) Tent does not collapse to chance level when tested on a long term variant of CIFAR10-C gradual. (c) CIFAR10-C exhibits great variations between individual corruptions, similar to ImageNet-C.
(c) Pretrained Accuracy on C10-C Severity 5 Noises
G Compute details
We conduct all experiments on Nvidia RTX 2080 TI GPUs with 12GB memory per device. All experiments except our study on larger models were conducted on a single GPU. For CoTTA experiments, we use data parallel training on 2 GPUs. A bulk of the compute spent for this work was on computing baseline accuracies on the calibration dataset, which contains 463M images.
H Software and Dataset Licenses
- ImageNet-C [12]: Creative Commons Attribution 4.0 International, https://zenodo.org/record/2235448
- ImageNet-C [12], code for generating corruptions: Apache License 2.0 https://github.com/hendrycks/robustness
- ImageNet-3D-CC [16]: CC-BY-NC 4.0 License
https://github.com/EPFL-VILAB/3DCommonCorruptions
H.2 Models
- PyTorch's [31] Backbones
https://pytorch.org/vision/stable/models.html
- Adaptive BN [29, 38]: Apache License 2.0, https://github.com/bethgelab/robustness
- Tent [47]: MIT License, https://github.com/DequanWang/tent
- RPL [35]: Apache License 2.0, https://github.com/bethgelab/robustness
- CoTTA [48]: MIT License, https://github.com/qinenergy/cotta
CPL [8]: MIT License, https://github.com/locuslab/tta_conjugate
- EATA [30]: MIT License https://github.com/mr-eggplant/EATA