DELTA: DEGRADATION-FREE FULLY TEST-TIME ADAPTATION
Authors: Bowen Zhao
Chen Chen
Shu-Tao Xia
Venue: ICLR 2023 poster (2023)

Abstract: Fully test-time adaptation aims at adapting a pre-trained model to the test stream during real-time inference, which is urgently required when the test distribution differs from the training distribution. Several efforts have been devoted to improving adaptation performance. However, we find that two unfavorable defects are concealed in the prevalent adaptation methodologies like test-time batch normalization (BN) and self-learning. First, we reveal that the normalization statistics in test-time BN are completely affected by the currently received test samples, resulting in inaccurate estimates. Second, we show that during test-time adaptation, the parameter update is biased towards some dominant classes. In addition to the extensively studied test stream with independent and class-balanced samples, we further observe that the defects can be exacerbated in more complicated test environments, such as (time) dependent or class-imbalanced data. We observe that previous approaches work well in certain scenarios while show performance degradation in others due to their faults. In this paper, we provide a plug-in solution called DELTA for Degradation-freE fuLly Test-time Adaptation, which consists of two components: (i) Test-time Batch Renormalization (TBR), introduced to improve the estimated normalization statistics. (ii) Dynamic Online re-weighTing (DOT), designed to address the class bias within optimization. We investigate various test-time adaptation methods on three commonly used datasets with four scenarios, and a newly introduced real-world dataset. DELTA can help them deal with all scenarios simultaneously, leading to SOTA performance.

Published as a conference paper at ICLR 2023
DELTA: DEGRADATION-FRE FULLY TEST-TIME ADAP- TATION*
Bowen Zhao' Chen Chen Shu-Taa Xia 14.8
'Tsinghua University, Teeneett TEG AI, OPPO research institute, *Peng Cheng Laboratory zbw mails. tsinghua.cdu.cn, chen 664chen @ gmail.com, xiast @ ww..tsinguua..ee edu.cn
ABSTRACT
Fully test-time adaptation aims at adapting a pre-trained model to the test stream during real-time inference, which is urgently required when the test distribution differs from the training distribution. Several efforts have been devoted to improv- ing adaptation performance. However, we find that two unfavorable defects are concealed in the prevalent adaptation methodologies like test-time batch normal- ization (BN) and self-lcarning. First, we reveal that the normalization statistics in test-time BN are completely affected by the currently received test samples, resulting in inaccurate estimates. Second, we show that during test-time adap- tation, the parameter update is biased towards some dominant classes. In addi- tion to the extensively studied test stream with independent and blass-balanced samples, we further observe that the defects can be exacerbated in more compli- cated test environments, such as (time) dependent or plass-imalaanced data. We observe that previous approaches work well in certain scenarios while show per- formance degradation in others due to their faults. In this paper. we provide a plug-in solution called DELTA for Degradation-fret: ful.yy Test-time Adaptation, which consists of two components: (i) Test-time Batch Renormalization (TBR), introduced to improve the estimated normalization statistics. (ii) Dynamic Online we-eehhTing (DOT), designed to address the class bias within optimization. We investigate various test-time adaptation methods on three commonly used datasets with four scenarios, and newly introduced real-world dataset. DELTA can help them deal with all scenarios simultancously, leading to SOTA performance.
INTRODUCTION
Models suffer from performance decrease when test and training distributions are mis- matched (Quinonero-Cande) et al., 2008). Numerous studies have been conducted to narrow the performance gap based on a variety of hypotheses/setting. Unsupervised domain adaptation meth- ods (Ganin et al., 2016) necessitate simultaneous access to labeled training data and unlabeled target data, limiting their applications. Source-free domain adaptation approaches (Liang et al., 2020) only need trained model and do not require original training data when performing adaptation. Nonethe- less, in a more difficult and realistic setting. known as fully test-time udaptation (Wang et al., 2021), the model must perform online adaptation to the test stream in real-time inference. The model is adapted in a single pass on the test stream using a pre-trained model and continuously arriving test data (rather than a prepared target set). Offline iterative training or extra heavy computational burdens beyond normal inference do not meet the requirements.
There have been several studies aimed at fully test-time adaptation. Test-time BN (Nado et al., 2020) / BN adapt (Schneider et al., 2020) directly uses the normalization statistics derived from test samples instead of those inherited from the training data, which found to be beneficial in reducing the performance gap. Entropy- Entropy-minimization minimization-baas basedd methods, such as TENT (Wang et al., 2021), further optimize model parameters during inference. Contrastive learning (Chen et al., 2022), data augmentation (Wang al., 2022a) and uncertainty-aware optimization (Niu al., 2022) have been introduced to enhance adaptation performance. Efforts have also been made to address test-time adaptation in complex test environments, like LAME (Boudiaf et al., 2022).
work done by Bowen Zhao (during internship) and Chen Chen at Tencent.

---

Published as a conference paper at ICLR 2023
Table 1: Comparison of fully test-time adaptation methods against the pre- trained model on CIFARI00-C, DELTA achieves improvement in all scenarios.
IS+CB
DS+CB
IS+CI
DS+CI
Figure 1: IS+CB / DS+CB: the test stream which is inde-
pendently/ / dependently sampled from a class-balanced test distribution; IS+CI/ DS+CI: independently dependently drawn from class-imbalanced test distribution. Each bar represents a sample, each color represents a category.
Samurio
TENT
LAME
DELTA (Oars)
DS-CH I5-CI
DS1CI
Despite the achieved progress, we find that there are non-negligible defects hidden in the popular methods. First, we take a closer look at the normalization statistics within inference (Section 3.2). We observe that the statistics used in BN adapt is inaccurate in per batch compared to the actual pop- ulation statistics. Second, we reveal that the prevalent test-time model updating is biased towards some dominant categories (Section 3.3). We notice that the model predictions extremely imbal- adaptation
anced on out-of-distribution data, which can be exacerbated by the self-learning-bass methods. Besides the most common independent and class- baass-balanced test samples considered in isting studies, following Boudiaf et al. (2022), we investigate other three test scenarios as illustrated in Figure 1 (please see details in Section 3.1) and find when facing the more intricate test streams, like dependent samples or class-imbalanced data, the prevalent methods would suffer from severe performance degradation, which limits the usefulness of these test-time adaptation strategies.
To address the aforementioned issues, we propose two powerful tools. Specifically, to handle the in- accurate normalization statisties, we introduce test-time batch renormalization (TBR) (Section 3.2), which uses the test-time moving averaged statistics to rectify the normalized features and considers normalization during gradient optimization. By taking advantage the observed test samples, the calibrated normalization is more accurate. We further propose dynamic online re- re-weighting (DOT) (Section 3.3) to tackle the biased optimization, which is derived from cost-sensitive learning. To bal- ance adaptation, DOT assigns low/high weights to the frequent/infrequent categories. The weight mapping function is based on a momentum-updleee upddttd class-frequency vector that takes into account multiple sources of category bias, including the pre-trained model, the test stream, and the adap- tation methods (the methods usually do not have an intrinsic bias towards certain classes, but can accentuate existing bias). TBR can be applied directly the common BN-based pre-trate pre-trained mod- els and does not interfere with the training process (corresponding to the fully test-time adaptation setting), and DOT can be easily combined with other adaptation approaches as well.
Table compares our method to others on CIFAR100-C across various scenarios. The existing test-time adaptation methods behave differently across the four scenarios and show performance degradation in some scenarios. While our tools perform well in all four scenarios simultancously without any prior knowledge of the test data, which is important for real-world applications. Thus, the whole method is named DELTA (Degradation freE fuLly Test-time Adaptation).
The major contributions of our work are as follows. We expose the defects in commonly used test-time adaptation methods, which ultimately harm adaptation performance. (ii) We demonstrate that the defects will be even more severe complex test environments, causing performance degra- dation. (iii) To achieve degradation-free fully test-time adaptation, we propose DELTA which com- prises two components: TBR and DOT, to improve the normalization statistics estimates and mit- igate the bias within optimization. (iv) We evaluate DELTA three common datasets with four scenarios and a newly introduced real-world dataset, and find that it can consistently improve the popular test-time adaptation methods on all scenarios, yielding new ttate-of-the--e results.
2 RELATED WORK
Unsupervised domain adaptation (UDA). In reality, test distribution is frequently inconsistent with the training distribution, resulting in poor performance. UDA aims to alleviate the phenomenon with the collected unlabeled samples from the target distribution. One popular approach is to align the sta- tistical moments across different distributions (Gretton et al., 2006; Zellinger et al., 2017; Long et al., 2017). Another line of studies adopts adversarial training achieve adaptation (Ganin et al., 2016;
2

---

Published as a conference paper at ICLR 2023
Long al., 2018). UDA has been developed for many tasks including object classification (Saito et al., 20177detectin (Li et al., 2021) and semantic segmentation (Hoffman et al., 2018).
Source-free domain adaptation (SFDA). SFDA deals with domain gap with only the trained model and the prepared unlabeled target data. To be more widely used, SFDA methods should be built on common source model trained by standard pipeline. SHOT (Liang et al., 2020) freezes the source model's classifier and optimizes the feature extractor via entropy minimization, diversity regularization, and pseudo-labeling. SHOT incorporates weight normalization, ID BN, and label- smoothing into backbones and training, which do not exist in most off-the-shelf trained models, but its other ideas can be used. USFDA (Kundu al., 2020) utilizes synthesized samples to achieve compact decision boundaries. NRC (Yang et al., 2021b) encourages label consistency among local target features with the same network architecture as SHOT. GSFDA (Yang et al., 2021a) a) further expects the adapted model performs well not only on target data but also on source data.
Fully test-time adaptation (FTTA). FTTA is a more difficult and realistic setting. In the same way that SFDA does not provide the source training data, only the trained model is provided. Unlike SFDA, FTTA cannot access the entire target dataset; however, the methods should be capable of do- ing online adaptation on the test stream and providing instant predictions for the arrived test samples. BN adapt (Nado et al., 2020; Schneider et al., 2020) replaces the normalization statistics estimated during training with those derived from the test mini-batch. On top of it, TENT (Wang et al., 2021) optimizes the affine parameters in BN through entropy minimization during test. EATA (Niu et al., 2022) and CoTTA (Wang et al., 2022a) study long-term test-time adaptation in continually changing environments. ETA (Niu et al., 2022) excludes unreliable and redundant samples from the opti- mization. AdaContrast (Chen et al., 2022) resorts contrastive learning to promote feature learning along with a pseudo label refinement mechanism. Both AdaContrast and CoTTA utilize heavy data augmentation during test, which will increase inference latency. Besides, AdaContrast modifies the model architecture as in SHOT. Different from them, LAME (Boudiaf et al., 2022) does not rectify the model's 'ss parameters but only the model's output probabilities via the introduced unsupervised objective laplacian adjusted maximum-likellhen estimation.
Class-imbalanced learning. Training with class-imbalanced data has attracted widespread atten- tion (Liu et al., 2019). Cost-sensitive learning (Elkan, 2001) and resampling (Wang al., 2020) are the classical strategies to handle this problem. Ren et al. (2018) designs a meta-learning paradigm to assign weights samples. Class-balancee loss (Cui al., 2019) uses the effective number of samples when performing re-weighting. Decoupled training (Kang et al., 2020b) learns the feature extractor and the classifier separately. Menon et al. (2021) propose logit adjustment from a statis- tical perspective. Other techniques such as weight balancing (Alshammari et al., 2022; Zhao et al., 2020), contrastive learning (Kang et al., 2020a), knowledge distillation (He et al., 2021), etc. have also been applied to solve this problem.
3 DELTA: DEGRADATION-FRET FULLY TEST-TIME ADAPTATION
3.1 PROBLEM DEFINITION
Assume that we have the training data Dirain = [xx, v:) ptrain (zr y). where E X' is the input and y E y' = (1,2,.... K} is the target label; [100 ao) denotes the model with parameters Og and normalization statistics an learned or estimated on Dimin, Without loss of generality, we denote the test stream as Dies= = ((x,, (;);;;))))) Litee } pist (zz v). where {u;) are not available actually, the subscript j also indicates the sample position within the test stream. When (x, y) # pirain (x, y) (the input/output space X'/3² is consistent between training and test data). f(o..ao) may perform poorly on Disst Under fully test-time adaptation scheme (Wang et al., 2021), during inference step L. 2 1, the model f(0.-1.9-111 receives a mini-batch of test data {xm++b) b=1 with B batch size (m4 the number of test samples observed before inference step t), and then elevates itself to f(o,.a) based
on current test mini-batch and outputs the real-time predictions {pm,+t) B (D E R*). Finally, the evaluation metric is calculated based on the online predictions from each inference step. Fully test-time adaptation emphasizes performing adaptation during real-time inference entirely, i.e. the training process cannot be interrupted, the training data is no longer available during test, and the adaptation should be accomplished in single pass over the test stream.
3

---

Published as a conference paper at ICLR 2023
The most common hypothesis is that Diest is independently sampled from Pie y). However, in real environment, the assumption does not always hold, e.g., samples of some classes may appear more frequently in certain period of time, leading to another hypothesis: the test samples are depen- dently sampled. Most studies only considered the scenario with Hass-balanced test samples, while in real-world, the test stream can be class-imbalanced! We investigate fully test-time adaptation under the four scenarios below, considering the latent sampling strategies and the test class distri- bution. For convenience, we denote the scenario where test samples are independentll//eeeeddenn sampled from a class- baass-balancedd test distribution as IS+CB / DS+CB; denote the scenario where test samples are sespenndentyydpeedeenly. sampled from a caass-imallanned test distribution as IS+CI/ DS+CI, as shown in Figure 1. Among them, IS+CB is the most common scenario within FTTA studies, and the other three scenarios also frequently appear in real-world applications.
3.2 A CLOSER LOOK AT NORMALIZATION STATISTICS
We revisit BN (loffe & Szegedy, 2015) briefly. Let v E BB be a mini-batch of features with C' channels, height S and width S'. BN normalizes u with the normalization statistics μe, RCC V² 1ー v-' = 7 V² + 3, where 7,3 E ICC are the learnable affine parameters, (y.3) C 0. We mainly focus on the first part + v² (all the discussed normalization meth- ods adopt the affine parameters). In BN, during training. fe, a are set to the empirical mean μ bathh and standard deviation o hutch calculated for each channel f hhalchhc [c]
BSS Eb.e,e' b(b, c, s'),
1 Ed,,.r (u(b, C, 81] puotchhcc) + e, where is a small value to avoid division batoh [c] = V BSS
by zero. During inference, 1i, a are set to urma oeme which the exponential-moving moving-average (EMA) estimates over training process (ao is formed by the EMA statistics of all BN modules).
However, when (x. * patrai [z, y). studies found that replacing μ م o with the statistics of the test mini-batch: fihatchh ghatch can improve model accuracy (Nado et al., 2020) (for clarify, statistics estimated on test samples are denoted with '). The method is also marked as "BN adapt" (Schneider et al., 2020).
nommttaatin
TEMA
Normalization statistics are inaccurate within each test mini-batch. We conduct experiments on CIFARI00-C. From Figure we can see that the statistics hhatch hatch used in BN adapt fluctuate dramatically during adaptation, and are inaccurate in most test mini-batches. It made
Diagnosis I:
Test min batch
Test mára butch
(a) p.. IS+CB
(b) μ.. DS+CB
should be noted that for BN adapt, predictions online based on real-time statistics, so poor estimates can have negative impact on performance. More seriously, the estimates in the DS+CB scenario are worse. In Ta- ble 2, though BN adapt and TENT can improve accuracy compared to Source (test with the fixed pre-trained model f(80.00)) in IS+CB scenario, they suffer from degradation in the DS+CB cases. Overall, we can see that the poor statistics severely impede test-time adaptation because they are derived solely from the current small mini-batch.
Grobat wdwaddtt-eeee
m mibbcch
(c IS+CB
(d) 0, DS+CB
Figure 2: Normalization statistics in different scenarios on CIFAR 100-C.
Treatment I:
Test-time batch renormalization (TBR) simple and powerful tool to improve the normaliza- tion. It is natural to simply employ the test-time moving averages fema gema to perform normalization during adap- tation, referring to as TEMA, where i.mma = a 1 + (1 (1-a)- sg(puucch, armm = a @ + (1-a) a) sg( (āhulch). sg(.) stands for the operation of stopping gradient, e.g., the Tensor.detachf function in PyTorch, a is a smoothing coef-
Table 2: Average accuracy (%) of 15 corrupted sets on CIFARIOG
Method
IS+CB
DS+CB
Source
53.5
53.51..00
BN udapt
64.3
27.3
BN adapt+TEMM
64.8
63.5+0.51
TENT
68.5
23.7+1.04
TENT+TEMA
21.8.2.51
26.2
TENT+TBR
68.8...13
64.1
'Reaarding training class distribution, in experiments, we primarily use models learned on balanced training
data following the benchmark of previous studies. Furthermore, when pitin (y) is skewed, some techniques are commonly used to bring the model closer to the one trained on halanced data, such YTBB-sub (Section 4), where the trained model is learned with logit adjustment on class-imbalanced training data.
4

---

Published as a conference paper at ICLR 2023
ficient. TEMA consistently improve BN adapt: the normalization statistics in Figure 2 become more stable and accurate, and the test accuracy in Table is improved as well.
However, for TENT which involves parameters update, TEMA destroy the trained model as shown in Table 2. As discussed in loffe & Szegedy (2015), simply employing the moving averages would neutralize the effects of gradient optimization and normalization, as the gradient descent optimization does not consider the normalization, leading to unlimited growth of model parameters. Thus, we introduce batch renormalization (loffe, 2017) into test-time adaptation, leading to TBR, which is formulated by
f batch
sg( h-utch gema
d sg(
(p
f
d,
where
i barch
Gema
We present a detailed algorithm description in Appendix A.2. Different from BN adapt, we use the test-time moving averages to rectify the normalization (through r and d). Different from the TEMA, TBR well compatible with gradient-bar based adaptation methods (e.g.. TENT) and can improve them summarised in Table 2. For BN adapt, TEMA is equal to TBR. Different from the original batch renormalization used in the training phase, TBR is employed in the inference phase which uses the statisties and moving averages derived from batches. Besides, the adaptation starts with trained model f[0,.ao]- TBR discards the warm-up and truncation operation to r and d, thus does TBR can be applied directly to common pre-traines trained
introduce additional model with BN without requiring the model to be trained with such calibrated normalization.
3.3 A CLOSER LOOK AT TEST-TIME PARAMETER OPTIMIZATION
Building on BN adapt, TENT (Wang et al., 2021) further optimizes the affine parameters 7, 3 through en- tropy minimization and shows that test-time parameter optimization can yield better results compared to em- ploying BN adapt alone. We further take closer look at this procedure.
Predictions
Prediction
Prodictoons
50
0'0
0'0
Sorted
Sorted
Sorted
(Cean S-Ca, Source)
(Gauss S-CD, DV adtpl=TTMM)
[Gause (eCC) TIN
(Saun, ITCC. Sourcs)
(Causs S-c0, T
(Gause DS-cn TD.T-TBR)
Figure 3: Per-class number of predictions under combina- tions of [data, scenario, method).
Diagnosis I: the test-time op- timization biased towards dominant classes. We evaluate the model on IS+CB and DS+CB coussian-noise-coruye
Table 3: Standard Deviation (STD), Range (R) of per-class
number of predictions and accuracy (Ace, %) Gauss data.
test data (Gauss) of CIFAR100-C. We also test the model on the original clean test of CIFAR 100 for comparison. Figure depicts the per-class number of predictions, while Table 3 shows the corresponding standard deviation, range (maximum subtract minimum), and accuracy. We draw the following five conclusions.
IS+CB
DS+CB
Method
STD
Acc
STD
Acc
Souree
158.3
956.0=0.0
27.0a00
158.3400
956.040.
27.0+00
BN TEMA
1844..2
121 121.6.a7
58 0100
19.8.1.1
130.0
56.7
TENT=TAR
35.8
269.8
62.2404
52.4
469
469.24104.2 $7.1
TENT+TBR-DOT
20.4g1.1 122.0 63.9202 25.5ga. 164.6443.0 60.4
Predictions are imbalanced, even for a model trained on Blass-balanee training data and tested balanced test set with piest (I., y/) ptriin (z. y): the "clean" curve in Figure 3 (left) with standard deviation 8.3 and range 46. This phenomenon is also studied in Wang et al. (2022b). Predictions becomes more imbalanced when piest y) * purinnz; y) as shown in Figure (left): the ranges are 46 and 956 on the clean and corrupted test set respectively.
BN adapt+TEMA improves accuracy (from 27.0% to 58.0%) and alleviates the prediction imbal- ance at the same time (the range dropped from 956 to 121.6).
Though accuracy is further improved with TENT+TBR (from 58.0% to 62.2%). the predictions become more imbalanced inversely (the range changed from 121.6 to 269.8). The entropy mini- mization loss focuses on data with low entropy, while samples some classes may have relatively lower entropy owing to the trained model, thus TENT would aggravate the prediction imbalance. On dependent test streams, not only the model accuracy drops, but also the predictions become more imbalanced (range 269.8 range 469.2 on independent/dependenn samples for TENT+TBR), as the model may be absolutely dominated by some classes over a period of time in DS+CB scenario.
5

---

Published as a conference paper at ICLR 2023
Algorithm 1: Dynamic Online reweighTinng (DOT)
Input: inference step 1 0: test stream samples (2;}; pre-trained model /(80.a0): dlass-frequenee vector zo; loss function L; smooth coefficient X.
while the test mini-batch (Zm,-bl6=1 18 arrives do
f= ++++1
[pm, 1 1(0-1.aa)) <- Forward( + S ((0-11 // output predictions
for = 1 to B do
km b = arg max ke( Pm // stedicted label
Wmy+b 1/ (2 (omy // assign sample weight
Wm = B Wm En= Wm, b= 1,2, , normalize sample weight
1= B 2bn1 Wm L(pme+s) // combine sample weight with loss
((Dnar) + Backward & Update(l, ((0 // update
10
21 < A (1-A)
Eb=1 Pmuth // update 2
The imbalanced data is harmful during the normal training phase, resulting in biased models and poor overall accuracy (Liu et al., 2019; Menon et al., 2021). Our main motivation is that the test-time adaptation methods also involve gradient-based optimization which is built on the model predictions; however, the predictions are actually imbalanced, particularly for dependent or class- dlassimballanced streams and the low-entropy-empen emprapy--empasizzedd adaptation methods. Therefore, we argue that the test-time optimization is biased towards some dominant classes actually, resulting in inferior performance. A vicious circle formed by skewed optimization and imbalanced predictions.
Treatment I: Dynamic online re-weighting (DOT) can alleviate the biased optimization. Many methods have been developed to deal with class imbalance during the training phase, but they face several challenges when comes to fully test-time adaptation: (i) Network architectures immutable. (ii) Because test sample class frequencies are dynamic and agnostic, the common constraint of making the output distribution uniform (Liang et al., 2020) is no longer reasonable. (iii) Inference and adaptation must occur in real-time when test mini-batch arrived (only single pass through test data, no iterative learning).
Given these constraints, we propose DOT presented in Algorithm 1. DOT mainly derived from class-wise re-weighting (Cui et al., 2019). To tackle the dynamically changing and unknown class frequencies,
momentum-upddated class-frequency vector RR instead (Line 10 of Algorithm 1), which is initiated with z[k] K. k = 1, K. For each inference step, assign weights to each test sample based on its pseudo label and the current z (Line 5.6 of Algorithm 1). Specifically, when 2 is relatively large, during the subsequent adaptation, DOT will reduce the contributions of the 1thh class samples (pseudo label) and emphasize others. It is worth noting that DOT can alleviate the biased optimization caused by the pre-trained model (e.g.. inter-class similarity), test stream (e.g., class-imbalanced scenario) simultaneously.
DOT is general idea to tackle the biased optimization, some parts in Algorithm have multi- ple options, so can be combined with different existing test-time adaptation techniques. For the "Forward (-)" function (Line 3 of Algorithm 1), the discussed BN adapt and TBR can be in- corporated. For the loss function L(-) (Line 8 of Algorithm 1), studies usually employ the en- tropy minimization loss: C(pb) = "EK, Po[k] log po[k] or the cross-entropy loss with pseudo labels: C(pb)
-pp111112 log pb [ (commonly, only samples with high prediction con- fidence are utilized,
pre-defined threshold). Similarly, for entropy minimization, Ent- W (Niu et al., 2022) also discards the nigh-entropy samples and emphasizes the low-entropy ones: log Ps [kc], where H, is the entropy of sample T/..
((ps) = =T -Ho EK
EXPERIMENTS
Datasets and models. We conduct experiments on common datasets CIFAR100-C, ImageNet- C (Hendrycks & Dietterich, 2019), ImageNet-R (Hendrycks et al., 2021), and a newly introduced video (segments) dataset: the subset of BouTTube-BuundiggBxee (YTBB-sub) (Real et al., 2017). CIFAR 100-C ImageNet-C contains 15 commution types, each with 5 severity levels; we use the

---

Published as a conference paper at ICLR 2023
highest level unless otherwise specified. ImageNet-R contains various styles (e.g.. paintings) of Ima- geNet categories. Following Wang et al. (2022a); Niu et al. (2022), for evaluations on CIFARIO0-C, adopt the trained ResNeXt-29 (Xie et al., 2017) model from Hendrycks et al. (2020) as oo) for ImageNet-C/ -R, we use the trained ResNet-50 model from Torchvision. The models are trained on the corresponding original training data. For YTBB-sub, we use a ResNet--8 trained the related images of COCO. Details of the tasks, datasets and examples are provided in Appendix A.I.
Metries. Unless otherwise specified, we report the mean accuracy over classes (Acc, %) (Liu al., 2019); results are averaged over 15 different corruption types for CIFAR 100-C and ImageNet-C in the main text, please see detailed performance on each corruption type in Appendix A.5, A.6.
Implementation. The configurations are mainly followed previous work Wang et al. (2021; 2022a); Niu et al. (2022) for comparison, details are listed in Appendix A.3. Code available online.
Baselines. We adopt the following SOTA methods as base- lines: pseudo label (PL) (Lee et al., 2013), test-time aug- mentation (TTA) (Ashukha et al., 2020), BN adaptation (BN adapt) (Schneider et al., 2020; Nado et al., 2020), test-time en-
Table 4: Acc in IS+CB scenario.
Method
CIFARI00-C ImageNet-C
Source
53.510.00
18.0
TTA
17.7
tropy minimization (TENT) (Wang et al., 2021), marginal en- tropy minimization with one test point (MEMO) (Zhang et al., 2021), efficient test-time adaptation (ETA) (Niu et al., 2022), entropy-based weighting (Ent-W) (Niu et al., 2022), lapla- cian adjusted maximum-likelihood estimation estmiiaiii (LAME) (Boudiaf et al., 2022), continual test-time adaptation TC:TTN/OOTTA*: w/wo resetting) (Wang et al., 2022a). We combine DELTA with PL, TENT, and Ent-W in this work.
BN adapt
64.640.03
31.5
MEMO
23,9
ETA
19.3
48.0
LAME
50.810.08
17.2
COTTA
65.5
34.4=0.11
CoTTA
R
17.3
34.8
PL
68 68.040.13
40.2+0.11
+DELTA
68.7 68.7+0.12
41.8+0.08
+0.7
+1.6
TENT 68 68.740.16
42.7+0.08
Evaluation in IS+CB scenario. The results on CIFAR100-C are reported in Table 4. As can be seen, the proposed DELTA consistently improves the previous adaptation approaches PL (gain 0.7%). TENT (gain 0.8%), and Ent-W (gain 0.8%). achiev- ing new otate-of-theearr performance. The results also indi- cate that current test-time adaptation methods indeed suffer from the discussed drawbacks, and the proposed methods can help them obtain superior performance. Then we evaluate the meth- ods on the more challenging dataset ImageNet-C. Consistent with the results on CIFAR 100- C, DELTA remarkably improves the existing methods. As the adaptation batch size (64) is too small compared to the class number (1,000) ImageNet-C, the previous methods un- dergo more severe damage than on CIFAR100-C. Consequently, DELTA achieves greater gains on ImageNet-C: 1.6% gain over PL, 2.4% gain over TENT, and 5.6% gain over Ent-W.
+DELTA 69. 69.540.03
45.1
+0.8
+2.4
Ent- W
69.3
44.310.41
+DELTA
70.1
49.9
+0.8
+5.6
Evaluation in DS+CB scenario. To simu- late dependent streams, following Yurochkin et al. (2019), we arrange the samples via the Dirichlet distribution with a concentra- tion factor > O (the smaller is, the more concentrated the same-class samples will be, which is detailed in Appendix A. We test models with p E {1.0, 0.5, 0.1}. The exper- imental results are provided in Table 5 (we provide the results of more extreme cases with p 0.01 in Appendix A.4). The repre- sentative test- time adaptation methods suffer from performance degradation in the depen- dent scenario, especially on data sampled with small p. DELTA successfully helps models adapt to environments across different concentration factors. is worth noting that DELTA'S DS+CB results are close the IS+CB results, e.g., TENT+DELTA achieves 69.5% and 68.5% ac- curacy on IS+CB and DS+CB (p 0.5) test streams from CIFAR 100-C.
Table 5: Ace in DS+CB scenario with varying p.
CIFAR100-C
ImageNet-C
Method
1.0
0.5
0.1
1.0
0.5
0.1
Source
$3.5-0.00
53
23.5
18.020.0
18.0=a.00
18.02000
BN adapt
53.0=0.45
49.0.a.2
35.2.0.44
21.8
19.2_a.0
ETA
55.4_0.03
50.5
34.5
27.6
22.4_0.30
9.7
LAME
60.3-0.25
61.8422
65.420.41
21.9400s
22.7=a.05
24.7+008
CoTTA CoTTA*
53.8_0.0:
50.040:28
36.320.63
23.42016
20.5_0.03
12.6
54.1_0.6 50.2ga
36.1
27
20.3_a.55
12.81000
PL.
54.9_0.54
50.1
34.8
25.940.18
22.5_a.14
13.0g000
+DELTA
68.0_o.s
67.5ga.3
66.0_o.4
40.5
39.9_a.07
37.34010
+13.1
+17.4
+31.2
+14.6
+17.4
+24.3
TENT
54.6_0.52
49.74g.40
33.7
26.020
22.1_a.12
12.1
+DELTA
68.9 0.20 68.5
67.1
43.7
43.1
43.1_
40.3
+14.3
+18.8
+33.4
+17.7
+21.0
+28.2
Eat-W +DELTA
55.4 n.s
1aa5
14.5
17.41a.
13.0
am
4.1
69.4
68.8
67.120.45
48.3
47.4_0.4
43.2+a.11
+14.0
+18.3
+32.6
+30.9
+34.4
+39.1
Evaluation in IS+CI and DS+CI scenarios. Following Cui et al. (2019), we resample the test samples with an imbalance factor æ (the smaller π is, the more imbalanced the test data will be,
7

---

Published as a conference paper at ICLR 2023
Table 7: Results on in-distri-
Table 6: Mean acc in IS+CI, DS+CI scenarios with different I,
bution test set of RIFAR100.
IS+C1
DS+CI (.5)
Method
Accuracy
Method
CIFARI00CC
I
CIFARIOD
ImageNet-C
0.1
0.05
0.1
0.05
0.1
0.05
0.1
0.05
Source
78.940.00
BN adapt
76.1
BN adapt
Source
64.340.16
53.3
44.240.45
53.3
31,52004
17.9.u.0
31.420.10
17.9-0.00
49.8
53.3
49.9
53.3
20.0.0.22
17.9-0.00
20.5
17.9
TENT 78.520.16
68.2
6882
47.4
47.1.0.18
21.1
51,010st 21.7.0.0 50
21.020.9
+DELTA
78.920.u (+0:4)
LAME
50.610.3 50.8
10.9
17.2
17.2
60/4.03
59.61..4
21. 21.5
Ent-W
78.610.19
CoTTA
65.120.13 65.1g9.5
34,220.26
34.2.0.16
50.5
50.5.0:0
21.420.11
22.0a6.55
+DELTA
29.0 (+0.5)
CoTTA
67.0
66.9
10.0
34,610.7
p4.3
10.7 Lo52 50.61o.s
21.6.0.56
22.1 10:21
PL
67.2an.2: 67.3
39,4
39.3
30.2
50.6
22.8...as
23.1
+DELTA
67.610.36 67.61a.46
40.91..36
40.720.m
66.610.3
66.3
38.8_
38.5
+0.4
+0.3
+1.5
+1.4
+15.9
+15.7
+16.0
+15.4
TENT
67.2 67.7
42.2=020
42.0aa.21
50.320.41
50.220.00
22.3-0.25
22.5
+DELTA
68.5 50.50
68.6440.00
44.420:6
44.2.0.52
67.720.41
67.5g..70
42.1 28
41.946.41
+0.8
+0,9
+2.2
+2.2
+17.4
+17.3
+19.8
+19.4
1878
Fnt.W W
38.35
40.8=0.00
39.5
51.120.4
51.0go.5
11.320.5t
10.8
+DELTA
69.1
69.2
48.420.1
47.720.1
68.0100.3
67.82000
45.4.033
44.8
+1.0
+7.6
+16.9
+34.1
+34.0
Figure 4: Across architecture
which is detailed in Appendix 1). We test models with π E {0.1, 0.05) (similarly, we show the extreme experiments with π 0.001 in Appendix A.4). Table 6 summarizes the results in IS+CI and DS+CI scenarios, with the following observations: (i) Under class-imbalanced scenario, the performance degradation is not as severe as under dependent data. This is primarily because the imbalanced test data has relatively little effect on the normalization statistics. DELTA works well on the imbalanced test stream. (ii) The hybrid DS+CI scenario can be more difficult than the individual scenarios. DELTA can also boost baselines in the hybrid scenario. (iii) Though the low-entropy- emphasized method Ent-W improves TENT in IS+CB scenario (Table 4), it can be inferior to TENT in dependent or class-imbalanced scenarios (the results on ImageNet-C in Table 5,6). The reason is that Ent-W leads to a side effect- - amplifying the class bias, which would neutralize or even overwhelm its benefits. DELTA eliminates Ent-W's side effects while retaining its benefits, Ent- W+DELTA always significantly outperforms TENT+DELTA.
Evaluation on realistic out-o--distribution datasets ImageNet-R and YTBB-sub. ImageNet-R is inherently ilass-imbalanced and consists of mixed variants such as cartoon, art, painting, sketch, toy, etc. As shown in Table 8, DELTA also leads to consistent im- provement on it. While compared to ImageNet-C,, ImageNet-R is collected individually, which consists of more hard cases that are still difficult to recognize for DELTA, the gain is not as great as on ImageNet-C. C. For YTBB-sub, dependent and class-imbalanced sam- ples are encountered naturally. We see that classical methods suffer from severe degradation, whereas DELTA assists them in achieving good performance.
Table 8: Mean acc on ImageNet-R and YTBB-sub.
Method ImageNett R YTBB sub
Source
38.420.0
74.010.0
BN adapt
41.910.1
$1.4102
48.3
51.520.0
TENT
44.7
$1.71027
+DELTA
45.320.05
75.720.1
+0.5
+24.0
Ent W
48.3
$1.52003
+DELTA
49.6
76.2
+1.3
+24.7
Evaluation on in-distribution test data. A qualified FTTA method should be "safe" on in- distribution datasets, i.e., (z, y) = pirain (I,y) y). According to Table 7, (i) DELTA continues to improve performance, albeit slightly: (ïi) most adaptation methods can produce comparable results to Source, and the combination with DELTA even outperforms Source on in-distribution data.
Evaluation with different architectures. Figure 4 indicates that DELTA can help improve previous test-time adaptation methods with different model architectures. More analyses (e.g., evaluations with small batch size, different severity levels) are provided in Appendix A.4.
Contribution of each component of DELTA. DELTA consists of two tools: TBR and DOT. In Table 9, we analyze their contributions on the basis of TENT with four scenarios and two datasets. Row #1 indicates the results of TENT. Applying either TBR or DOT alone on TENT brings gain in most scenarios and datasets. While, we find that TBR achieves less improvement when the test stream is IS+CB and the batch size is large (e.g., performing adaptation with TBR alone on the IS+CB data of CIFAR100-C with batch size of 200 does not improve TENT). However, when the batch size is relatively small (e.g., ImageNet-C, batch size of 64), the benefits of TBR will be- come apparent. More importantly, TBR is extremely effective and necessary for dependent samples.
8

---

Published as a conference paper at ICLR 2023
DOT can consistently promote TENT or TENT+TBR in all scenarios, espe- cially when the class number is large. These results demonstrate that both the inaccurate normalization statis- ties and the biased optimization are detrimental, TBR and DOT can effec- tively alleviate them.
Table 9: Ablation on the effectiveness of each component (on top of TENT) measured in various scenarios: IS+CB, DS+CB (p=0.5), IS+CI (T=0.1), DS+CI (p=0.5, ==0.05).
CIPAR100
ImageNes-C
DOT
IS
DS
IS:CI
DSI
IS:CB
DS.CB
IS:CI
DSICI
68.72m 49, 49.7am 22.2 50.24100 42.7 42.7a00 22.1 42,0440 22.5-w2s
68.9 20m 67.420 67.920 on 66.6.0 to 43.400
20.9.0 42.8 39.6_000
69.1 joor 50.6 68.. 51.0a000 46.320m 23 43.9.0.s ss 24,8 24.8_
V < 69.5 68.5 30:5 10.5 45.1 43.1 10 41.9.
Comparing DOT with other tech- niques for class imbalance. On the basis of Ent- W+TBR, Table 10 compares DOT against the follow- ing strategies for solving class imbal- ance. Diversity-based weight (Div- W) (Niu et al., 2022) computes the cosine similarity between the arrived test samples' prediction and a moving average one like z, then only employs the samples with low similarity to up- date model. Although the method is proposed to reduce redundancy, we find it can resist class imbalance too. The method relies on a pre- defined similarity threshold determine whether to use a sample. We report the results of Div-W with varying thresholds (shown in parentheses). We observe that the threshold is very sensitive and the optimal value varies greatly across datasets. Logit adjustment (LA) (Menon et al., 2021) shows strong performance when training on imbalanced data. Following Wang et al. (2022b), we can per- form LA with the estimated frequenyyyc vector z in test-time adaptation tasks. While we find that LA does not show satisfactory results here. We speculate that this is because the estimated class distribution is not accurate under the one-pass adaptation and small batch size, while LA requires a high-quality class distribution estimate. KL. divergence regularizer ( (KI-div) (Mummadi et al., 2021) augments loss function to encourage the predictions of test samples to be uniform. While, this is not always reasonable for TTA, e.g., for the uniform will hurt the performance conversely. We examine multiple regularization strength options (shown in parentheses) and report the best two. The results show that KL-div is clearly inferior in dependent or class- iluss-imbllaneed scenarios. We further propose another strategy called Sample-drop dropp It records the (pseudo) categories of the test samples that have been employed, then Sample-drop will directly discard a newly arrived test sample (i.e., not use the sample to update the model) if its pseudo category belongs to the majority classes among the counts. This simple strategy is valid but inferior to DOT, it completely drops too many useful samples.
Table 10: Ablation on different techniques for class imbal- ance (on top of Ent-W+TBRY measured in various scenarios (same in Table 9).
CIFAR100.C
ImageNes.C
Method
IS+CB
DS-CB
IS+CI
DS+CI
IS-CB
DS-CB I8+CI
DS+CI
Di-W (0.05)67.55 Sanaz 68 I 66 Banas 67, Lauw 48 Kancc 45 Laula 47.9 as 02n o
Die- (0.1) 69. Sao as 68 68.620 68 Sga so 67.620 48 48.4gnic 43 43.0go si 47.7g0 s0 39 640 so
Da-W (0.2)
69. 69.7u0.30 69.2 68 1866133 67 67.4.ool 46.400 40.33a0 46.5.4.3 37.5a0
Di-W (0.4)
69.7 68.0.0 68440.33 67.2 43.6...54 37.5 44.....7 35.
70.0aa.a 66.9 as 69 22a.7 66.4
2a.2s
6anus 43.1 27.5
KI-div (1e2)
47 2000 46/6 36.52122
KL--div (1e3)
28.9
27.7 23.1 22.5
Sample-drop 70.1 68.7
69.0gs
67.5.0 49.5 20 46.92om 48.2.0.31 42.6.0
DOT
a0.1 68.8 69.1 67.8 49.9 pance 47.4 48.4... 44.300
test data, forcing the outputs to be
Impacts of a in TBR and À in DOT. Similar to most methods, when the smoothing coefficient a (or X) is too small, the adaptation may be unstable; when (or X) is too large, the adapta- tion would be slow. Figure 5 provides the ablation studies of a (left) and X (right) on the DS+CB (p = 0.5) samples of CIFAR 100-C (from the validation set). We find that TBR and DOT perform reasonably well under wide range of and X.
215
Searce TENT
TENT
635
130
100
015
630
225
100
Figure 5: Impacts of a and A.
5 CONCLUSION
In this paper, we expose the defects in test-time adaptation methods which cause suboptimal or even degraded performance, and propose DELTA to mitigate them. First, the normalization statistics used in BN adapt are heavily influenced by the current test mini-batch, which can be one-sided and highly fluctuant. We introduce TBR to improve it using the (approximate) global statistics. Second, the optimization is highly skewed towards dominant classes, making the model more biased. DOT alleviates this problem by re- balbanann the contributions of each class in online manner. The combination of these two powerful tools results in our plug-in method DELTA, which achieves improvement in different scenarios (IS+CB, DS+CB, IS+CI, and DS+CT) at the same time.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGMENTS
This work is supported in part by the National Natural Science Foundation of China under Grant 62171248, the R&D Program of Shenzhen under Grant JCYJ202208 18101012025, the PCNL KEY project (PCL2021 A077) and Shenzhen Science and Technology Innovation Commission (Research Center for Computer Network (Shenzhen) Ministry of Education).
REFERENCES
Shaden Alshammari, Yu-Xiong Wang, Deva Ramanan, and Shu Kong. Long-tailed recognition via weight balancing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6897-6907, 2022.
Arsenii Ashukha, Alexander Lyzhov, Dmitry Molchanov, and Dmitry Vetrov. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. International Conference on Learning Representations, 2020. URL https: wwwpp/ppwee......c/// net forum?
Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter- online test- time adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp.
2022.
Dian Chen, Dequan Wang, Trevor Darrell, and Sayna Ebrahimi. Contrastive test-time adaptation. In CVPR, 2022.
Yin Cui, Menglin Jia, Tsung-Ye Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVÉ conference on computer vision and pattern recognition, pp. 9268 9268-9277, 2019.
Charles Elkan. The foundations of cost-sensitive learning. In International joint conference on artificial intelligence, volume 17, pp. 973-978. Lawrence Erlbaum Associates Ltd, 2001.
Yaroslav Ganin, Evgeniya Ustinova, Hana Ajakan, Pascal Germain, Hugo Larochelle, François Laviolette, Mario Marchand, and Victor Lempitsky. Domain-adversarial training of neural net- works. The journal of machine learning research, 17(1)2096-20300 2016.
Arthur Gretton, Karsten Borgwardt, Malte Rasch, Bernhard Schölkopf, and Alex Smola. A kernel method for the two-sample-problem. In B. Schölkopf, J. Platt, and T. Hoff- man (eds.), Advances in Neural Information Processing Systems, volume 19. MIT Press. 2006. URL https: httpps:/pproceedingg www.eeeedddin.....eee.eeeeee.eee wwtpppppe//////EL..... e 391
.ddf.
Yin-Ye Yin He, Jianxin Wu, and Xiu-Shen Wei. Distilling virtual examples for long-tailed recognition. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 235-244, 2021.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. In International Conference on Learning Representations, 2019, URL
https: opwnreview..e.eeeeeeee/ee net worumr forommidiuJnm6 Ym.
Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshmi- narayanan. AugMix: A simple data processing method to improve robustness and uncertainty. Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Dan Hendrycks, Steven Basart, Norman Mu, Saurav Kadavath, Frank Wang, Evan Dorundo, Rahul Desai, Tyler Zhu, Samyak Parajuli, Mike Guo, et al. The many faces of robustness: A critical analysis of -ut-of-distribution generalization. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 3340-8349, 2021.
Judy Hoffman, Eric Tzeng, Taesung Park, Jun Yan Zhu, Phillip Isola, Kate Saenko, Alexei Efros, and Trevor Darrell. Cycada: Cycle-consistent adversarial domain adaptation. In International conference on machine learning, pp. 1989-1998. Pmlr, 2018.
10

---

Published as a conference paper at ICLR 2023
Sergey loffe. Batch renormalization: Towards reducing minibatch dependence in batch- normalized models. Advances in neural information processing systems, 30, 2017.
Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International conference on machine learning, pp. 448 456.. PMLR. 2015.
Bingyi Kang, Yu Li, Sa Xie, Zehuan Yuan, and Jiashi Feng. Exploring balanced feature spaces for representation learning. In International Conference Learning Representations, 2020a.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yan- nis Kalantidis. Decoupling representation and classifier for long- lang-tailed recognition. In Interna- tional Conference on Learning Representations, 2020b. URL https: http///pppeeeeieeeeeeeeeeeeeeeee neet/
Jogendra Nath Kundu, Naveen Venkat, Rahul M V, and R. Venkatesh Babu. Universal source-free domain adaptation. June 2020.
Dong-Hyun Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on challenges in representation learning, ICML, volume 3, pp. 896, 2013.
Lei Li, Ke Gao, Juan Cao, Ziyao Huang, Yepeng Weng, Xiaoyue Mi, Zhengze Yu, Xiaoya Li, and Boyang Xia. Progressive domain expansion network for single domain generalization. 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 224-233. 2021.
Jian Liang Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In International Conference on Machine Learning, pp. 6028-6039. PMLR, 2020.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2537 2546, 2019.
Mingsheng Long, Han Zhu, Jianmin Wang, and Michael Jordan. Deep transfer learning with joint adaptation networks. In International conference machine learning, pp. 2208-2217. PMLR, 2017.
Mingsheng Long, Zhangjie Cao, Jianmin Wang, and Michael I Jordan. Conditional adversarial domain adaptation. Advances in neural information processing systems, 31, 2018.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. In International Conference on Learning
Representations, 2021. URL https: rppeneevieww net forum? 3
Chaithanya Kumar Mummadi, Robin Hutmacher, Kilian Rambach, Evgeny Levinkov, Thomas Brox, and Jan Hendrik Metzen. Test-time adaptation to distribution shift by confidence maxi- mization and input transformation. arXiv preprint arXiv:2106. 14999, 2021.
Zachary Nado, Shreyas Padhy, D Sculley, Alexander D' A'mour Balaji Lakshminarayanan. and Jasper Snoek. Evaluating prediction-time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 109666 2020. 2020.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 16888-16905. PMLR, 17-23 Jul 2022. URL https: /eattts::/prree proceed ngs .mlr. press v11 62 m222nni2222 html.
Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence.
Dataset shift in machine learning. Mit Press, 2008.

---

Published as a conference paper at ICLR 2023
Esteban Real, Jonathon Shlens, Stefano Mazzocchi, Xin Pan, and Vincent Vanhoucke. Youtube- boundingboxes: A large high-precision human- annotated data set for object detection in video. 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 7464-7473, 2017.
human-annotated
Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International conference on machine learning, pp. 4334 4334-4343. PMLR, 2018.
Kuniaki Saito, Yoshitaka Ushiku, and Tatsuya Harada. Asymmetric tri-training for unsupervised domain adaptation. International Conference on Machine Learning, pp. 2988-2997. PMLR, 2017.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Ad- vances Neural Information Processing Systems, 33:11539-11551, 2020.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Repre- sentations, 2021. URL https: /ppere/veeeeeeeeeeeeemeeeeeeeeee net/ for
Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 7201-7211, 2022a.
Tao Wang. Yu Li, Bingyi Kang, Junnan Li, Junhao Liew, Sheng Tang, Steven Hoi, and Jiashi Feng. The devil is in classification; simple framework for long-tail instance segmentation. In Euro- pean conference on computer vision, pp. 728-744. Springer, 2020.
Xudong Wang. Zhirong Wu, Long Lian, and Stella X Yu. Debiased learning from naturally im- balanced pscudo-labels. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14647-14657, 2022b.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual trans- formations for deep neural networks. In Proceedings of the IEFE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Shiqi Yang, Yaxing Wang, Joost van de Weijer. Luis Herranz, and Shangling Jui. Generalized source-free domain adaptation. In 2021 IEEE/CVF International Conference Computer Vision (ICCV), pp. 9958-8967, 2021a. doi: 10. D0.110//CCVV49922..000 2.8800888....
Shiqi Yang, Yaxing Wang, Joost van de weijer, Luis Herranz, and SHANGLING JUI. Exploit- ing the intrinsic neighborhood structure for source-free domain adaptation. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021b. URL https: //ppnreeeeeeeeeeeeeeeeeeeeeeeeee net/ forum? d=ueGDv 6 HmO.
Mikhail Yurochkin, Mayank Agarwal, Soumya Ghosh, Kristjan Greenewald, Nghia Hoang, and Yasaman Khazaeni. Bayesian nonparametric federated learning of neural networks. In Interna- tional Conference on Machine Learning, pp. 7252-7261. PMLR, 2019.
Werner Zellinger, Thomas Grubinger, Edwin Lughofer, Thomas Natschläger, and Susanne Saminger-Platz. Central moment discrepancy (CMD) for domain-invariant representation learn- ing. In International Conference on Learning Representations, 2017. URL https:// openreview. htwpiiiwwweeeeeeee forum ?aderuumidiikBBB.. -id-std------ee mcel.
M. Zhang, S. Levine, and C, Finn. MEMO: Test time robustness via adaptation and augmentation. 2021.
Bowen Zhao, Xi Xiao, Guojun Gan, Bin Zhang, and Shu-Tao Xia. Maintaining discrimination and fairness class incremental learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 13208-13217, 2020. 2020.
12

---

Published as a conference paper at ICLR 2023
Art
Cartoon
Deviantart
Graffiti
Graphic
Misc
Origami
Painting
Sculpture
Sketch
Sticker
Tattoo
Toy
Videogame
Figure 6: Different renditions of class n01694178 (African chameleon) from ImageNet-R
A APPENDIX
A.1 DATASETS
Examples of ImageNet-R and ImageNet-C are shown in Figure and Figure 7 respectively. ImageNet-R Hendrycks al. (2021) holds variety renditions (sketches, graphics. paint- ings, plastic objects, cartoons, graffiti, origami, patterns, deviantart, plush objects, sculptures, art, tattoos, toys, embroidery, video game) of 200 ImageNet classes, resulting in 30,000 images. CIFAR100-C and ImageNet-C are established in Hendrycks & Dietterich (2019). CIFAR 100-C contains 10,000 images with corruption types: Gaussian Noise (abbr, Gauss), Shot Noise (Shot), Impulse Noise (Impul), Defocus Blur (Defoc), Frosted Glass Blur (Glass), Motion Blur (Motion), Zoom Blur (Zoom). Snow, Frost, Fog, Brightness (Brit), Contrast (Contr), Elastic, Pixelate (Pixel), JPEG. There 50,000 images for each corruption type ImageNet-C, others the same as CIFAR100-C.
For the real-word applications with dependent and dlass-mbalaanee test samples, consider automatic video content moderation task (e.g., for the short-video platform), which needs to recog- nize the categories of interest from the extracted frames. It is exactly natural DS+CI scenario. We collect 1686 test videos from YouTube, which are annotated in mouTube-Boundinghae 49006 video segments named YTBB-sub here. We consider 21 categories. For the trained model, we adopt model (ResNet18) trained on the related images from COCO dataset. Thus. there is natural difference between the training domain and test domain. The consecutive video segments form the natural dependent samples (an object usually persists over several frames) as shown Figure 8. Moreover, the test class distribution is also skewed naturally as shown in Figure 8.
dataset. extracted from these videos and form the stream in this experiment,
To simulate dependent test samples, for each class, we sample qj ~ Dirs(p), @a E R and allocate qu.j proportion of the th class samples piece j, then the J pieces are concatenated to form test stream in our experiments (J is set to for all experiments); > a concentration factor, when p is small, samples belong to the same category will concentrate in test stream.
To simulate class-imbllanced test samples, we re-sample data points with an exponential decay in frequencies across different classes. We control the degree of imbalance through an imbalance factor n, which defined as the ratio between sample sizes of the least frequent class and the most frequent class.
For DS+CI scenario, we mimic a class class-imbalanee imbalanced test set first, then the final test samples are dependently sampled from it.
13

---

Published as a conference paper at ICLR 2023
Worte Kachler
Wor Miller
W Khehler
Chekler
Brightness
JPEG
Pixelate
Elastic
Contrast
Fog
Frost
Snow
Zoom Blur
Motion Blur
Frosted Glass Blur
Defocus Blur
Shot Noise
Impulse Noise
Gaussian Noise
Figure 7: Different corruption types of class nO nO166 1944778 (African chameleon) from ImageNet-C C.
category: best
potted plant truck
train
boat
bus
36GMLT9BB
OPAN 1299N1
S wowooUnoe 1000
BUTTIXSCVII
1e9pZO-VEU
ONeg9vTOSto
172. MVLLeU
3ZyPlewx. n8
2yDeK ADDK7WyDUM
OvK 6B21KEA
video ID:
(a) The natural dependent samples in YTBB-sub. Each bar represents a sample, each color represents
category. The videos can be found hwtp://wwwwww...././ttttttcctcce above video ID)".
# Samples
3000
2000
1000
10
Class
(b) The test class distribution.
Figure 8: Characters of YTBB-sub dataset.
A.2 THE ALGORIITM DESCRIPTION OF TBR
We present the detailed algorithm description of TBR in Algorithm 2.
14

---

Published as a conference paper at ICLR 2023
Algorithm 2: Test-time Batch Renormalization (TBR) module
Input: mini bati- test features [xxxXXSx'' with batch size B. C channels, height S and width S"; learnable affine parameters E IR 3 E R current test-time moving mean n I and standard deviation demm E RR, smoothing coefficient a.
f
butch
[c]
Ei.,,' uib, G, 8, c= 1,2, ,C// get mean (for each channel)
BSB
I buich
BSB Cula G, 8, - fa [c))² +e, c= 1,2, ,C get standard deviation (for each channel)
r
geme
11 got
solA m -pmm
// get d
gess
ヒ- E-A A
0²
gharch
r d // normalize
e* ".0" + // scale and shift
f <- a fm
+ (1 a). sg(abuic) // update A
Jama t a- jmm
a) sg(bbulc)) // update g
Output: V², ma gm
A.3 IMPLEMENTATIONS
We use Adam optimizer with learning rate of 1e-3, batch size of 200 for CIFAR100-C; SGD opti- mizer with learning rate of 2.5e-4, batch size of 64 for ImageNet-C/-R: SGD optimizer with learn- ing rate of 2.5c-4, batch size of 200 for YTBB-sub. For DELTA, the are roughly selected from {0.9, 0.95, 0.99, 0.999} on validation sets, e.g., the extra sets with cor- ruption types outside the 15 types used in the benchmark. The smoothing coefficient a in TBR is set to 0.95 for CIFAR100-C and ImageNet-C/-E 0.999 for YTBB- À in DOT is set to 0.95 for ImageNet-C/-R and 0.9 for CIFARIOO-C/ YTBB-sub.
a and
Then, summarize the implementation details of the compared methods here, including BN adapt, PI. TENT, LAME, ETA, Ent-W, and COTTA (COTTA*). Unless otherwise specified, the optimizer, learning rate, and batch size are the same as those described in the main paper. For BN adapt, we fol- low the operation in Nado et al. (2020) and the official code of TENT (https: htt//gitttt hub. com/ DequanWang / ttntt.. i.e., using the test-time normalization statistics completely. Though one can introduce lyper-parameter to adjust the trade-off between current statistics and those inherited from the trained model (ap) (Schneider al., 2020), we find this strategy does not lead to significant improvement and its effect varies from dataset to dataset. For PL and TENT, besides the normaliza- tion statistics, we update the affine parameters in BN modules. The confidence threshold PL. is set to 0.4, which can produce acceptable results in most cases. We adopt/modify the official implemen- tation https: //gittubtbb// com/ http//Deeaaaaggg/egeeeeee /tent to produce the results of TENT/PL. For LAME, we the k-NN affinity matrix with 5 nearest neighbors following Boudiaf al. (2022) and the official implementation tps : ///ggg /t//ttttuut com/ f veai /LAME. For ETA, the entropy constant threshold is set to 0.4 x In K (K is the number of task classes), and the similarity threshold is set to 0.4/0.05 for CIFAR/ImageNet experiments following the authors' suggestion and official implementation https: / / hub. com noom/rr-aa eggplant aaaatantt For Ent-W, the entropy con- stant threshold is set to 0.4 or 0.5 times In K. For CoTTA, the used random augmentations include color jitter, random affine. gaussian blur, random horizontal flip, and gaussian noise. 32 augmen- tations are employed in this method. The learning rate is set to 0.01 for ImageNet experiments following official implementation https: //gittubbbb...g. thub. com/ qi menerrgggy// cotta. The restora- tion probability is set to 0.01 for CIFAR experiments and 0.001 for ImageNet experiments. The augmentation threshold is set to 0.72 for CIFAR experiments and 0.1 for ImageNet experiments. factor is set to 0.999 for all experiments. CoTTA optimizes all learnalle parameters during adaptation.
The seponential-moving-eae
A.4 ADDITIONAL ANALYSIS
Fully test-time adaptation with small (test) batch size. In the main paper, we report results with the default batch size following previous studies. Here, we study test-time adaptation with much smaller batch size. The small batch size brings two serious challenges: the normalization statistics can be inaccurate and fluctuate dramatically; the gradient-based optimization can be noisy. Previ-
15

---

Published as a conference paper at ICLR 2023
study (Niu et al., 2022) employs sliding window with L samples in total (including B previous samples, assuming L > B, L%B 0 here) to perform adaptation. However, this strat- egy significantly increases the computational cost: 1/2 x forward and backward, e.g., 64x when B =, = 64. We employ another strategy, called "fast-inference and slow-update". When the samples arrive, infer them instantly with the current model but do not perform adaptation; the model is updated with the recent L samples every & mini-batches. Thus, this strategy only needs 2x for- ward and 1x backward. Note that the two strategies both need to cache some recent test samples, which may be bit against the "online adaptation". We evaluate TENT and DELTA on the IS+CB test stream of CIFAR 100-C with batch sizes 128, 16, 8, and The results are listed in Table 11. We find that TENT suffers from severe performance degeneration when the batch size is small, which due to TENT always using the normalization statistics derived from the test mini-batches, thus it is still affected by the small batch size during "fast-inference.. With the assistance of DELTA, the performance degradation can significantly alleviated: it only drops by 0.7% (from 69.8% to 69. 69.1%) when B
Table II: Results (classification accuracy, %) with different batch sizes on IS+CB test stream of CIFARI00-C.
Method
128
16
8
Source
53.5
53.5
53.5
53.5
TENT
68.7
64.9
59.9
1.6
TENT+DELTA
69.8
69.4
69.0
69.1
The initialization of TBR's normalization statistics. As described in Section 3.2, TBR keeps the moving normalization statistics jisme, Gema we usually have two ways to initialize them: using the statistics Abbach batch derived from the first test mini-batch (First); using the statisties s the "First" initialization worth noting that "First" is not reasonable for too small batch size. We perform TENT+DELTA with the above two initialization strategies and different batch sizes on the IS+CB test stream of CIFAR 100-C. Figure 9 summaries the results, we can see that when the batch size is too small, using the inherited normalization statistics initialization is better; when the batch size is acceptable (just > 8 for CIFAR 100-C), using the "First" initialization strategy is superior.
oma inherited from the trained model (Inherit). In the main paper, strategy. However,
72.5
Accuracy (%)
70.0
67.5
65.0
Inherit
62.5
First
60.0
16
8
Batch size
Figure 9: Comparison of two TBR initialization strategies on top of TENT+DELTA in IS+CB sce- nario on CIFAR100-C.
Performance under different severity levels on CIFAR100-C and ImageNet-C. In the main pa- per, for CIFAR 100-C and ImageNet-C, we report the results with the highest severity level 5 follow- ing previous studies. Here, we investigate DELTA on top of TENT with different severity levels on CIFAR100-C (IS+CB scenario). Figure 10 presents the results. We observe that (i) the corruption level increases, the model accuracy decreases; (ii) DELTA works well under all severity levels.
Performance in extreme cases. We examine the performance of DELTA with more extreme ditions: DS+CB with = 0.01, IS+CI with π = 0.001. Table 12 shows DELTA can manage the intractable cases.
Influence of random seeds. As fully test-time adaptation is established based on a pre-trained model, i.e., does not need random initialization; methods like PL, TENT, Ent-W, and our DELTA
16

---

Published as a conference paper at ICLR 2023
76
Accuracy (%)
74
72
TENT
70
DEETA
5
Severity Level
Figure 10: Comparison under different severity levels on CIFARI00
Table 12: Performance in extreme cases.
DS+CB (p 0.01)
IS+CI (π 0.001)
Source
18.0
17.9
BN adapt
6.8
31.1
ETA
3.3
44.1
LAME
26.0
17.4
COTTA
7.0
33,5
CoTTA
7.2
33.6
PL
6.6
37.9
+DELTA
34.2
38.9
TENT
6.0
39,8
+DELTA
36.7
41.8
Ent-W
1.4
39.9
+DELTA
36.5
45.1
also do not bring random initialization. As result, the adaptation results are always the same one fixed test stream. However, the random seeds can affect sample order in our experiments. We study the influence of random seeds Gauss and Shot data (IS+CB scenario) of ImageNet-CC with seeds {2020, 2021, 2022, 2023]. The results TENT and DELTA are summarized in Table 13, the methods
from which one scenario. For fair comparison, all methods are investigated under the same sample order for each specific scenario in experiments.
greatly affected by the sample order within the same
Table 13: Influence of random seeds. Classification accuracies (%) are reported on two kinds of corrupted data (IS+CB) of ImageNet-C under four random seeds (2020, 2021, 2022, and 2023).
TENT
TENT+DELTA
Data
2020
2021
2022
2023
2020
2021
2022
2023
Gauss
28.672
28.434
28.774
28.796
31.186
30.916
31.270
31.208
Shot
30.536
30.496
30.370
30.458
33.146
33.140
33.124
32.994
Ablation DOT. We examine the performance DOT with another way get the sample weights (Line 5,6 in Algorithm 1). One can discard line 5 and modify line 6 to adopt the original soft probabilities: Wm,+ = >E k= 1/(2-11(A) [A]] + c) Pm We compare the hard label strategy (Algorithm 1) with soft one in Table 14 (on the basis of Enw-W+BBR, on ImageNet-C). We find that both strategies work well in all scenarios, demonstrating the effectiveness of the idea of DOT. The performance the soft strategy is slightly worse than the hard strategy in some scenarios. However, think it is difficult to say "hard labels are necessarily better than soft labels" or "soft labels are necessarily better than hard labels", for example, the two strategies both exist in recent semi-supervised methods: hard label in FixMatch, soft label in UDA.
17

---

Published as a conference paper at ICLR 2023
Table 14: Ablation on DOT.
IS+CB
DS+CB P 1.0
DS+CB
DS+CB
IS+CI
IS+CI
DS+CI
DS+CI
P 0.5
p 0.1
0.1
π 0.05
0.5,
0.1 P 0.5, πT 0.05
Hard
49.9
48.3
47.4
43.2
48.4
47.7
44.8 45.4
Soft
49.7
48.0
47.3
43.0
48.3
47.5
45.1
44.5
A.5
RESULTS EACH CORRUPTION TYPE ON CIFAR100-C.
Table 2 has compared the usages of different normalization statistics, we further provide the detailed results of all corruption types in Table 15.
Table 16 presents the results of all corruption types under different batch sizes and the two initial- ization strategies for normalization statistics TBR, the averaged results have been illustrated in Table 11 and Figure 9 respectively.
Table 17 summarises the detailed performance on IS+CB test stream with different severity levels.
Table 18 compares the test-time adaptation methods in IS+CB scenario; Table 19 for DS+CB test stream (p= 1.0), Table 20 for DS+CB test stream (p = 0.5), Table 21 for DS+CB test stream (p = 0.1); Table 22, 23 for IS+CI data with 0.1, 0.05; Table 24 / Table 25 for DS+CI test data with = 0.5 and π = 0.1 / π 0.05.
A.6 RESULTS OF EACH CORRUPTION TYPE ON IMAGENET-C.
Table 26 compares the test-time adaptation methods IS+CB scenario and Table further com- pares them with different model architectures; Table 28, Table 29, and Table 30 for DS+CB test streams with / = 1.0, p = 0.5 and p= 0.1, respectively; Table 31, for IS+CI data with क = 0.1, m 0.05; Table 33/ / Table 34 for DS+CI test data with 0.5 and π = 0.1 / 0.05. The results in Table 15-Table 34 are obtained with seed 2020.
Table 15: Comparison of the normalization statistics IS+CB and DS+CB test streams of in terms of classification accuracy (%).
CIFAR100-C with
Method
Gauss Shot Impul Defoc Glass Motion Zoom Snow Frost Fog Brit Contr Elastic Pixel JPEG Avg
Source
27.0 32.0 60.6
70.7
45.9
69.2
71.2
60.5
54.2
49.7 70.5 44.9
62.8
25.3
$3.5
IS+CB scenario
adapt
$7.6 59.0
56.9
72.3
58.0
70.3
71.8
64.8
64.8
58.1
69.7
64.0
66.7
64.4
TdaA++TEMA
58.0
59.7
57.1
72.5
58.6
70.3
72.5
65.3
65.5
58.3 74.1
64.4
67.0
59.2
64.9
TENT
62.4
64.7
67.3
74.3
62.5
72.4
74.2
69.4
67.6
66.8
75.6
71.8
66.9
71.3
62.6
68.7
TENT+TEMA
19.4
14.9
16.4
31.9
14.8
25.0
28.9
24.3
25.0
19.331.1
24.3
25.5
26.0
18.2
23.0
TENT+TBR
62.1
64.7
67.7
74.6
62.0
72.6
74.0
69.7
67.9
67.8
76.2
71.6
67.1
71.8
63.3
68.9
DS+CB scenario
BN adapt
24.1
24.7
23.4
30.2
23.2
29.9
29.8
26.6
27.2
24.2
30.0
28.6
25.7
27.8
23.8
26.6
BN adapt+TEMA
56.0
57.8
55.2
70.7
56.7
68.6
70.2
63.2
63.6
56.6
71.7 67.8
62.2
64.8
57.2
62.8
TENT
21.2
22.7
21.9
26.6
20.0
25.5
26.6
23.0
22.2 21.7
26.3
21.6
21.7
24.7
20.3
23.1
TENT-TEMA
18.0
17.3
15.2
34.2
18.6
26.3
36.6
18.9
27.2
24.6
36.2
25.8
26.5
28.6
20.4
25.0
TENT+TBR
55.8
60.0
58.8
70.7
57.2
67.4
69.7
64.4
62.8
61.2
64.0
60.9
67.1
56.4
63.1
18

---

Published as a conference paper at ICLR 2023
Table 16: Comparison of different batch sizes and the initialization strategies for TBR's normaliza- tion statistics on IS+CB test stream of CIFARI00-CA in terms of classification accuracy (%).
Method
Init
Gauss
Shot Impul
Defoc Glass Motion Zoom Snow Frost Fog
Brit Contr Elastic Pixel JPEG Avg
Source
27.0
32.0
60.6
70.7
45.9
69.2
71.2
60.5
54.2
49.7
70.5 5 44.9
62.8
25.3
58.8
53.5
TENT, B=128
62.4
64.7
67.3
74.3
62.5
72.4
74.2
69.4
67.6
66.8
75.6
71.8
66.9
71.3
62.6
68.7
TENT, Ba16
58.7
63.8
70.8
58.7
68.8
70.3
65.8
64.1
63.3 66.9
62.7
67.6
64.9
TENT, R=8
54.0
56.1
58.6
65.9
53.0
64.1
65.4
61.0
58.6
57.8
67.1
62.9
58.1
62.8
53.8
59.9
TENT, B=1
1.5
1.5
1.6
1.6
1.6
1.8
1.7
1.8
166
1.5
1.6
1.6
1.5
1.8
1.6
1.6
TENT+DELTA, B=128
Inherit
62.4
63.9
69.0
75.3
63.2
73.2
74.8
69.8
69.2
66.6
76.0
71.3
67.4
69.7
64.3
69.1
TENT+DELTA, B=128
First
64.0
69.1
75.3
63.3
73.0
74,6
70.3
69.4
68.1 76.7
72.9
67.6
72.3
69.8
TENT+DELTA, B=16
Inherit 62.3 64.0 69.1
75.2
63.1
73.3
74.8
69.6
69.3 66.7 76.0
70.8
67.3
69.7
64.3 69.0
TENT.DELTA, B=16
First
63.5 65.5 68.2
74,8
63.2
72.7
74.6
70.2
69.3 67.7 76.2
72.4
67.5
71.9
63.9 69.4
TENT=DELTA, B=8
Inherit
62.4 64.0 69.0
75.2
63.1
73.3
74.8
69.7
69.4 65.9
71.2
67.3
69.6
64.2 69.0
TENT#DELTA, B=8
First 63.1 65.1 67.1
74.8
62,4
72.6
74.3
69.9
69.2
65.2
71.2
67.0
71.6
63.0 68.9
TENT+DELTA, B=1
Inherit
62.2 64.0 68.9
75.3
63.1
73.2
74.7
69.7
69.4
66.6
76,1
71.6
67.4
69.6
64.4 69.1
TENT#DELTA, B=1
First 60.0 62.0 64.4
71.4
59.5
69.0
71.4
65.6
65.7
62.9
64.0
63.6
68.6
59.8
65.4
Table 17: Classification accuracy (%) on IS+CB test stream of CIFARI00-C with different severity levels (B- 128).
Method
Level Gauss Shot Impul Defoc Glass Motion Zoom Snow Frost Fog Brit Contr Elastic Pixel JPEG Avg
Source
64.2
70.9
77.6
78.9
54.4
77.0
76.8
76.5
74.2 78.4 78.7 78.2
74.4
76.5
70.5
73.8
49.3
63.6
75.3
78.1
56.6
75.1
76.6
69.9
69.7 76.1 77.7
75.2
75.0
72.3
65.8
70.4
36.5
47.2
73.1
76.8
60.6
72.3
75.4
69.6
62.1 76.6
71.9
73.7
69.1
64.1 66.8
31.2
40.6
68.0
75.2
39.5
72.4
74.0
65.2
61.1 65.8 74.9
65.7
68.9
52.3
62.5
61.2
27.0
32.0
60.6
70.7
45.9
69.2
71.2
60.5
54.2 49.7 70.5
44.9
62.8
25.3
58.8
53.5
TENT
73.8 75.8
77.2
77.9
69.4
76.6
77.1
76.3
75.8 78.0
77.3
73.3
76.3
71.3 75.6
70.2
73.7
76.0
77.9
69.6
75.6
76.8
73.4
74.0 76.277.8
75.2
74.8
76.0
68.5
74.4
66.7 70.0
74.5
77.4
68.7
73.7
76.4
72.1
71.4 74.9
74.4
73.9
75.6
66.3 72.9
64.4 68.3
70.9
76.3
62.6
74.2
75.3
70.5
70.4 72.276.9
74.1
70.5
74.3
65.2
71.1
62.4 64.7
67.3
74.3
62.5
72.4
74.2
69.4
67.6
65.6
71.8
66.9
71.3
62.6
68.7
TENT+DELTA
74.4 76.1
78.0
78.7
70.3
77.2
77.7
77.1
76.6
78.6 78.3
74.5
77.1
72.0 76.3
70.9 74.7 76.4
78.4
70.3
75.8
77.4
74.5
74.8
76.9 78.4 76.9
75.4
77.0
69.8
75.2
67.8 70.2
75.3
77.9
69.8
74.5
76.8
73.1
72.5
75.6
76.6
74.8
76.5
67.6
73.8
65.6 69.2
72.5
76.9
63.4
74.9
76.0
71.0
71.3
73.277.9
75.8
71.3
75.1
66.4
72.0
64.0 66.0
69.1
75.3
63.3
73.0
74.6
70.3
69.4
68.1
76.7
72.9
67.6
64.6
69.8
Table 18: Classification accuracy (%) on IS+CB test stream of CIFAR100-C.
Method
Gauss Shot Impul Defoc Glass Motion Zoom Snow Frost Fog Brit Contr Elastic Pixel JPEG Avg
Source
27.0 32.0
60.6
70.7
45.9
69.2
71.2
60.5
54.2 49.7 70.5 44,9
62.8
25.3
58.8
53.5
BN adapt
57.9 59.3 57.3
72.4
58.2
70.3
72.1
65.1
65.0
73.5 69.7
64.3
67.1
58.8
64.6
ETA
63.2 65.3 66.9
75.1
63.2
73.1
74.9
70.0
69.7
66.5 76.5
73.6
67.7
72.0
64.0 69.5
LAME
24.1 29.0 59.2
69.0
42.8
67.0
68.9
58.3
50.7
67.5 39.2
60.3
21.4
56.7 50.7
CoTTA
60.0 61.8 60.1
72.6
60.2
70.5
72.3
64.8
65.5 56.7 69.9
64.3
68.4
62.6 65.5
CoTTA
60.0 62.2 60.8
73.2
62.3
71.9
73.7
67.0
75.8 72,9
67.5
67.5
PL
61.8 64.5 65.0
74.6
62.0
72.1
74.2
68.9
68.4 64.8
72.0
66.8
70.8
61.9 68.2
PL+DELTA
62.8 64.8 66.3
74.3
62.7
72.7
74.6
69.4
68.5 65.7
72.8
66.8
71.3
62.7 68.7
TENT
62.8 65.4 66.3
74.8
62.3
72.8
74.6
69.6
68.6
66.8 76.1
72.3
67.3
71.6
63.5 69.0
TENT+TBR
62.5 64.9 67.0
74.8
62.1
72.9
74.3
69.8
68.3 76.6
72.0
67.1
71.9
63.0 68.9
TENT+DOT
63.6 65.7 66.9
75.1
63.0
73.1
74.8
69.8
69.0 67.1 76.2
73.2
67.6
71.8
63.8 69.4
TENT+DELTA
63.5 65.7 67.8
75.1
63.3
73.1
74.7
70.3
69.3 67.4
72.8
67.8
72.3
63.6 69.6
Ent-W
63.5 65.5 67.2
75.1
63.2
73.1
74.8
70.1
69,8
76.1
76.6
73.5
67.7
72.0
64.1 69.6
Ent- Bm:-W-TBR-Di-W.0..5
60.3 63.5 63.8
73.5
60.8
71.8
73.7
68.6
66.2
63.8 74.9
71.8
66.7
69.9
61.7 67.4
63.5 65.3 67.0
75.2
62.7
72.8
74.7
70.0
69.4
66.7
73.2
67.1
71.7
63.8
69.3
Ent- --TWR--DDDDD-00.
63.8 65.6 68.1
75.3
63.1
73.4
75.0
70.7
70.0 67.4770
73.5
67.3
72.5
64.1
69.8
Ent- W+TBR-DiV-WO.4)
63.6 65.4 68.2
75.3
63.1
73.3
70.8
66.3
73.6
72.6
64.0
69.7
Ent W+TBR+LA
64.0 65.9 68.4
75.4
63.5
73.6
75.1
71.0
70.2
67.6
73.8
67.6
72.8
64.5
70.0
Ent W+TBW++SSSae
-Samle--amp
64.1 66.2 68.6
75.8
63.8
73.5
75.5
70.9
70.2
67.7
77.0
73.9
68.2
72.8
64.4
70.2
WDLTAA
64.2 66.1 68.5
75.6
63.6
73.5
75.2
71.2
70.3
68.0
74.0
68.0
72.8
64.7
70.2
19

---

Published as a conference paper at ICLR 2023
Table 19: Classification accuracy (%) DS+CB (p 1.0) test stream of CIFARI00-C.
Method
Gauss Shot Impul Defoc Glass Motion Zoom Snow Frost Fog Brit Contr Elastic Pixel JPEG Avg
Source
27.0
32.0
60.6
70.7
45.9
69.2
71.2
60.5
54.2 49.7 70.5 44.9
62.8
25.3
58.8
53.5
BN adapt
47.2
47.8
46.2
59.5
47.2
57.4
58.8
52.2
53.2 46.7 59.9 S7.4
51.9
54.4
46,9
52.4
ETA
50.1
51.2
52.6
60.3
49.4
58.7
60.2
55.2
54.7 51.2 61.0 58.0
53.2
56.8
50.1 54.9
LAME
28.0 34.2
68.6
52.1
80.7
70.5
61.1 57.4 79.3 49.2
73.3
COTTA
49.1 51.2
49.7
57.7
49.3
58.6
52.8
53.6
46.6 60.0
53.6
52.6
57.3
50.9
53.3
CoTTA*
49.1 51.3
49.5
57.4
49.8
56.6
58.4
53.1
54.1
59.1
59.1
54.2
53.3
57.1
52.8
53.5
PL
49.9 50.5
51.5
60.0
48.3
58.2
60.4
54.2
54.6
50.4 60.7 60.7
S7.4
53.0
56.5
49.2
54.3
PL+DELTA
61.3 62.9
64.4
73.9
61.8
71.7
74.0
68.1
68.0 63.974.9 71.2
66.2
70.1
62.2
67.6
TENT
49.3 50.7
52.6
59.9
48.7
57.8
59.5
53.8
53.5
60.7 56,8
52.7
56.1
49.4
54.1
TENT+DELTA
62.3 64.4
66.7
74.5
62.6
72.0
74.3
68.9
68.5 65.8
72.0
66.8
71.4
63.4
68.6
Ent-W
50.0
51.3
52.9
60.3
49.3
58.9
60.3
54.9
54.9
51.1
57.8
53.1
56.7
50.0
54.8
Ent-W+DELTA
62.7
64.9
67.4
74.6
62.7
72.6
74.4
69.6
69.2
66.1
75.7
72.4
66.8
71.7
64.2
69.0
Table 20: Classification accuracy (%) on DS+CB (p = 0.5) test stream of CIFARI00-C.
Method
Gauss Shot Impul Defoc Glass Motion Zoom Snow Frust Fog Brit Contr Elastic Pixel JPBG Avg
Source
27.0
32.0
60.6
70.7
45.9
69.2
71.2
60,5
54.2 49.7 70.5
44.9
62.8
25.3
58.8
53.5
BN adapt
43.8
45.2
43.9
56.2
44.5
54.7
55.5
49.1
50.0
47,9
54.2
48.7
51.2
45.0
49.5
ETA
45.8
47.5
48.9
56.4
45.3
54.5
55.8
51.2
51.2 48.1
53.8
4944
53.1
45.9
50.9
LAME
28.5
34.8
53.5
79.6
81.8
71.9
62.7 58.681.1
50.6
74.5
26.9
69.5
61.6
CoTTA
46.9
48.3
46.5
55.1
46.6
54.2
55.2
49.3
50.6 43.4
50.8
49.3
54.2
48.3
50.4
CoTTA*
46,9
48.4
46.5
54.5
47.2
53.8
54.4
50.0
51.2 56,1
51.5
50.3
53.9
49.4
50.5
PL.
45.4
47.0
47.8
56.0
45.7
54.3
55.7
50.8
51.3 47.
$2.6
49.4
52.7
45.9
50.6
PL+DELTA
61.3 62.5
63.2
73.1
61.3
70.8
73.6
68.0
67.0
74.5
70.0
65.7
69.7
61.2 67.0
TENT
44.8 46.7
48.4
45.5
54.0
55.2
50.0
50.1
47.3 56.6
$2.2
48.4
52.6
50.2
TENT+TBR
59.7 62.4 64.6
73.3
60.7
70.7
67.3
66.6 64.2
68.9
65.0
69.5
61.0 66.7
TENT+DOT
45.9 47.5 49.3
56.8
46.4
54.8
55.8
50.8
51.1 48.2
53.6
49.6
53.0
46.4
51.1
TENT+DELTA
61.3 63.5 65.5
73.9
62.2
71.5
73.8
68.3
67.5 65.6
70.8
66.1
70.4
62.0
67.8
Ent-W
45.8 47.5 49.0
56.3
45.5
54.5
55.6
51.6
51.1 47.3 53.8
49.3
53.0
45.9
51.0
Ent- BReWWTTRRDDi-....O
61.5 64.0 64.1
73.8
60.7
71.7
73.5
67.6
68.2 64.2
71.0
66.4
70.3
62.2
67.6
Ent WO..
62.4 63.9 65.7
71.8
65.0
66.2
70.5
62.4
68.1
Eat- Www.R--R-DD-W.....
61.0 63.5 65.5
73.6
60.8
71.2
72.9
68.0
67.9 65.1 74.5
70.7
65.7
70.2
62.2
67.5
Ent W+TBR=Div-WOO
60.5 63.4 65.2
73.4
60.3
71.2
72.9
67.6
67.9
65.0 74.4
70.6
65.3
69.9
61.8
67.3
Ent-W+TBR+LA
60.0 62.8 64.2
72.3
59.5
69.9
71.7
66.8
66.8
63. 73.3
69.4
64.7
69.1
61.0
66.4
Ent- WWWTBR-Sapleeeee
61.9 64.2 65.6
61.8
71.7
73.8
68.3
68.4
74.9
71.4
66.2
70.7
62.7
68.1
Ent- W+DELTA
61.9 64.2 66.0
61.9
71.9
65.9
74.9
71.5
70.9
62.9
68.2
Table 21: Classification accuracy (%) on DS+CB (p- = 0.1) test stream of CIFAR100-C.
Method
Gauss Shot Impul Defoc Glass Motion Zoom Snow Frost Fog Brit Contr Elastic Pixel JPEG Avg
Source
27.0
32.0
60.6
70.7
45.9
69.2
71.2
60.5
54.2 49.7 70.5 44,9
62.8
25.3
58.8
53.5
BN udapt
31.5
32.8
31.4
40.4
31.0
39.3
40.0
35.3
35.7 31.5 40.7
38.0
34.5
36.7
31.7
35.4
ETA
31.2
32.2
32.6
38.4
30.4
37.7
38.4
34.6
34.7 32.2 39.4
36.3
33.2
36.3
31.2 34.6
LAME
30.3
36.9
73.2
84.3
57.2
83.3
85.0
76.7 66.4 63.1 84.7
54,3
79.2
28.6
73.9 65.1
COTTA
33.7 34.8
34.1
39.5
33.2
39.3
40.1
36.3
36.8 31.8 39.9
36.8
35.8
39.6
35.3 36.5
COTTA*
33.7 35.0
33.7
39.0
33.2
38.7
39.4
35.8
36.6 31.8 39.1
36.1
35.6
38.8
35.5 36.1
PL
32.2 32.0
32.5
39.2
30.7
37.8
39.2
35.0
35.0 32.1 39.5
36.8
33.5
36.9
31.2 34,9
PL+DELTA
59.2 61.0
61.6
72.0
58.8
70.1
72.2
66.2
65.2 61.
69.2
63.5
67.4
59.6 65.4
TENT
29.9
32.1
37.8
30.0
36.6
37.6
33.6
33.3 31.4 38.1
34.4
32.0
36.0
30.1 33.6
TENT+DELTA
60.3 62.7
63.1
72.7
60.2
66.7
65.9 63.4
69.8
68.5
66.3
Ent-W
31.0
32.1
32.7
38.3
30.1
37.7
38.5
34.5
34.5
32.0
36.0
32.9
36.2
30.6 34.4
Ent- W+DETA
60.2 62.3
63.5
72.3
59.6
70.0
72.3
67.3
66.3
63.2
70.3
64.2
69.2
60.5
66.3
20

---

Published as a conference paper at ICLR 2023
Table 22: Classification accuracy (%) IS+CI (sT 0.1) test stream of CIFARI00-C.
Method
Gauss Shot Impul Defoc Glass Motion Zoom Snow Frost Fog
Brit Contr Elastic Pixel JPRG Avg
Source
26.2
31.7
60.1
70.3
45.7
69.5
71.2
60.1
53.9 49.7 69.7
45.1
62.5
25.6
58.9
53.3
BN adapt
58.0
58.8
56.7
71.6
58.3
69.6
71.5
64,9
65.1
58.6 72.9
68.7
64.4
66.3
58.5
64.3
ETA
62.6
63.7 65.2
73.6
62.9
71.6
73.8
68.6
68.9 65.675.2
72.1
65.9
70.7
62.7 68.2
LAME
23.5 28.6 59.4
68.8
43.3
67.1
68.8
58.2
50.9 46.667.1
39.4
60.4
21.6
56.7
50.7
CoTTA
59.8 61.3 59.7
7188
59.8
69.6
71.6
64,4
65.3 56.5
68.5
64.2
68.2
62.5
65.1
CoTTA*
59.8 61.9 60.1
72.0
61.7
70.9
72.6
66.2
67.4 54.1
71.3
67.3
71.5
66.3 66.8
PL
61.7 62.3 62.8
73.1
61.7
71.1
63.774.3
71.3
69.7
61.3 67.2
PLEDEETAA
62.4 63.0 63.2
73.4
61.3
71.9
73.5
67.2 68.3 64.0
71.5
65.6
70.1
62.2 67.5
TENT
61.7 63.3 63,9
73.0
62.3
71.4
73.1
67.6
68.1 74.1
71.4
65.5
70.7
62.5
67.6
TENT+TBR
61.6 63.8 64.4
73.3
62.2
71.5
73.6
68.0
68.0 74.9
71.4
65.5
71.0
63.0
67.8
TENT+DOT
62.4 63.6 64.7
73.1
62.6
71.6
73.7
68.0
68.6 64.
71.8
66.1
70.7
63.0
68.0
TENT+DELTA
62.5 64.3
62.4
71.3
66.1 75.1
71.6
71.1
68.3
Ent-W
62.5 63.8 65.2
73.6
62.9
71.7
73.7
68.5
68.9 65.5
72.0
66.3
70.7
62.9
68.2
Ent- T
61.1 62.0 62.6
73.0
60.8
71.1
73.1
66.9
63.44.1
70.2
65.6
68.6
60.5
66.7
Ent - (0.11)
62.5 63.5 64.8
73.7
62.8
72.0
74,2
68.5
68.7 65.5
71.7
66.7
70.7
62.4
68.2
W+TBR+
63.3 64.1 66.2
73.9
63.2
72.0
73.8
68.9
69.5 65.
72.5
66.8
71.2
62.9
68.7
Ent- W
62.7 63.7 65.7
73.5
62.9
71.8
74.2
68.3
69.5 65.
73.1
66.5
70.9
62.9
68.5
B-T-W+TBR+LA
63.6 64.6 66.4
74,2
63.7
72.1
70.1
66.0
73.3
67.2
718
63.4
69.0
Ent- W+TBR=Sample-dro
63.3 64.6 65.8
73.8
63.6
72.2
74.0
69.5
69.7
75.6
72.5
67.0
71.5
63.1
68.8
Eat-W+DEL.T
63.9 64.8 66.4
74.1
63.7
72.2
74.4
69.2
70.5
75.6
73.3
67.0
71.6
63.3
69.1
Table 23: Classification accuracy (%) IS+CI (I 0.05) test stream of CIFAR100-C.
Method
Gauss Shot Impul Defoc Glass Motion Zoom Snow Frost Fog Brit Contr Elastic Pixel JPEG Avg
Source
26.2
31.8
60.5
70.5
46.4
68.9
70.6
59.8
53.7 50.3 70.4 44.9
61.8
24.7
58.2 53.3
BN adapt
56.7
58.0
55.5
71.4
57.5
69.5
71.1
64.7
64.1
57.5
69.0
63.1
66.2
58,0 63.6
ETA
61.3 63.2
64.6
73.6
61.5
72.2
73.3
68.1
67.7 65.0
71.4
65.6
70.2
62.8 67.7
LAME
23.2
67.9
43.8
67.8
58.2
50.5 47.1 67.7
39.8
20.6
COTTA
58.4 60.6
71.6
58.2
69.4
71.2
63.5
64.2 55.6
68.6
62.4
67.9
CoTTA*
58.4 60.9 59.1
72.0
59.9
70.9
71.8
65.2
66.5 58.6
71.0
65.7
70.5
65.2 66.0
PL
60.4 62.1 62.9
72.8
60.8
71.4
72.7
67.7
67.1 62.6
71.3
65.4
69.4
61.4 66.8
PLADELTA
61.0 63.1 62.8
73.2
61.8
71.6
73.2
67.9
67.6 63.5
71.4
65.3
69.6
62.0 67.2
TENT
61.0 63.4 64.0
73.3
60.6
71.7
73,2
68.7
66.9 64,9
71.0
65.1
70,0
62.0 67.3
TENT+DELTA
61.7 64.8 65.6
73.5
62.1
71.2
73.4
69.0
68.6
65.6
71.1
66.1
70.6
63.1 68.1
Ent-W
61.4 63.2 64.7
73.7
61.5
72.1
73.2
68.4
67.8 64.9
71.3
65.6
70.1
62.7 67.7
EDD-W+DELTA
62.8 64.4 65.6
74.4
62.5
72.3
74.1
69.1
68.9
65.2
73.0
66.1
71.7
63.0
68.6
Table 24: Classification accuracy (%) on DS+CI (p 0.5, 0.1) test stream CIFARI00-C.
Method
Gauss Shot Impul Defoe Glass Motion Zoom Snow Frost Fog Brit Contr Elastic Pixel JPEG Avg
Source
26.2
31.7
60.1
70.3
45.7
69.5
71.2
60.1
53.9 49.7 69.7
45.1
62.5
25.6
58.9 53.3
BN adapt
44.9
45.6
44.7
56.1
44.8
54.4
56.4
49.4
50.8 44.3 56.5
53.4
49.2
51.7
46.0 49,9
ETA
46.5
47.0
48.6
56.2
46.1
55.1
56.5
51.4
51.9 47.1 56.7
53.5
49.3
53.4
46.7 51.1
LAME
27.2
34.0
68.5
80.0
52.5
78.9
80.5
70.3
60.5
66.678..
49,9
72.7
26.4
68.6 60.3
COTTA
47.0 48.3
47.5
55.0
47.0
54.6
55.5
49.7
51.6 55.9 50.3
50.5
55.2
49.0 50.7
COTTA*
47.0 48.3
47.6
54.7
48.0
54.1
54.7
49.7
51.7 45.1 55.2
50.1
50.6
54.7
50.6 50.8
PL
46.0 45.9
47.7
56.0
45.8
55.4
56.4
50.7
50.7 56.4
53.5
48.8
53.2
46.8 50.6
PL+DELTA
60.3 62.1
62.9
72.6
60.9
70.9
72.4
66.6
67.4 62.273.5
69.9
65.6
69.3
62.5 66.6
TENT
46.3 46.1
47.6
55.8
45.2
54.7
55.6
49.8 50.5 46.4
51.3
48.6
52.4
45.2 50.2
TENT+DELTA
62.5 63.7
64.9
62.2
70.8
67.6
68.0
65.7
70.5
66.6
69.9
67.8
Ent-W
46.7
46.9
48.7
56.1
46.1
55.0
56.3
51.2
51.9
47.7
57.1
$3.5
49.2
53.2
51.1
Ent- W+DETA
62.4 63.9
65.0
73.5
61.9
71.4
73.5
68.1
68.8
65.4
70.7
66.2
70.4
63.3 67.9
21

---

Published as a conference paper at ICLR 2023
Table 25: Classification accuracy (%) on DS+CI (p 0.5, 5, 0.05) test stream of CIFARI00-C.
Method
Gauss Shot Impul Defoc Glass Motion Zoom Snow Frost Fog Brit Contr
Elastic Pixel JPEG Avg
Source
26.2
31.8
60.5
70.5
46.4
68.9
70.6
59.8
53.7
50.3
70.4
44,9
61.8
24.7
58.2
53.3
BN adapt
43.0
45.0
42.3
55.4
44.0
54.2
$4.9
49.1
49.4
43.8 56.2
53.1
48.5
51.3
44.3
49.0
ETA
45.4
46.4
46.8
56.2
45.3
54.7
54.8
50.7
50.1 46.5 56.4
S2.2
48.8
53.0
45.5 50.2
LAME
27.1
33.3
67.6
78.7
51.7
77.1
78.9
68.5
59.7 56.0 49.3
70.1
25.1
66.6
59.2
CoTTA
46.5
47.3
45.3
54.5
45.5
53.7
55.0
48.6
49,9
42.4 56.0
49.0
49.1
53.5
47.2 49.6
CoTTA*
46.5
47.8
45.5
54.1
46.2
53.4
54,2
48.8
50.6
43.5 54.2
49.4
49.6
52.8
48.7 49.7
PL
44.3 45.8
46.4
45.2
54.2
50.7
56.5
52.4
49.1
$2.0
45.5 49,9
PLADELTA
59.3 61.1 62.2
71.6
59.4
70.3
70.8
66.3
65.5
61474.0
69.0
64.5
67.5
59.8 65.5
TENT
44.7 46.7
45.7
55.3
44.6
53.8
53.7
50.0
48.6
46.1
55.5 50.0
48.9
52.0
44.6 49.3
TENT+TBR
58.8 61.6
62.5
72.2
58.6
70.3
70.9
67.0
64.8 62.5
68.1
63.4
68.5
59.4 65.5
TENT+DOT
45.2 47.1 46.7
55.6
45.4
54.3
54,3
50.9
49.7
49.7 47.3
51.7
49.2
52,9
45.6 50.1
TENT+DELTA
60.3 62.3 63.7
72.9
60.3
70.3
71.3
67.8
66.2 64.1
68.7
64.3
69.1
60.7 66.4
Ent-W
45.6 46.4
56.0
45.4
54.9
$4.9
46.8 56.3
52.2
48.6
53.1
45.1 50.2
Bon-WABRRDDi-W00.0
60.9 62.3 62.9
73.0
59.5
70.7
72.0
67.0
66.2
64.4
69.8
64.9
69.0
60.7 66.4
Ent WwTWBR--DDi......1
61.2 62.8 64.5
73.5
59.9
71.3
71.8
67.4
66.4
63.7
70.6
65.2
69.5
61.0 66.9
Ent-W+TBR+Div-W..
60.4 62.4 63.6
73.5
59.4
70.7
72.0
67.1
6588
74.4 70.1
64.5
69.5
60.4 66.5
Ent- (.--WWBRRDDD-
59.7 62.3 63.3
72.9
59.4
70.6
71.9
67.0
65.8 64.3 69.6
63.9
69.4
60.3 66.2
Ent W+TBR=LA
59.2 61.6 62.3
58.9
69.5
71.3
65.7
62.8
63.3
68.2
65.5
Ent W+TBR+ WampRRSSmple-drup
60.9 62.6 63.7
73.2
60.0
70.6
72.0
66.9
66.6
64.1
74,9
69.4
64.6
69.9
61.1 66.7
Ent-W+DEL W+DELTA
61.2 62.9 64.0
73.7
60.4
71.1
72.3
67.4
67.0
64.2
74.7
70.2
64.7
69.8
61.0
67.0
Table 26: Classification accuracy (%) on IS+CB test stream ImageNet-C.
Method
Gauss Shot Impul Defoe Glass Motion Zoom Snow Frust Fog Brit Contr Elastic Pixel JPEG Avg
Source
2,2
2.9
1.9
17.9
9.8
14.8
22.5
16.9
23.3
24.4 58.9
5.4
17.0
20.6
31.6
18.0
TTA
4.1
4.9
4.5
8.2
12.9
25.8
14.0
19.1
21.3
12.4
14.6
24.6
33.6
17.7
BN adapt
15.2
15.8
15.8
15.0
15.3
26.4
38.8
34.3
33.1
47.8 65.3
16.8
43.9
48.9
39.7
31.5
MEMO
7.5
8.7
9.0
19.7
13.0
20.7
27.6
25.3
28.8
32.1 61.0
11.0
23.8
33.0
37.5
23.9
ETA
35.6
37.5
36.2
33.7
33.1
47.7
52.5
51.9
45.8
60.0
44.7
57.8
60.9
55.2
48.0
LAME
1.6
2.4
1.3
17.6
9.1
13.9
21.9
15.6
22.5
52.6
5.2
15.2
19.9
31.1
17.2
CoTTA
17.6
18.0
17.4
15.6
18.2
31.2
43.6
36.6
35.1
66.5
19.5
46.3
54.9
42.6
34.4
CoTTA
17.6
22.7
29.7
38.1
45.2 60.1
26.4
53.4
46.8
35.1
PL
26.2
26.2
27.0
25.2
24.3
37.2
46.5
43.3
39.5
55.0
30.2
51.2
55.7
49.1
40.2
PL
27.7
29.4
28.5
27.0
26.1
38.1
47.9
44.1
40.7
55.9
34.1
$2.9
56.6
50.3
41.8
TENT
28.7 30.5
30.1
28.0
27.2
41.4
49.4
47.2
41.2
67.4
26.5
54.6
58.5
52.5
42.7
TENT+TBR
29.5 31.4
30.9
28.8
28.0
41.9
50.3
47.7
41.8
58.3
26.9
55.4
59.3
53.3 43.5
TENT+DOT
30.5
32.3
31.6
29.3
42.5
49.9
47.8
42.2
67.5 37.5
55.4
58.8
52.9
44.4
TENT+DELTA
31.2 33.1
32.1
30.2
42.9
48.2
43,0
68.
37.9
56.2
59.5
53.6
45.1
Ent-W
34.5
29.0
33.1
29.6
26.3
47.4
52.2
51.9
45.6
59.9
17.8
57.8
60.9
55.0
44.6
Ent W.-WTRRDDD---.0.05
36.1
37.9
37.8
34.4
33.5
49.1
$3.3
53.2
46.7
60.9 68.5
45.1
58.9
61.7
56.0
48.9
Fat- WWTHR--Di-W(O...
35.3
37.3
36.3
33.6
32.2
49.1
53.4
53.1
46.6
61.06.66...0 68.4
43.1
58.7
61.7
55.9
48.4
Ent- W.TWWBRRRDDv-.....
32.5
35.4
33.5
26.7
25.8
48.9
53.0
52.9
46.2
60.9 68.4
31.1
58.7
61.7
56.0
46.1
Ent- W+TER-Div-WO..)
28.7
32.8
19.3
48.9
$3.0
60.8 68.4
13.9
61.7
56.0
43.5
Ent W+T
26.7
22.4
29.6
20.3
20.0
49.2
53.4
52.9
46.7
60.7 68.0
10.1
58.8
61.5
56.0
42.4
Ent- W+TBR+
SSmplee-app
37.0
38.9
38.2
35.8
35.4
49.6
53.8
53,3
47.4
61.0
68.5
46.4
59.1
62.0
56.4
49,5
Ent- DELTA
38.1
39.6
39.0
36.3
36.5
49.9
54.0
53.5
47.6
61.1
68.4
46.9
59.2
61.9
56.6
49.9
22

---

Published as a conference paper at ICLR 2023
Table 27: Classification accuracy (%) on IS+CB test stream of ImageNet-C with different architec-
tures.
Method
Gauss Shot Impul Defoc Glass Motion Zoom Snow Frost Fog Brit Contr Hlastic Pixel JPEG Avg
ResNet18
Source
1.2
1.8
1.0
11.4
8.7
11.2
17.6
10.9
16.5
51.3
3,4
16.8
23.1
29.6
14.6
TENT
22.3
24.7
22.2
20.3
21.1
32.2
41.1
37.8
33.7
49.0
59.2
19.5
46.9
50.6
45.8
35.1
TENT+DELTA
24.5
26.8
24.4
22.6
23.7
34.0
42.7
38.9
35.4
50.2
60.3
27.5
48.5
51.9
47.0
37.2
Ent-W
27.1
30.7
24.3
22.3
17.5
37.6
44.2
42.5
37.8
$1.5
59.9
5.5
49.5
52.9
48.5
36.8
31.7
33.8
32.0
29.0
30.3
40.2
46.1
44.2
39.7
53.1 60.9
36,9
51.5
54.7
49.8
42.3
ResNet50
Source
2.2
2.9
1.9
17.9
9.8
14.8
22.5
16.9
23.3
24.4
5.4
17.0
20.6
31.6
18.0
TENT
28.7
30.5
30.1
28.0
27.2
41.4
49.4
47.2
47.3
26.7
54.6
58.5
52.5
42.7
TENT+DELTA
31.2
32.1
30.2
58.5
37.9
59.5
Ent-W
34.5
29.0
33.1
67.8
17.8
60.9
Ent- W+DELTA
38.1
39.6
39.0
36.3
36.5
49.9
54.0
53.5
47.6
61.1
68.4
46.9
59.2
61.9
56.6
49.9
ResNer101
Source
3.5
4.3
3.5
21.9
13.1
19.2
26.5
21.0
26.7
28.1 61.4
7.2
24.3
35.0
42.3
22.5
TENT
32.6
34,0
33.2
32.2
32.4
45.1
53.0
50.8
45.0
59.6 69.1
33.8
58.6
61.1
55.8
46.4
TENT+DELTA
35.1
37.4
35.6
34,9
35.1
46.8
54.6
51.8
46.7
60.7
69.9
42.6
60.1
62.3
57.2
48.7
Ent-W
36.1
20.8
37.3
33.6
31.7
50.3
55.6
54.9
46.8
62.4
69.8
19.7
61.1
63.2
58.2
46.8
FAI-WADHLTA
40.9
43.0
41.9
39.8
40.1
53.1
57.4
56.5
50.8
63.4
70.2
50.6
62.3
64.2
59.8
53.0
ResNet152
Source
3.6
4.4
3.3
11.9
24.8
25.5
28.9
27.7
5.2
42.2
22.5
TENT
34.0
36.8
35.3
34.0
54.0
61.3
70.7
35.5
62.4
TENTADELTA
36.6
39.2
37.7
36.7
36.3
55.6
54.0
48.4
02.471..
44.0
63.3
50.2
Ent-W
38.7
33.4
34.6
36.6
33.2
57.4
56.9
46.5
71.0
29.3
62.7
64.8
60.0
49.5
Ent WDDETA
42.6
45.4
44.5
42.0
42.2
55.5
58.9
58.5
52.7
65.5
71.4
51.9
63.7
65.8
61.2
54.8
WideResNer50
TENT
34.5
37.2
34.7
30.6
31.6
45.2
52.0
51.1
45.8
60.5
69.9
38.4
58.3
61.7
54.9
47.1
TENTADELTA
36.7
39.6
37.2
33,5
34.6
47.4
54.5
53.0
47.6
62.2
71.2
44.1
60.3
63.4
56.9
49.5
Ent-W
34.0
37.1
33.6
25.0
27.7
51.0
54.7
55.5
49.9
62.8
70.4
24.9
60.7
63.9
57.6
47.3
Ent- W+DELTA
41.1
44.9
42.9
38.6
39.3
53.4
57.3
57.6
51.8
64.7
71.4
52.0
62.4
65.7
59.8
53.5
ResNeX150
TENT
33.3
36.2
34.2
32.3
30.9
45.5
52.2
45.9 59.6 69.3
39.0
57.1
61.5
53.8 46.8
TENT+DELTA
35.3
38.5
36.1
33.5
52.1
60.5
69.9
43.9
62.4
Ent-W
31.4
37.5
34.7
34.0
25.2
49.1
70.0
49.1
64.3
Ent- -+DELT
40.7
43.6
42.0
39.5
39.1
53.1
56.7
56.6
51.1
63.2
70.4
50.7
61.5
64.9
58.2
52.8
Table 28: Classification accuracy (%) on DS+CB (p = 1.0) test stream of ImageNet-C.
Method
Gauss Shot Impul Defoc Glass Motion Zoom Snow Frost Fog Brit Contr Elastic Pixel JPEG Avg
Source
2.2
2.9
1.9
17.9
9.8
14.8
22.5
16.9
23.3
24.4 58.9
5.4
17.0
20.6
31.6
18.0
BN adapt
10.6
10.9
10.9
10.2
10.3
17.5
25.8
23.5
23.1
33.0
11.3
30.2
33.3
27.0
21.6
ETA
17.0
19.2
18.2
14.1
12.0
25.9
31.1
30.9
26.8
38.6
46.1
18.9
36.0
38.7
33.5
27.1
LAME
1.8
2.7
1.5
22.4
11.3
17.2
28.4
19.8
28.4
29.8 74.4
5.9
20.0
25.6
40.4
22.0
COTTA
12.2
12.5
12.8
9.5
11.2
19.7
28.4
24.7
23.9
47.4
12.8
31.1
37.0
28.4
23.2
COTTA*
12.2
14.9
16.2
12.3
14.2
18.9
24.2
24.4
25.2
41.0
15.3
30.8
35.5
31.2
23.1
PL
15.9
15.6
16.4
14.4
13.9
23.1
29.8
28.1
26.2
37.3
12.8
34.2
37.2
32.2
25.6
PL+DELTA
26.3
27.4
27.1
43.0
54.8 66.6
32.7
55.6
48.6
40.5
TENT
16.1
16.8
15.1
14.1
28.8
24.9
36.5
9.3
37.8
25.7
TENT+DELTA
29.6
31.7
30.4
29.1
28.6
41.5
49.8
47.0
42.1
67.6
35.7
54.9
58.5
52.0
43.7
Ent-W
4.2
2.8
3.1
2.9
3.6
11.3
20.2
20.0
12.5
34.4
1.7
32.0
37.1
21.5
16.8
E
35.6
37.9
36.0
34.4
34.4
47.9
52.8
$1.9
46.5
60.1
67.8
44.2
$7.9
60.8
55.4
48.3
23

---

Published as a conference paper at ICLR 2023
Table 29: Classification accuracy (%) on DS+CB (p 0.5) test stream of ImageNet-C.
Method
Gauss Shot Impul Defne Glass Motion Zoom Snow Frost Fog
Brit Contr Elastic Pixel JPEG Avg
Source
2.2
2.9
1.9
9.8
14.8
22.5
16.9
23.3
24.4 58.9
5.4
17.0
20.6
31.6
18.0
BN adapt
9.6
9.9
9.8
8.8
9.1
15.8
22.8
21.0
20.8
29.541.9
10.3
26.7
29.5
24.2
19.3
ETA
13.9
15.5
13.3
11.1
10.3
21.4
26.2
26.1
22,9
40.4
13.4
30.9
33.2
29.3
22.8
LAME
1.9
2.8
1.6
23.6
11.7
17.8
29.4
20.4
29.4
30.5 76.1
6.2
20.8
26.4
41.5
22.7
CoTTA
10.8
11.0
11.0
7.8
10.4
17.4
25.0
22.0
21.6
32.042.6
9.9
27.9
32.5
25.8
20.5
CoTTA*
10.8
13.3
14.3
12.9
17.1
22.0
21.8
22.8 27.6
14.8
27.8
32.6
28.0
21.0
PL
14.2
13.4
14.5
12.5
11.5
20.0
26.3
23.4
33.242.4
11.1
30.5
33.1
28.5
22.6
PLADELTA
25.6
27.3
26.2
25.2
24.6
36.0
45.7
42.9
39.3
54.4 66.5
31.0
50.6
55.0
47.9
39.9
TENT
13.9
14.6
14.5
12.6
11.7
19.0
26.1
25.2
21.5
33.2
6.5
30.5
33.1
28.9
22.2
TENT+TBR
27.3
28.5
28.2
26.0
25.4
38.9
48.5
46.0
39.6
57.1 67.3
18.5
53.6
57.6
51.2
40.9
TENT+DOT
15.5
16.5
15.9
14.0
20.9
27.1
25.9
23.5 33.7
15.2
31.4
33.5
29.5
23.9
TENT+DELTA
29.1
30.9
27.8
40.3
67.3
58.1
43.1
Ent-W
2.9
2.5
3.5
1.4
1.0
7.1
11.9
15.1
8.5 27.7
1.3
22.2
31.1
20.1
12.9
Ent W
32.4
34.6
33.3
27.2
28.3
45.2
51.3
50,5
44.3
59.3
36.0
$7.0
60.1
54.3
45.4
30.1
33.4
31.1
25.5
21.7
44.8
51.1
50.4
43,4 59.3
16.3
56.9
60.2
54.3
43.1
Ent- W-TBR-DD-DD-
23.7
30.5
26.5
19.7
12.2
44.3
51.1
50.5
41.1 59.4
7.0
56.8
60.2
54.2
40.3
Ent W+TBR-DIv-(004)
17.1
15.3
22.2
4.8
43.7
51.1
59.56744
56.7
60.3
54.2
37.1
Ent W+TBR+IAA
10.9
7.2
14.3
5.1
5.0
35.0
39.9
39.3
23.2
53.6
4.1
44.6
47.2
42.4
27.9
Ent- W+TBR+ W4TBR--ample-drop
33.7
36.4
35.1
31.9
30.8
46.7
52.2
51.2
45.6
60.067.6
40.4
57.3
60.6
54.7
46,9
Ent-W+DELTA
34.9
37.5
35.8
32.7
32,3
46.7
52.3
51.5
46.0
59.7
67.3
42,8
57.3
60.4
54.9
47.5
Table 30: Classification accuracy (%) on DS+CB (p 0.1) test stream of ImageNet-C.
Method
Gauss Shot Impul Defoe Glass Motion Zoom Snow Frost Fog Brit Contr Elastic Pixel JPEG Avg
Source
2.2
2.9
1.9
17.9
9.8
14.8
22.5
16,9
23.3
24.4 58.9
5.4
17.0
20.6
31.6
18.0
BN udapt
6.3
6.4
6.2
5.6
5.6
9.8
13.9
13.6
13.4
18.4 26.1
6.4
16.5
18.1
15.0
12.1
ETA
4.6
5.2
5.1
2.3
2.8
7.0
12.3
11.6
10.9
22.5
2.3
14.5
17.1
14.4
10.0
LAME
1.9
2.9
1.6
26.2
12.8
19.8
32.7
22.8
32.5
33.8 80.0
6.6
22.5
29.0
45.5
24.7
CoTTA
7.1
7.0
7.1
5.0
6.2
10.5
15.0
14.1
13.7
19.3
6.4
17.1
19.6
15.8
12.7
COTTA*
7.1
8.2
8.7
6.3
10.3
13.4
14.3
14.5
17.9
7.9
16.9
19.4
17.3
12,9
PL
7.7
7.6
10.8
15.4
15.0
20.0
5.0
17.5
19.4
17.0
13.1
PL+DELTA
23.4
24.6
24.0
22.0
21.5
43.4
40.0
37.3
65.0
26.1
52.5
45.4
37.2
TENT
7.4
7.8
7.8
6.2
5.9
8.9
14.7
12.5
11.6
19.0
3.0
16.8
18.5
16.5
12.1
TENT+DELTA
26.7
28.2
27.3
25.0
24.8
37.1
46.6
43.6
39.6
55.1 65.7
27.2
51.6
55.6
49.0
40.2
Ent-W
1.5
0.6
1.4
1.1
0.8
2.3
4.6
4.4
3.1
8.4
15.5
0.5
7.0
9.7
5.7
4.4
EnE-W+DELTA
30.4
33.1
31.4
26.8
28.1
42.2
48.9
48.2
42.6
65.4
31.5
54.4
57.8
51.5
43.3
Table 31: Classification accuracy (%) on IS+CI (IT 0.1) test stream of ImageNet-C.
Method
Gauss Shot Impul Defoc Glass Motion Zoom Snow Frost Fog Brit Contr Elastic Pixel JPRG Avg
Source
2.4
3.0
1.9
17.8
9.7
14.7
22.4
16.5
23.1
58.9
16.9
20.4
31,5
17.9
BN adapt
15.0
15.8
15.4
15.1
25.6
39.1
34,4
33.2
47.8 65.1
17.5
444
48.8
39.8
31,5
ETA
34.6
36.7
35.7
33.1
32.5
46.5
$1.9
51.3
45.4
67.6
44.8
57.3
60.9
55.1
47.5
LAME
1.8
2.5
1.5
17.5
9.0
13.9
21.8
15.1
22.3
58.5
5.3
14.9
19.8
30.9
17.2
CoTTA
17.1
17.8
17.5
15.9
16.7
30.2
43.2
36.8
35.7 51.9
17.7
47.1
54.0
42.8
34.1
CoTTA*
17.1
22.0
24.1
22.2
28.0
35.7
35.2
35.8 42.857.8
22,9
44.8
50.4
45,3
33.5
PL
24.9
24.8
25.9
24.3
23.4
36.2
45.7
39.5
54.6 66.5
28.6
55.5
48.5
39.4
PLADELTA
26.4
27.7
27.0
26.3
24.9
37.5
46.9
43.3
40.2
66.3 33.3
52.1
56.5
49.8
40.9
TENT
27.8
29.3
29.2
28.1
26.6
40.8
48.7
46.5
41.0
57.2
25.7
53.6
58.2
51.9
42.1
TENT+TBR
28.5 30.1
29.7
28.7
27.3
41.3
49.9
47.0
41.7
67.6
25.1
54.5
59.0
$2.9
42.7
TENT+DOT
29.8 31.6
30.9
29.4
28.8
41.7
49.4
47.0
42.1
57.3 67.3 36.8
54.9
58.6
52.4
43.9
TENT+DELTA
30.7
32.5
29.3
42.0
50.5
42,9
57.867.7 36.4
53.1
44.4
Ent-W
23.2
21.7
29.4
19.1
19.6
46.7
51.7
$1.0
39.0
58.9 10.1
$7.2
60.5
$4.9
40.7
Ent- W.TBR-DDi--W.0055
34.1
37.4
36.4
32.5
32.9
47.7
$2.9
52.1
45.7
60.0 42.6
57.8
61.7
55.7
47.8
Ent W+TBR-Div.Wo0
1(0.)
34.5
36.1
35.9
32.4
32.0
48.0
$2.9
$2.1
45.8
68.0 40.2
$7.9
61.5
55.7
47.5
Won-W+BBR-Div...
32.5
34.1
35.3
30.0
29.7
47.6
52.7
51.9
45.5
59.7 68.0 30.2
$7.9
61.5
55.7
46.1
Ent- -DBB-
29.2
27.5
34.3
27.4
25.1
47.8
52.8
51.8
44.7
68.5
58.0
61.4
55,8
43.3
24.8
23.5
34.6
25.1
20.4
48.2
$2.9
52.2
45.0
67.7
4.2
58.0
61.3
55.8
42.2
Ent- W4TTR+Sammppp
W+TBR-Sampp--drrrp
36.1
37.8
37.3
33.7
33.2
47.3
52.9
52.1
46.0
59.7
68.0
43.7
57.9
61.5
55.5
48.2
Ent- W+DELTA
36.6
38.6
37.8
34.9
34.4
47.7
$2.6
51.9
46.1
59.5
44.6
57.9
60.9
55.4
48.4
24

---

Published as a conference paper at ICLR 2023
Table 32: Classification accuracy (%) IS+CI (х 0.05) test stream of ImageNet-C.
Method
Gauss Shot Impul Defoc Glass Motion Zoom Snow Frost Fog Brit Contr
Elastic Pixel JPEG Avg
Source
2.2
2.9
1.9
18.0
10.0
14.6
22.5
16.6
23.1
24.4 58.4
5.5
16.9
20.6
31.5
17.9
BN adapt
15.1
15.2
15.6
14.9
15.8
25.6
39.0
34.6
33.2
47.8 64.7
17.2
44.1
48.2
39,9
31.4
ETA
34.2
35.0
32.0
32.0
52.0
50.6
45.0
59.4
43.4
57.0
60.3
54.5
47.0
LAME
2.4
17.7
13.9
21.9
15.4
22.3
22.7
5.2
15.1
19.8
17.2
COTTA
17.3
17.4
17.8
15.4
17.1
29.8
43.1
37.5
35.4
51.9
19.3
53.3
42.5
34.0
CoTTA*
17.3
21.6
23.8
19.9
22.9
29.3
37.4
35.7
36.6
44.5 59.0
24.2
45.5
51.8
45.9
34.4
PL
24.2
24.6
25.8
24.7
23.5
36.2
45.8
42.7
38.9
65.9
27.0
49.0
55.0
48.0
39.0
PL+DELTA
26.1
27.3
27.1
25.8
25.3
36.2
46.8
43.2
39.9
54.8 66.4
32.6
51.1
55.4
48.8
40.5
TENT
27.1
29.0
28.8
27.7
27.1
40.3
49.1
46.4
40.7
57.1 66.6
24.8
53.1
57.8
51.3
41.8
TENT+DELTA
30.1
32.3
31.2
29.6
29.6
41.4
50.0
47.4
42.4
67.2
35.3
55.1
58.5
52.6
44.0
Ent-W
17.2
13.4
25.6
15.8
12.1
45.9
51.0
50.4
44.6
66.9
10.0
56.5
60.0
54.1
38.9
Ent-W+DELTA
35.7
38.2
37.1
34.1
33.8
46.5
51.7
51.1
45.6
66.0
43.5
57.0
59.3
54.5
47.5
Table 33: Classification accuracy (%) on (p 0.5, π 0.1) test stream of ImageNet-C.
Method
Gauss Shot Impul Defoc Glass Motion Zoom Snow Frost Fog Brit Contr Elastic Pixel JPEG Avg
Source
2.4
3.0
1.9
17.8
9.7
14.7
22.4
16.5
23.1
24.2 58.9
5.5
16.9
20.4
31.5
17.9
BN adapt
9.7
10.1
10.1
9.3
9.5
16.1
24.2
21.7
21.4
30.8 43.5
11.2
27.0
30.9
25.7
20.1
ETA
11.9
12.7
12.7
8.6
7.3
18.8
27.1
25.9
22.8
34.041.9
7.9
30.7
34.9
30.3
21.8
LAME
2.0
2.8
1.5
22.4
11.4
17.1
27.5
19.4
28.0
23.5
6.0
19.8
24.9
40.0
21.7
COTTA
11.4
11.6
11.7
9.8
10.4
17.9
26.4
22.9
22.6
33.
11.6
28.6
33.8
27.3
21.6
COTTA*
11.4
13.9
14.9
11.7
13.3
17.9
22.8
22.7
23.5
29.2
14.6
28.2
33.1
29.6
21.7
PL
14.4
12.5
14.0
12.6
11.8
20.2
27.2
25.3
24.1
43.9
10.7
29.8
34.2
29.8
23.0
PLADELTA
24.8
25.5
25.4
23.6
23.0
34.9
44.8
41.0
38.8
53.2 65.9
29.6
49.7
54.1
47.3
38.8
TENT
12.9
13.9
14.3
12.8
18.5
27.0
25.0
21.7
34.1
6.6
30.1
34.5
30.1
22.4
TENT+DELTA
28.3
30.1
29.1
27.2
48.3
45.2
41.2
56.3
31.0
57.2
42.2
Ent-W
1.6
1.6
2.4
2.3
1.3
5.6
12.9
13.5
11.1
16.7
40.4
1.1
16.8
17.4
16.6
10.8
Ent-W+DELTA
32.2
35.0
34.1
30.5
29.4
44.8
50.7
49.5
44.5
58.1
66.6
36.6
55.7
58.4
53.7
45.3
Table 34: Classification accuracy (%) on DS+CI (p 0.5,
0.05) test stream of ImageNet-C.
Method
Gauss Shot Impul Defne Glass Motion Zoom Snow Frost Fog Brit Contr Elastic Pixel JPRG Avg
Source
2.2
1.9
10.0
14.6
22.5
16.6
23,1
24.4 58.4
16.9
20.6
31.5
17.9
BN adapt
9.8
9.9
10.5
9.6
9.5
15.8
23.8
22.1
21.8
31.443.9
11.2
26.9
30.6
25.8
20.2
ETA
9.1
11.2
12.5
3.9
7.8
18.2
26.3
25.6
22.4
34.441.9
6.4
30.7
33.7
29.7
20.9
LAME
1.8
2.7
1.6
21.9
11.5
16.9
27.3
19.3
28.3
71.4
5.8
20.2
24.6
39.1
21.5
CoTTA
11.3
11.4
12.1
8.9
10.0
17.9
26.3
23.4
23.0
44.9
10.1
29.0
34.1
27.6
21.6
CoTTA*
11.3
13.6
14.9
12.1
13.4
17.8
23.1
23.3
23.5
29.1
40.0
13.9
28.4
33.1
29.8
21.8
PL
13.3
11.2
14.7
12.4
12.4
19.6
27.1
25.7
24.2
34.7
30,0
22.8
PL+DELTA
23.8
25.0
25.1
23.4
22.6
33.6
44.3
41.2
38.7
53.1 65.4
27.9
48.9
53.6
46.6
38.2
TENT
12.6
13.6
14.3
12.6
11.4
17.2
26.6
25.2
21.7
34.643.0
6.0
29.6
34.2
30.3
22.2
TENT+TBR
24.7
26.6
26.9
24.8
24.7
37.1
47.0
44.4
39.0
66.2
15.5
51.0
56.3
50.0
39.3
TENT+DOT
15.4
16.6
16.4
14.6
14.5
20.1
27.7
26.5
24.1
35.4
13.6
31.4
35.2
31.3
24.4
TENT+DELTA
27.5
29.4
27.2
38.4
56.0 66.4
50.5
41.5
Ent-W
0.9
1.5
3.6
0.8
1.4
5.9
11.7
10.8
8.9
23.0
0.5
18.0
23.5
13.9
10.7
Ent- WrTWT+BB+Dii D(0-.
27.0
28.5
29.4
21.3
23.3
40.1
48.5
48.1
42.1
57.3
13.4
54.4
58.3
52.6
40.7
24.3
28.8
28.8
16.5
22.0
40.0
48.6
48.1
41.5
57.1 66.2
6.9
54.7
58.5
52.5
39.6
Ent- W-TER-DD----....
20.6
22.6
24.4
9.4
15.0
39.9
49.2
48.5
42.4
57.2 66.4
3.2
54.7
58.7
52.5
37.6
Ent- W+TBR-Div-W0.4)
12.5
10.7
15.0
13.4
57.1 66.4
58.6
$2.7
35.2
Ent- -AA
7.5
7.6
13.0
6.4
33.8
39.7
39.3
30.4
46.3 54.1
1.7
44.3
47.3
42.7
27.8
Ent- W+TBR-SSmple-dr
27.9
32.2
30.9
24.3
27.0
40.8
48.9
48.2
41.9
56.4
65.8
29,3
54.0
58.0
52.2
42.5
Ent- MD-D+
30.8
34.4
33.0
28.7
29.3
42.8
49.7
49.1
43,9
57.2
36.7
54.9
58.6
52.7
44.5
25


Reviews:
[{"review_id": "zKlsqTLD5m", "reviewer": "Reviewer_BBcu", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "**Summary:**\n\nThis paper presents a new Test-time adaptation (TTA) method called DELTA for debiased fully TTA. To be specific, the authors 1) introduce batch renormalization to alleviate the bias in normalization statistics, 2) propose dynamic online re-weighting (DOT) to address the class bias within optimization.", "summary_of_the_review": "The proposed method alleviates the bias resulted from the small mini-batch and dominant class, which is simple and effective. Thus I vote for accept. But some importance details in TBR are missing or unclear. I hope the authors can address my concerns.", "strength_and_weaknesses": "**Positive Points:**\n\n - The authors point out the normalization statistics may be biased with the current test mini-batch and the optimization gradient may be biased with the dominant class.\n\n - The proposed methods including TBR and DOT are simple and effective.\n\n **Negative Points:**\n\n - In Section 3.2, I suggest the authors can provide more descriptions of stopping gradient operation. In my opinion, the word \"stopping gradient\" is confusing and I can not understand the meaning of this. Some readers may be unfamiliar with this and have to check for the orginal batch renormalization paper (Ioffe, 2017). Beside, the batch renormalization seems to be different from the original one. Thus I strongly recommend the authors can provide a detailed Algorithm of TBR like Algorithm 1 (DOT) of the paper.\n\n - About TBR, some details are missing. What is the output of batch renormalization? $v^*$ or $\\gamma v^* + \\beta$? And do $\\hat{\\sigma}^{ema}$ and $\\hat{\\mu}^{ema}$ need to be updated?\n\n- I noitice the authors mentioned \"TBR discards the warm-up and truncation operation\". Thus, in this case, $v^{*} = (v - \\hat{\\mu}^{batch}) / \\hat{\\sigma}^{batch} \\cdot r + d = (v - \\hat{\\mu}^{batch}) / \\hat{\\sigma}^{batch} \\cdot (\\hat{\\sigma}^{batch} / \\hat{\\mu}^{ema}) + (\\hat{\\mu}^{batch} - \\hat{\\mu}^{ema}) / \\hat{\\sigma}^{ema} = (v- \\hat{\\mu}^{ema}) / \\hat{\\sigma}^{ema}$. I found that it is the same as the original BN because the authors remove truncation operation, right? The differences between the proposed TBR and the regular BN are unclear.\n\n - In the analysis of the results in Table 2, the authors state \"Different from the TEMA, TBR is well compatible with gradient-based adaptation methods\". Could the authors give more explanations?\n\n - In Table 3, what are variance and range? Do the authors calculate them in some features of a layer?\n\n - In Table 3, what are the differences between the baselines efficient test-time adaptation (ETA) and entropy-based weighting (Ent-W)?\n\n  - Would the authors release the code upon acceptance?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ojL16JigFlt", "reviewer": "Reviewer_E7Gz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents a method named Debiased Fully Test-time Adaptation (DELTA) to address the biased issue in test-time adaptation. Specifically, the authors conduct experiments to verify the claims that 1) the normalization statistics tend to fit the current test mini-batch, and 2) the test-time adaptation optimization would bias to some dominant classes. The authors first adopt the renormalization technique to alleviate the biases in the normalization statistics of batch normalization, and then devise a re-weighting module to assign different weights for test samples with different pseudo labels to address the optimization issue. Extensive experiments on the ImageNet-C, ImageNet-R, and CIFAR 100-C demonstrate the effectiveness of the proposed method. However, I have some concerns about this paper. My detailed comments are as follows. ", "summary_of_the_review": "This paper proposes a new setting of test-time adaptation, namely class imbalance in the mini-batch, and proposes to address this problem from the de-bias perspective, which is interesting. Though the pure technical contribution is not very significant, it is new in the area of TTA. Some results are still missing to convince me regarding its effectiveness.", "strength_and_weaknesses": "Strength:\n\n1.\tThis paper introduces a new setting, namely the class imbalance issue during the test-time process, in the test-time adaptation.\n\n2.\tThe authors dig out the issues of existing test-time adaptation methods (i.e., the bias issue) from a new perspective.\n\n3.\tExtensive experiments on the ImageNet-C, ImageNet-R, and CIFAR 100-C demonstrate the effectiveness of the proposed method.\n\nWeakness:\n\n1.\tThe technical contribution is not very significant. For example, the TBR just adopts the batch renormalization technique from the initial Batch Normalization paper, and the re-weighting technique is derived from class-wise re-weighting.\n\n2.\tAs referred to the Treatment II, some components of DOT have multiple options. However, the ablation studies about the components of DOT and the selection of the components of DOT may be missing.\n\n3.\tIn Table 8, when conducting experiments on the real-world out-of-distribution dataset ImageNet-R, the improvement of DELTA is marginal compared with that on the ImageNet-C dataset, more discussions are required.\n\n4.\tIn Table 5, how about the performance of DELTA with \\rho<0.1 (e.g., 0.01?), i.e., totally concentrate on one same class during a period. Similarly, could the authors provide more results regarding smaller \\pi<0.05 (e.g., 0.001)? I am curious about these results since any value of \\pi and \\rho may appear in practice.\n\n5.\tIn Table 10, could the authors further provide the results of Div-W+Fisher regularization (namely EATA, this is the full version of ETA in Niu et al, 2022)?\n\n6.\tFigure 1 is somewhat confusing. It is hard to distinguish the difference between IID and CI, non-IID and CI & non-IID.\n\n7.\tIn Section 3.1, it would be better to detail describe each scenario and the difference between the previous TTA setting.\n\n8.\tIn Treatment II of Section 3.3, it would be better to extend the “L x” to “Line x of Algorithm 1” to improve the readability of the paper.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "nXV-HOfDzjm", "reviewer": "Reviewer_MFwz", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper diagnosis two shortcomings of existing approaches to fully test-time adaption and proposes two methodological extensions to overcome these shortcomings. The first shortcoming is that estimates of means and variances in batch normalization (BN) have relatively high error on a per-batch level; this is addressed by replacing BN by batch renormalization (Ioffe, 2017). The second shortcoming identified is that self-learning in test-time adaptation can reinforce imbalanced class distributions; this is addressed by a dynamic online re-weighting of test samples.", "summary_of_the_review": "The paper contains novel contributions that are well motivated and have sufficient empirical support. However, the paper in its current form, lacks clarity since some basic terminologies are used inconsistently with their definitions in statistics/machine learning. Moreover, there remain some doubts regarding DOT that need to be addressed. While the paper has sufficiently novel contributions, is needs a major revision before it can be accepted in my opinion.\n\n## Update after author response ##\nThe authors have largely revised the manuscript and addressed my main concerns. I am no leaning towards recommending acceptance of this paper and have improved my score accordingly.", "strength_and_weaknesses": "Strength:\n * Introduction and Related Works are well written.\n * The presentation and motivation of the methods is nicely structured (I like the split into diagnosis and treatment)\n * Using Batch Renormalization (BRN) instead of Batch Normalization is a simple but very good idea\n * The experimental evaluation is extensive and supports the proposed methods\n\nWeaknesses:\n * Confusing use of terminology: \n    + Using IID to refer to a setting with covariate shift ($p^{train}(x|y) \\neq p^{test}(x|y)$) because samples are drawn independently is misleading since IID typically refers to situation where the test data is drawn IID from the training distribution. A better term might be \"Independent Samples + Covariate Shift\" (IS-CS) vs \"Dependent Samples + Covariate Shift\" (DS +CS)\n    + Similarly, denoting class distribution shift ($P^{train}(y) \\neq P^{test}(y)$) as \"Class Imbalance\" is confusing. This indicates that the author assume that $P^{train}(y)$ is uniform over classes (and $P^{test}(y)$ is not) - the experiments are also conducted in this setting but it is a more special setting than dealing with $P^{train}(y) \\neq P^{test}(y)$. Thus, the paper should either make more clear that $P^{train}(y)$ is assumed as the uniform distribution over classes or the term \"Class Shift\" should be used instead of \"Class Imbalance\".\n    + Section 3.2 confuses the \"bias\" and the \"variance\" of an estimator: judging from Figure 2, \"BN adapt\" has an empirical mean very close to \"Global\" and thus provides a (nearly) unbiased estimate of the global mean/standard deviation. However, as the authors not, \"BN adapt\" has high fluctuations and thus provides a high _variance_ estimate of the global mean/standard deviation. Accordingly, Batch Renormalization is a variance reduction technique here and not a debiasing technique.\n   + Also, the usage of the term \"bias\" in Section 3.3 is unfortunate: TENT etc. do not have an intrinsic _bias_ towards certain classes and it is also no a priori necessary to \"overweight\" rarely predicted classes; only if one assumes that  $P^{test}(y)$ is uniform, an intervention like overweighting is recommended (and such an intervention actually _biases_ the method to predicting classes uniformly in expectation).\n   + In summary, the methods discussed in this paper are not \"debiasing\" but rather a variance reduction technique and a useful inductive biases (uniform class frequency). It is unfortunate that the term \"debiased\" even made it into the title of the paper and the name of the method.\n * Doubts about the method DOT: \n   * The authors motivate DOT with \"[the] common constraint of making the output distribution uniform Liang et al. (2020) is no longer reasonable.\". However, it seems that DOT is doing exactly that: assigning weight to samples inversely proportional to the frequency of their predicted label encourages the output distribution to be uniform. Can the authors comment on that?\n   * Using a \"hard label\" in DOT (line 5 in Algorithm 1) seems suboptimal. Why not use the soft labels?\n   * The weight in line 6 can in principle become arbitrary large; it seems more appropriate to define it as $1 / (z_t[k^*_{m_t + b}] + \\epsilon)$ for some small $\\epsilon$\n   * Comparison to other approaches for controlling the output distribution are missing. For instance, Mummadi et al. (https://arxiv.org/abs/2106.14999) proposed a KL-based diversity regularizer for matching the empirical output distribution to a prior output distribution. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "AIekOYWkFQa", "reviewer": "Reviewer_Hnwy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes two improvements to test time adaptation in the streaming data setting: one to address the possibility of non IID test data, and one to address the possibility of class imbalance in the test distribution. These improvements are relatively plug-and-play with existing popular adaptation approaches, e.g., adaptation via batch normalization and entropy minimization. Taken together, these improvements seem to consistently improve adaptation performance on CIFAR and ImageNet distribution shift test sets in which non-IID-ness and class imbalance are artificially introduced.", "summary_of_the_review": "In summary, the paper is reasonably well written and the experiments are sound, but there are several important improvements that can be made as laid out above. I am recommending weak reject and am happy to discuss further.\n\nEdit after author response\n---\n\nI apologize for my late response. I believe that the authors have adequately addressed my concerns and I am updating my recommendation to weak accept.", "strength_and_weaknesses": "Strengths\n---\n\n- The paper provides a number of experiments and ablations which demonstrate the effectiveness of the proposed approach.\n- The paper is reasonably well written and well motivated.\n- The paper studies a potentially important scenario that has received less attention in prior papers studying test time adaptation to distribution shift.\n\nWeaknesses\n---\n\n- There are no error bars in the experimental results.\n- Use of the term \"bias\" seems incorrect -- see next section for more detail.\n- There are no experiments studying a real world scenario in which the problems of non-IID-ness and class imbalance occur naturally.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster