SoTTA: Robust Test-Time Adaptation on Noisy Data Streams
Authors: Taesik Gong
Yewon Kim
Taeckyung Lee
Sorn Chottananurak
Sung-Ju Lee
Venue: NeurIPS 2023 poster (2023)
Keywords: test-time adaptation
domain adaptation
deep learning
machine learning
TL;DR: This paper investigates the impact of out-of-interest samples for existing test-time adaptation (TTA) algorithms and present a robust TTA algorithm against out-of-interest samples.

Abstract: Test-time adaptation (TTA) aims to address distributional shifts between training and testing data using only unlabeled test data streams for continual model adaptation. However, most TTA methods assume benign test streams, while test samples could be unexpectedly diverse in the wild. For instance, an unseen object or noise could appear in autonomous driving. This leads to a new threat to existing TTA algorithms; we found that prior TTA algorithms suffer from those noisy test samples as they blindly adapt to incoming samples. To address this problem, we present Screening-out Test-Time Adaptation (SoTTA), a novel TTA algorithm that is robust to noisy samples. The key enabler of SoTTA is two-fold: (i) input-wise robustness via high-confidence uniform-class sampling that effectively filters out the impact of noisy samples and (ii) parameter-wise robustness via entropy-sharpness minimization that improves the robustness of model parameters against large gradients from noisy samples. Our evaluation with standard TTA benchmarks with various noisy scenarios shows that our method outperforms state-of-the-art TTA methods under the presence of noisy samples and achieves comparable accuracy to those methods without noisy samples. The source code is available at https://github.com/taeckyung/SoTTA.

SoTTA: Robust Test-Time Adaptation on Noisy Data Streams
Taesik Gong"*
Yewon Kim
Taeckyung Lee
Sorn Chottananurak Sung-Ju Leet
Nokia Bell Labs *KAIST
taesik. www.ggoooooa---lll-laaa.ae.e.-aa com
Cyewon. e, kim, sornl profs] ac.
Abstract
Test-time adaptation (TTA) aims to address distributional shifts between train- ing and testing data using only unlabeled test data streams for continual model adaptation. However, most TTA methods assume benign test streams, while test samples could be unexpectedly diverse in the wild. For instance, unseen object or noise could appear in autonomous driving. This leads to a new threat to existing TTA algorithms; we found that prior TTA algorithms suffer from those noisy test samples as they blindly adapt to incoming samples. To address this problem, we present Screening out Test-Time Adaptation (SoTTA), novel TTA algorithm that is robust to noisy samples. The key enabler of SoTTA two-fold: (i) input- wise robustness via high-confidence uniform-class sampling that effectively filters out the impact of noisy samples and (ii) parameter- wise robustness via entropy- sharpness minimization that improves the robustness of model parameters against large gradients from noisy samples. Our evaluation with standard TTA benchmarks tf-e--t
with various noisy scenarios shows that our method outperforms TTA methods under the presence of noisy samples and achieves comparable ac- curacy to those methods without noisy samples. The source code is available https:/ gi github....... cww htt///teekuugSSTTTC
Introduction
Deep learning has achieved remarkable performance in various domains [6, 8, 33], but its effectiveness is often limited when the test and training data distributions are misaligned. This phenomenon, known as domain shift [31], is prevalent in real-world scenarios where unexpected environmental changes and noises result in poor model performance. For instance, in autonomous driving, weather conditions can change rapidly. To address this challenge, Test-Time Adaptation (TTA) [1, 5, 29, 38, 39, 44] has emerged as promising paradigm that aims to improve the generalization ability of deep learning models by adapting them to test samples, without requiring further data collection or labeling costs.
While TTA has been acknowledged as a promising method for enhancing the robustness of machine learning models against domain shifts, the evaluation of TTA frequently relies on the assumption that the test stream contains only benign test samples of interest. However, test data can be unexpectedly diverse in real-world settings, containing not only relevant data but also extraneous elements that are outside the model's 's scope, which we refer to noisy samples (Figure 1). For instance, unexpected noises can be introduced in autonomous driving scenarios, such as dust on the camera or adversarial samples by malicious users. As shown in Figure 2, we found that most of the prior TTA algorithms showed significantly degraded accuracy with the presence of noisy samples (e.g., 81.0% 31.0% + 52. 1% in TENT [38] and 82.2% - 54.8% in CoTTA [39]).
Equal contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Online data stream
Claselficaton accuracy (%)
80-
Clean TTA
60
Update
Acc: 81%
Noisy TTA
2
Update
Acc: 52%
Source
0
Clean
2 Nolay
stete
PL
TENT
COTTA
SAR
SoTTa
Benign sample
Noisy sample
Figure I: Unlike prior assumptions (Clean TTA), real-
Figure 2: Average classification accuracy (%) of existing TTA methods and our method (SoTTA) ( on CIFAR The perfor- mance of existing methods degrades when noisy data are mixed into the test stream (Noisy) compared with the original assump- tion (Clean). Higher is better.
world test streams could include unexpected noisy sam-
ples out of the model's scope (Noisy TTA), such as
glare, fallen leaf covering the lens, unseen objects (e.g..
flamingo), and noise in autonomous driving scenar-
ios. The accuracy of existing TTA methods degrades
in such cases.
To ensure the robustness of TTA against noisy samples, an intuitive solution might be screening out
noisy samples from the test stream, Out-of-distributin (OOD) detection [10, 11, 18, 19, 20, 21, 24, 25, 43] a representative method for this, as it tries detect whether sample is drawn from the same distribution the training data or not. Similarly, open-set domain adaptation (OSDA) [30, 35] and universal domain adaptation (UDA) [34, 42] generalize the adaptation scenario by assuming that unknown classes are present in test data that are not in training data. However, these methods require access to a whole batch of training data and unlabeled target data, which do not often comply with TTA settings where the model has no access to train data at test time due privacy issues [38] and storing a large batch of data is often infeasible due to resource constraints [12]. Therefore, how to make online TTA robust under practical noisy settings is still an open question.
In this paper, we propose Screening-ou samples. SoTTA achieves robustness to noisy samples in two perspectives: (i) input-wise robustness and (ii) parameter-wise robustness. Input-wise robustness aims to filter out noisy samples so that model will be trained only with benign samples. We achieve this goal via High-confidence Uniform- class Sampling (HUS) that avoids selecting noisy samples when updating the model (Section 3) Parameter- wise robustness pursues updating the model weights in way that prevents model drifting due to large gradients caused by noisy samples. We achieve this via shhropy-sharpness minimization (ESM) that makes the loss landscape smoother and parameters resilient to weight perturbation caused by noisy samples (Section 3.2).
Adaptation (SoTTA) that is robust to noisy
We evaluate SoTTA with three common TTA benchmarks (CIFARIO-C, CIFAR 100-C, and ImageNet- [9]) under four noisy scenarios with different levels of distributional shifts: Near, Far, Attack, and Noise (Section 2). We compare SoTTA with eight state-of-the-art TTA algorithms [1, 17, 27, 28, 29, 38, 39, 44], including the latest studies that address temporal distribution changes in TTA [1, 28, 29, 39, 44]. SoTTA showed its effectiveness with the presence of noisy samples. For instance, in CIFARIO-C, SoTTA achieved 80.0% accuracy under the strongest shift case (Noisc), which is 22.3%p improvement via TTA and 6.4%p better than the best baseline [44]. In addition. SoTTA achieves comparable performance to sate--f-theeaal TTA algorithms without noisy samples, e.g., showing 82.2% accuracy when the best baseline's 's accuracy is 82.4% in CIFARIO-
Contributions. (i) We highlight that test sample diversity in real- world scenarios an important problem but has not been investigated yet in the literature. We found that most existing TTA algorithms undergo significant performance degradation with sample diversity. (ii) As solution, we propose SoTTA that is robust to noisy samples by achieving input-wise and parameter-wis wise robustness. (iii) Our evaluation with three TTA benchmarks (CIFAR 10-C, CIFAR100-C and ImageNet-C) show that SoTTA outperforms the existing basclines.

---

(a) Benign.
(b) Near.
(c) Far.
(d) Attack.
(e) Noise.
Figure 3: Five test sample scenarios considered in this work: Benign, Near, Far, Attack, and Noise.
Online data stream
(a) Input-wise robustness
(b) Parameter-wise robustness
Z
Benign sample
Cat
Update
dog
Dog
Noiey sample
Bird
dog bird
Figure 4: Overview of SoTTA. SoTTA achieves input-wise robustness via high-confidence uniform- class sampling (HUS) and parameter-wise robustness via entropy Satropy-sharpness minimization (ESM).
2 Preliminaries
Test-time adaptation. Let Ds (xS .3'} be source data and (Xi, E S x 3 be each instance and the label pair that follows a probability distribution
the source data (x y). Similarly, let DT (X' ./}} be target data and (x,. Mj) ) X² x V be each target instance and the label pair following a target probability distribution (x y). where Y is usually unknown to the learning algorithm. The covariate shift assumption [31] is given between source and target data distributions, which is defined as (x) * Px() and Ps(y(x) = Py (y(x). Given an off off-th--shelf model f(; θ) pre-trained from Ds. the (fully) test-time adaptation (TTA) [38] aims to adapt f(; ①) to the target distribution Py utilizing only xj given test time.
Noisy test samples. We define noivy test samples represent any samples that are not included in the target data distribution, i.e., & É XT, We use the term noisy to distinguish from cittof-distribution (OOD), TTA typically aims to adapt to OOD samples, such corrupted ones. Theoretically, there could be numerous categorics of noisy samples. In this study, we consider five scenarios: Benign, Near, Far2, Attack, and Noise. Figure 3 shows examples of these scenarios. Benign is the typical setting of TTA studies without noisy samples, Near represents a semantic shift [41] from target distribution, Far is severer shift where covariate shift is evident [41], Attack refers to intelligently generated adversarial attack with perturbation [40], and Noise refers to random noise. We focus on these scenarios to understand the impact of noisy samples in TTA as well the simplicity of analysis. Detailed settings are described in Section 4.
Methodology
Problem and challenges. Prior TTA methods assume test samples are benign and blindly adapt to incoming batches of test samples. The presence of noisy samples during test time can significantly degrade their performance for those TTA algorithms, which has not been explored in the literature yet. Dealing with noisy samples in TTA scenarios is particularly challenging access to the source data, (ii) no labels are given for target test data, and (iii) the model is continually adapted and thus desirable solution should apply varying models. This makes it difficult apply existing solutions that deal with a similar problem. For instance, out-of cottof-distributinn (OOD) detection studies [11, 19, 20, 43] are built on the assumption that model is fixed test time, and open-set domain adaptation (OSDA) methods [30, 35, 42] require labeled source and unlabeled target data for training.
(i) TTA has
We borrowed the term Near and Far from the OOD detection benchmark [41].

---

Methodology overview. To address the problem, we propose Screening Test-Time Adaptation (SoTTA), whose overview described Figure 4. SoTTA achieves robustness to noisy samples in two perspectives: (i) inpul-wise robustness via high-confidence uniform-class sampling that avoids selecting noisy samples when updating the model (Section 3.1), and (ii) parameter-wise robustness via antropy-sharpme minimization that makes parameters resilient to weight perturbation caused by noisy samples (Section 3.2).
3.1 Input-wise robustness via high-confidence uniform-class sampling
Our first approach is to ensure input- wise robustness noisy samples by filtering out them when selecting sam- ples for adaptation. As locating noisy samples without their labels is chal- lenging, our idea based on the em- pirical observation of the model pre- dictions with respect noisy samples.
20.0
2123456760
Aracc
hous
10.0
(a) Confidence distribution.
(b) Predicted class distribution.
Observation. Our hypothesis is that noisy samples have distinguished properties from benign samples due to the distributional shifts, and this could be observable via the models' prediction outputs. We investigate two types of features that work proxies for identifying benign samples: (i) confidence of the samples and (ii) predicted class distributions. Specifically, we compared the distribution of the softmax confidence (Figure 5a) and the predicted class distribution (Figure 5b) of benign samples with noisy samples. First, the confidence of the samples is relatively lower than that of benign samples. The more severe the distribution shift, the lower the confidence (e.g.. Far is less confidence than Near), which is also in line with findings of previous studies that pre-trained models show higher confidence on target distribution than out-of til-o--distributtoon data [10, 21]. Second, we found that noisy samples are often skewed terms of predictions, and this phenomenon is prominent in more severe shifts (e.g., Noise), except for Attack, whose objective make the model fail to correctly classify. These skewed distributions could lead to an undesirable bias in p(y) and thus might negatively impact the TTA objective, such entropy minimization [38].
Figure Model predictions on benign (CIFARIO-C) and noisy samples.
Algorithm Hfigh-ccofidence Uniform-class Sampling (HUS)
Input: test data stream Xt, memory M with capacity N for test time & E (1,, (1."".,7} do
Dr. x- f(x; θ)
if C(x θ) > Co then
Sampling confident data
[M| < N then
Add (x, Mt) to M
else
y" the most prevalent class(es) in M
if Úe $ y then
b Balancing classes
Randomly discard (x;, ū:) from M where Ú; )
else
Randomly discard (x; 0:) from M where ú; Úr
Add (Xe, Mc) to M
Solution. Based the aforementioned empirical analysis, we propose High-confidene Uniform- class Sampling (HUS) that avoids using noisy samples for adaptation by utilizing a small memory. We maintain confident samples while balancing their predicted classes in the memory. The selected samples in the memory are then used for adaptation. We describe the procedure as a pseudo-code code in Algorithm 1. Given target test sample x, HUS measures its confidence. Specifically, define the confidence C(x; θ) of each test sample x as:
C(x; +) = max
where 9 = fxx θ).
(1)
i=I..n
+== eioy

---

We store the sample if its confidence is higher than the predefined threshold Co. In this way, we maintain only high-confidence samples in the memory used for adaptation and thus reduce the impact of low-confidence noisy samples.
Furthermore, while storing data in the memory, we balance classes among them. Specifically, if the predicted class of the current test sample is not in the most prevalent class in the memory, then HUS randomly replaces one random sample in the most prevalent class with the new sample. Otherwise, if the current sample belongs to the most prevalent class in the memory, HUS replaces one random sample the same class with the current one. We can maintain classes uniformly with this strategy. which is effective for not only filtering out noisy samples but also removing class biases among samples when used for adaptation, which we found is beneficial for TTA. We found these two memory management strategies not only effectively reduce the impact of noisy samples for adaptation but also improve the model performance in benign-only cases by avoiding model drifting due to biased and low-confidence samples (Section 4).
With the stored samples in the memory, we update the normalization statistics and affine parameters in batch normalization (BN) [13] layers, following the prior TTA methods [29, 38, 44]. This is known as not only computationally efficient but also showed comparable performance improvement to updating the whole layers. While we aim to avoid using noisy samples for adaptation, a few noisy samples could still be stored the memory, e.g., when they are similar to benign samples or outliers. To be robust to temporal variances of the samples in the memory. we take the exponential moving average to update the BN statisties (means μ and variances o²) instead of directly using the statisties from the samples in the memory. Specifically, we update the means and variances of BN layers as: (i) م = (1 m) m) + my, and (ii) o? = (1 m) + maf, where m [0, 1] is a momentum hyperparameter. We describe updating the affine parameters in Section 3.2.
3.2 Parameter-wise robustness via entropy-sharpness minimization
Our second approach is to secure parameter-wise robustness noisy samples by training the model in a way that is robust to noisy samples. Our idea is based on the observation that the parameter update often cor- rupted with noisy samples, and this
10"
100
110"
to"
Gradient Norm
Accuracy (%)
16
Gradlent Nom
Accuracy
80
to
10
40
10
40
10"
20
10
10000
20000
10000
80000
10"
could be mitigated by smoothing the loss landscape with respect to param- eters.
Test samples time order
Test samples time order
(a) Without ESM
(b) With ESM.
Figure 6: Cumulative accuracy (%) of benign samples (CIFAR10-C) and gradient norm of noisy samples (Noise) as the adaptation proceeds. The dotted line refers to the source model accuracy.
Observation. While most existing
TTA algorithms utilize the entropy minimization loss [5, 38, 44], it could drift the model with high gradient samples [29]. We observed that adaptation with noisy samples often hinders the model from adapting to benign samples. Figure 6 shows an example of this phenomenon. Specifically, test samples consist of 10k benign samples and 10k noisy samples (Noise), which are randomly shuffled. We computed the cumulative accuracy for the benign test data and the gradient norm of noisy samples at each step. As shown in Figure 6a, the gradient norm of noisy samples dropped rapidly, indicating that the model is gradually adapting to these noise data. This leads to a significant accuracy drop for benign samples. The key question here is how to prevent the model from overfitting noisy samples. Figure 6b shows the result with entropy- sharppe-s minimization (ESM) that we explain in the following paragraph. With ESM, the gradient norm of noise data remains high, and the accuracy for benign samples improved after adaptation as intended.
Solution. To make the model parameters robust to adaptation with noisy samples, the entropy loss landscape should smoother so that the model becomes resilient to unexpected model drift due to noisy samples. To that end, we jointly minimize the naive entropy loss and the sharpness of the entropy loss and thus make the loss landscape robust to model weight perturbations by large gradicnts from noisy samples. Specifically, replace the naive entropy minimization E with the
5

---

entropy minimization (ESM) as:
min Es(x, θ min max E(x; θ c),
(2)
IelaaSp
where the entropy-sharpen Es (x, +) is defined as the maximum objective around the weight per- turbation with 1.2-norm constraint p. To tackle this joint optimization problem, we follow sharpness- aware minimization [4] similar to [29]. which originally aims to improve the generalizability of models over standard optimization algorithms such as stochastic gradient descent (SGD). We repur- pose this optimization to make the model robust to noisy samples in TTA scenarios.
Specifically, assuming p 1, the optimization can be approximated via first-order Taylor expan- sion:
(() ف arg max Exx: θ + c) m Vexx(x
The solution for this approximation problem is given by the classical dual norm problem:
((@) sign VV O)) V [()(==))1=+ (IVeF((x))))) 1/p
(4)
where 1/p 1/q 1. We set 2 for further implementation, following the suggestion [4]. By substituting ē(@) to the original entropy-sharpness minimization problem, the final gradient approximation is:
Ve θ) ~ Ve Exx, 0)(e+x())-
(5)
In summary, we calculate the catropy-sharppnn minimization objective via two steps. First, at time step f, it calculates the é(O;) with previous parameters and entropy loss. It generates the temporal model with the new parameters: θ, + ).))). Second, based on the temporary model. the second step updates the original model's parameters with the approximation in Equation 5. By putting it together with HUS (Section 3.1). parameters are updated by:
θ O.- V E(x, +) xuS (0) 0=0r-t
(6)
where 1j is the step size and to is the model adaptation interval. For simplicity, we set the model adaptation interval the same the memory capacity, i.e., to = N. In the experiments, we use N = 64, which is one of the most widely-used batch sizes in TTA methods [36, 38, 44).
Experiments
This section describes our experimental setup and demonstrates the results in various settings. Please refer to Appendix A and B for further details.
Scenario. To mimic the presence of noisy samples in addition to the original target test samples, injected various noisy datasets into target datasets and randomly shuffled them, which we detail the following paragraphs. We report the classification accuracy of the original target samples to measure the performance of the model in the presence of noisy samples. We ran all experiments with three random seeds (0, 1, 2) and reported the average accuracy and standard deviations in Appendix B.
Target datasets. We used three standard TTA benchmarks: CIFARIO-C, CIFAR100-C, and ImageNet-C [9] as our target datasets. All datasets contain 15 different types and five levels of corruption, where we use the most severe corruption level of five. For each corruption type, the CIFAR10-C/CIIC
100-C dataset has 10,000 test data with 10/100 classes, and the ImageNet-C dataset has 50,000 test data with 1000 classes. We use ResNet18 [8] as the backbone network. We pre-trained the model for CIFAR10 with training data and used the TorchVisia [23] pre-trained model for ImageNet.
Noisy datasets. Besides the target datasets (Benign) mentioned above, we consider four noisy scenarios (Figure 3): CIFAR100 [15]/mmaeeNet [3] (Near), MNIST [26] (Far), adversarial attack (Attack), and uniform random noise (Noise). As both CIFARIO 10-C and ImageNet-C have real-world images for object recognition tasks, CIFARI00 would be a near dataset to them. For CIFARI 10--C, we select ImageNet [3] for near dataset. We select MNIST as far dataset because they are not real-world images. We referred to the OOD benchmark [41] for choosing the term Near and Far and the datasets. For the adversarial attack, we adopt the Distribution Invading Attack (DIA), which is an

---

Table Classification accuracy (%) on CIFARIO-C for 15 types of corruptions under five scenarios: Benign, Near, Far, Attack, and Noise. Benign contains only benign target samples, while other scenarios include both benign and each type of noisy samples specified. Bold numbers are the highest accuracy. Averaged over three different random seeds.
Nosse
Blur
Weather
Digital
Mathod
Cooa
Shot
Imp.
Def.
Gila.
Zoom
Snow
Fro.
Fog
Brit.
Cont.
Elas.
Pix.
JPEG
Avg.
Seurce
26.0
35.2
24.7
56.7
52.0
67..4
64.5
78.0
67.0
74.1
91.5
33.0
76.6
73.2
57.7
BN Stats (27)
67.0
69.0
60.4
87.8
65.6
86.3
87.4
81.6
80.3
85.4
90.7
86.9
76.7
79.3
71.9
78.1
[17]
71.1
72.9
62.2
86.9
64.4
65.3
86.6
80.8
78.8
84.9
99.6
84.0
76.2
80.0
73.1
78.5
TENT (28)
74.5
77.6
66.6
86.2
66.2
86.9
85.8
83.7
81,3
86.0
91.1
86.9
77.9
K2.7
76.7
81.0
Reniga
LAME [1]
CoTTA [30]
21.8
76.9
78.6
72.3
88.2
53.3
70.9
52.1
86.8
SR.1
79.2
83,4
83.4
86.1
91.2
28.0
84-9
79.2
83.0
74.1
79.9
82.2
55.9
EADA [28]
76.0
78.2
68.2
88.4
70.1
81.4
38.4
BAL5
85.0
88.0
91.5
89.9
77.8
$4.8
78.4
82.4
SAR (29)
68.3
69.7
58.9
87,8
62.9
86.3
87.4
81.6
80.3
85.4
90.77
86.9
76.7
79.3
72.0
78.3
RoTTA [44]
65.2
87.2
81.3
76.9
57.3
75.5
SoTTA
75.0
77.5
68.8
88.8
20.7
87.5
89.0
85.4
84.0
88.2
91.9
83.9
79.8
83.9
78.3
82.2
Source
26.0
33.2
24.7
56.7
52.0
67.4
64.8
78.0
67.0
91.5
33.9
76.6
73.2
57.3
RN Stats [27]
64.9
66.3
58.2
84.1
62.7
84.2
85.0
85.1
82.5
85.5
92.4
85.3
76..5
67.4
70.3
76.6
PL
63.2
63.5
51.7
81.6
58.4
78.4
85.3
79.2
79.7
8013
8912
79.5
73.1
70.9
69.3
73.4
HENE [28]
64.7
81.3
59.6
83.8
79.6
78.3
88.6
83.7
73.5
74.3
Near
CoTTA [29]
LAME [1]
72.7
24.3
74.3
31.6
66.0
19.9
82.6
53.9
55.2
67.66
81.8
65-9
62.5
84.1
79.0
84.1
69.5
85.5
K2.5
75.1
91.1
901.1
69.9
28.4
78.1
75.0
76.1
74.2
79.3
78.4
56.4
EADA [28]
51.5
50.3
40.1
70.4
45.8
72.8
77.2
66.8
67.4
74.4
83.9
68.2
6002
67.3
62.1
63399
SAR [29]
59,0
78.2
55.4
78.8
7866
85.3
66.6
64.0
72.4
RoTEA (44)
66.3
68.3
39.4
86.0
65.2
85.4
87.0
83.5
82.8
86.4
92.4
84.3
77.1
71.6
71.6
T7.3
SOTTA
74.3
76.7
66.5
87.5
66.9
N6.4
87.88
84.4
85.8
N7.2
91.3
K8.4
76.7
N2.4
78,0
$1.4
Source
26.0
33.2
24.7
56.7
52.0
67.4
64.8
78.0
67.0
74.1
91.5
33.9
76.6
46.4
73.2
57.7
RN Stots [27]
62.7
66.1
56.0
86.5
60.7
84.2
87.2
79.6
78.4
85.7
80.9
80.2
75.5
69.8
65.0
75.2
[17]
55.2
$6.1
48.3
63.2
19.3
80.0
83.0
76.8
73.3
80.6
85.66
74.6
70.5
66.8
63.6
69.8
TENT [18]
$1.6
84.8
43.5
85.3
80.4
80.5
73.5
71.2
Far
OsTTA [39)
LAME [1]
22.8
67.4
29.6
71.1
59.4
19.3
83.3
53.2
61.2
50.4
62.3
64.6
84.3
60.7
60.4
79.1
67.9
60.4
K3.8
72.8
900.1
67.22
28.1
55.0
74.7
76.0
44.4
70.3
74.3
72.9
55.5
74.5
EATA [25]
40,0
46.4
34.9
73.5
35.2
59.3
76.4
61.1
59.5
68.4
85.2
55.2
46.5
53.5
49.0
56.3
SAR (29)
60.3
62.6
50.9
86.5
55.4
79.8
78.3
89.9
80.2
70.1
60.9
73.3
RoTTA [44]
67.3
69.8
61.]
88.1
66.0
86.6
S8.0
82.0
78.8
86.2
91.2
61.7
77.5
79.6
73.0
77.1
SOTTA
73.3
76.3
66.3
88.5
68.3
W6.8
8H.3
84.1
84.2
N7.2
92.0
89.0
77.8
K3J8
77.8
$1.6
Source
26.0
33.2
24.7
56.7
52.0
67.4
64.8
78.0
67.0
74.1
91.5
33.9
7666
46.4
73.2
57.7
BN Stats [27]
44.5
46.8
39.8
63.3
42.1
63.5
62.5
62.3
593
62.1
75.4
65.1
50.6
53.9
46.9
55.9
PL.
75.1
71.7
72.1
60.9
66.3
TENT [38]
62.9
65.2
56.7
74.8
54.5
75.4
75.2
73.5
71.8
72.1
83.0
73.7
63.1
70L0
64.0
68.9
Antock
LAME [3]
21.0
28.0
17.3
52.7
52.9
65.9
62.0
79.3
70.4
73.9
900.
28.4
75.6
44.2
74.2
55.9
CoTTA [39)
53.1
57.0
49.7
67.9
55.9
69.7
71.3
74.3
69.5
68.1
84.9
47.0
68.6
62.3
71.6
69.5
EATA [28]
66.6
75.2
57.2
14.7
75.1
73.5
83.5
76.0
64.3
70.9
SAR 129]
46.1
65.3
42.2
63.55
62.5
62.3
59.3
75.4
63.1
50.6
56.2
RoTTA [44]
69.7
71.8
62.9
88.6
67.8
87.4
88.7
85.2
80.1
87.1
91.8
63.55
78.4
80.6
74.1
78.4
SOTTA
78.2
80.8
72.3
90.1
73.6
89.2
901.
87.4
86.3
89.3
92.9
87.88
$1.3
86.6
$1.0
84.5
Source
26.0
33.2
24.7
$6.7
$2.0
67.4
64.8
78.0
67.00
74.1
91.5
33.9
76.6
46.1
73.2
57.7
BN Stats [27)
51.7
$3.9
45.5
52.7
41.5
51.0
55.1
62.8
63.8
53.8
76.9
55.8
46.8
54.8
56.4
54.8
[77
47.6
48.9
49.4
$4.1
61.9
56.5
45.2
$18
TENT (38)
$4,0
57.1
36.7
48.9
28.3
$0.5
$1.0
64.0
64.7
49.5
80.5
43.7
38.4
56.7
57.0
52.1
LAME [1]
21.8
28.6
18.5
$1.6
$0.8
64.3
60.9
78.4
67.3
71.7
90.5
27.0
75.1
43.0
73.4
54.9
Noss
CoTTA [39]
60.4
60.3
57.4
47.3
41.6
44.1
52.0
62.7
66.6
47.7
79.0
44.7
42.8
6012
601
548
[28]
12.2
32.7
25.0
42.6
20.1
27.5
36.0
SAR (29)
57.5
59.3
49.6
57.2
43.7
54.4
59.4
64.8
65.4
57.9
77.1
60.3
50.0
58.1
59.8
58.1
RoTTA [44]
64.4
66.9
$6.1
80.1
59.1
79.8
82.2
79.7
78.7
71.8
91.2
69.0
72.3
73.4
72.8
73.6
SoTTA
73,3
77.7
66.8
86.1
64.0
84.3
86.6
83.1
82,0
85.7
91.1
84.1
77.1
81.6
77.2
80.0
adversarial attack algorithm designed for TTA [40]. We inject the small perturbations to malicious samples to increase the overall error rate on benign data in the same batch. For the uniform random noise, we generate pixel-wise uniform-random images with the same size as the target images. We set the number of noisy samples equal to each target dataset as default, and we also investigated the impact of the number of noisy samples in the following ablation study (Figure 7). In cases where the number of noisy samples is different from the target datasets, we randomly resampled them. To ensure that the learning algorithm does not know the information of noisy samples beforehand, the pixel values of all noisy images are normalized with respect to the target dataset.
Baselines. We consider various state-of-the thee-- art TTA methods basclines. Source evaluates the pre-trained model directly on the target data without adaptation. Test-time batch normalization (BN stats) [27] updates the BN statistics from the test batch. Pseudo-Label (PL) [17] optimizes the trainable BN parameters via pseudo labels. Test entropy minimization (TENT) (38) updates the BN parameters via entropy minimization.
We also consider the latest TTA algorithms that improve robustness to temporal distribution changes in test streams to understand their performance to noisy samples. Laplacian adjusted maximum- likelihood estimation (LAME) [1] modifies the classifier output probability without modifying model internal parameters. Continual test-time adaptation (CoTTA) [39] uses weight-averaged and
7

---

Table 2:
Classification accuracy (%) on CIFARI00-C. Bold numbers are the highest accuracy. Averaged over three different random seeds for 15 types of corruption.
Method
Benign
Near
Far
Attack
Noise
Avg.
Source
33.2
33.2
33.2
33.2
33.2
33.2
Stats [27]
53.7 202
50.8 10.1
46.8 1.11
29.2 204
28.3
41.8
PL [17]
56.6 2.2
48.0
42.8 10.7
39.0
23.8 206
42.1 20.
TENT [38]
59.5 20.0
16.
40.0
31.9
20.0
39.5
LAME [1]
31.0 0.5
31.5 0.5
30.8 +0.7
31.0
+07
31.1 +06
CoTTA [39]
55.8
50.0
42.4
37.2
27.3
42,6
EATA [28]
23.5 1.9
6.1 0,3
4.8 +0.5
0.
2.4 002
8.1 0.3
SAR [29]
57.3 20.
a1.4
51.2
34.4
38.1
47.
RoTTA [44]
48.7
49.4
49.8
51.5 204
48
49.6
SoTTA
60.5
57.1
59.0
61.9
58.6
59.4
Table 3: Classification accuracy (%) on ImageNet-C. Bold numbers are the highest accuracy. Averaged over three different random seeds for 15 types of corruption.
Method
Benign
Near
Far
Attack
Noise
Avg.
Source
14.6 2.0
14.6 2.0
14.6 1.00
14.6 100
14.6 00
14.6 200
BN Stats [27]
27.1 2000
18.9
14.8 1.00
17.4 108
12.8 000
18.2
PL [17]
30.5 2..
6.9
5.2
18.1
3,4 106
12.8 202
TENT [38]
27.1
18.9
14.8
17.4
12.8
18.2
LAME [1]
14.4
x0.1
+4.4
14.0
14.3
14.3
CoTTA [39]
32.2 =.1
23.3 +02
17.6
28.3 +13
16.0
23.4
EATA [28)
38.0
25.6
23.1
26.1 26.1
20.7
26.7
SAR [29]
36.1 20.1
27.6
23.5
26.8
22.0
27.2
RoTTA [44)
29.7 20.0
25.6
29.2
32.0
31.2
29.5
SoTTA
39.8 8.00
27.9 a..
36.1 a..
41.1 a.1
39.0
36.8 a.0
augmentation--vvragee predictions and avoids catastrophic forgetting by stochastically restoring part of the model. Efficient anti-forgetting test-time adaptation (EATA) [28] uses entropy and diversity weight with Fisher regularization to prevent forgetting. Sharpness-awant and reliable optimization (SAR) [29] removes high-entrope samples and optimizes entropy with sharpness minimization [4]. Robust test-time adaptation (RoTTA) [44] utilizes the teacher-student model stabilize while selecting the data with sategory-balancee sampling with timeliness and uncertainty.
Hyperparameters. We adopt the hyperparameters of the baselines from the original paper official codes. We use the test batch size of 64 in all methods for a fair comparison. Accordingly, we set the memory size 64 and adapted the model for every 64 samples for our method and RoTTA [44]. We conduct TTA in an online manner. We used a fixed hyperparameter of BN momentum m 0.2 and updated the BN affine parameters via the Adam optimizer [14] with fixed learning rate of I 0.001 and a single adaptation epoch. The confidence threshold Co is set to 0.99 for CIFARIO-C. 0.66 for CIFARI00- and 0.33 for ImageNet-C. We set the sharpness threshold p 0.05 as previous works [4, 29J. We specify further details of the hyperparameters in Appendix A.
Overall result. Table shows the result CIFARI0-C for 15 types of corruptions under five noisy scenarios described in Figure 3. We observed significant performance degradation in most of the baselines under the noisy settings. In addition, the extent of accuracy degradation from the Benign case is more prominent in more severe shift cases (e.g., the degree degradation is generally Near < Far Attack Noise). Popular TTA basclines (BN stats, PL, and TENT) might fail due to updating with noisy samples. State-of-the-art TTA basclines that address temporal distribution shifts in TTA (LAME, CoTTA, EATA. SAR, and RoTTA) still suffered from noisy scenarios, as they are not designed to deal with unexpected samples. On the other hand, SoTTA showed its robustness across different scenarios. This validates the effectiveness of our approaches ensure both input- wise and parameter-wise robustness. Interestingly, we found that SoTTA showed comparable performance to -fate-of-the-arr baselines our high-confident uniform-class sampling strategy filters not only noisy samples but also benign samples that would negatively impact the algorithm's objective. This implies that there exist samples that more beneficial for adaptation, which aligns with the findings that high-entropy samples
Benign case as well. Our interpretation of this result is two-fold. First,

---

Table 4: Classification accuracy (%) and corresponding standard deviation of varying ablative settings in SoTTA on CIFARIO-4 1O-C. Bold numbers are the highest accuracy. Averaged over three different random seeds for 15 types corruption.
Method
Benign
Near
Far
Attack
Noise
Avg.
Source
57.7 10
57.7 1D
57.7 1.0
57.7 1
57.7 1o
57.7 + 1.
HC
34.9 448
13.6 +03
=
16.9 1.6
16.8 +0.
20.0 2.
UC
66,4 130
62.1 408
56.5
70.0
59.5
62.9
HC UC (HUS)
111
61.7
28.4
40.9 25.5
58.9 226
47.9 2:8
ESM
82.6 :02
77.9.00
72.8
83.4
60,5
75.4
HC + ESM
82.3 +02
80.9 #06
74,9 +224
83.5 +02
68.7 700
78.0 20
UC+ ESM
82.2 0.2
78.0
75.9 -0.
84.3 +01
77.7 +0.7
79.6 +22
HUS + ESM (SoTTA)
82.2
81.4
81.6
84.5 6.22
80.0 1.4
81.9 a.5
harm adaptation performance [29]. Second, entropy-sharpness minimization helps ensure both the robustness to noisy samples and the generalizability of the model by preventing model drifts from large gradients, leading to performance improvement with benign samples. We found similar patterns for the CIFAR CUFAR100-C (Table 2) and ImageNet-C (Table 3). More details are in Appendix B.
Impact of individual components of SOTTA We conducted ablative study further investigate the effectiveness of SoTTA's individual components. Table 4 shows the result of the ablation study for CIFARIO-C. For input-wise robustness, HC refers high-confidence sampling, and UC refers to uniform-class sampling. The two strategies are integrated into our high-confidence uniform-class sampling (HUS). ESM is our entropy-sharpine
minimization for parameter-we wise robustness. Note that we utilized FIFO memory with the same size for the UC case without HC and the native entropy minimization where we did not utilize ESM. Overall, the accuracy is improved as we sequentially added each approach of SoTTA. This validates our claim that ensuring both input-wiss and parameter- wise robustness via HUS and ESM a synergetic strategy to combat noisy samples in TTA.
Impact of the number of noisy samples. We also in- vestigate the effect of the size of noisy samples TTA algorithms. Specifically, for the CIFARIO-C dataset with the Noise case, we varied the noise samples from 5,000 (5k) to 20,000 (20k) while fixing the size of be- nign samples as 10,000. Figure shows the result. We observe that the accuracy of most baselines tends to deteriorate with a larger number of noisy samples and is sometimes even worse than without adaptation (Source). SoTTA showed its resilience to the size of noisy sam- ples with 1.9%p degradation from 5k to 20k samples. RoTTA also showed its robustness to noisy samples some extent, but the performance gain is around 6.4%p lower than SOTTA
Classification acturacy (%)
80
Source
5k
10k
20k
60
40
120
otats
Pk
TENT
LAME
COTTA
EATA
64R
ROTTA
SoTTA
Figure 7: Classification accuracy (%) vary-
ing the size of noisy samples CIFAR10-
C under Noise.
5 Related work
Test-time adaptation. While test-time adaptation (TTA) attempts to optimize model parameters with unlabeled test data streams, it lacks consideration of spoiling of the sample itself; i.e., it inevitably adapts to potential outlier samples mixed in the stream, such as adversarial data or mere noise. Most existing TTA algorithms directly optimize the model with incoming sample data. Test-time normalization [27, 36] updates the batch norm statistics in test-time to minimize the expected loss. TENT [38] minimizes the entropy of the model's predictions on a batch of test data. On the one hand, recent studies promote the robustness of the model, yet the consideration is limited to temporal distribution shifts of test data [1, 4, 5, 28, 29, 39, 44]. For instance, CoTTA [39] tries to adapt the model in continually changing target environment via self-training and stochastical restoring. reservoir
NOTE [5] proposes instance-aware batch normalization combined with prediction-bllance sampling to ensure model robustness towards temporally correlated data stream, which requires retraining with source data. We provide a method- wise comparison with EATA [28], SAR [29]. and RoTTA [44] in Appendix D.2. To conclude, while existing TTA methods seek the robustness of the

---

model to temporal distribution shifts, they do not consider scenarios where noisy samples appear in test streams.
Out-of-distributi detection. Out-of- cot-of-distribution (OOD) detection [10, 11, 18, 19, 20, 21, 24, 25, 43] aims to ensure the robustness of the model by identifying when given data falls outside the training distribution. A representative method is a thresholding approach [10, 19, 20, 21], that defines scoring function given an imput and pretrained classifier. The sample is detected as OOD data if the output of the scoring function is higher than a threshold. Importantly, OOD detection studies are built on the condition that training and test domains are the same, which differs from TTA's scenario. Furthermore, OOD detection assumes that model is fixed during test time, while model changes continually in TTA. These collectively make it difficult apply OOD detection studies directly to TTA scenarios.
Open-set domain adaptation. Open-set Domain Adaptation (OSDA) assumes that target domain contains unknown classes not discovered in a source domain [30, 35] in domain adaptation scenarios. These methods aim to learn a mapping function between the source and target domains. While the target scenario that a model could encounter unknown classes of data in the test time is similar to our objective, these methods do fit into TTA it assumes both labeled source and unlabeled target data are available in the training time. Universal domain adaptation (UDA) [34, 42] further generalizes the assumption of OSDA by allowing unknown classes to present in both the source and the target domains. However, the same problem still remains as it requires labeled source and unlabeled target data in training time, which do not often comply with TTA settings where the model has no access to train data at test time due to privacy issues [38].
6 Discussion and conclusion
We investigate the problem of having noisy samples and the performance degradations caused by those samples in existing TTA methods. To address this issue, we propose SoTTA that is robust to noisy samples by high-confidence uniform-class sampling and shtrops-sharppess minimization. Our evaluation with four noisy scenarios reveals that SoTTA outperforms state-of-the-ard TTA methods in those scenarios. We believe that the takeaways from this study are meaningful stride towards practical advances in overcoming domain shifts in test time.
Limitations and future work. While we focus on four noisy test stream scenarios, real-world test streams might have other types of sample diversities that are not considered in this work. Furthermore, recent TTA algorithms consider various temporal distribution shifts, such as temporally-correann streams [5] and domain changes [39]. Towards developing TTA algorithm robust any test streams in the wild, more comprehensive and realistic considerations should be taken into account, which we believe is meaningful future direction.
Potential negative societal impacts. As TTA requires continual computations for every test sample for adaptation, environmental concerns might be raised such carbon emissions [37]. Recent studies such as memory-economil TTA [12] might be an effective way to mitigate this problem.
Acknowledgments and Disclosure of Funding
This work was supported by the Institute of Information & communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) Voice Phishing Call Detection).
On-Device
10

---

References
[1] Malik Boudiaf, Romain Mueller, Ismail Ben Ayed, and Luca Bertinetto. Parameter-free online test-time adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 8344-8353, June 2022.
[2] Dhanajit Brahma and Piyush Rai. A probabilistic framework for lifelong test-time adaptation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3582-3591, June 2023.
[3] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- scale hierarchical image database. In 2009 IEEE Conference Computer Vision and Patlern Recognition, pages 248-255, 2009.
[4] Pierre Foret, Ariel Kleiner, Hossein Mobahi, and Behnam Neyshabur. Sharpness- aware min- imization for efficiently improving generalization. In International Conference on Learning Representations, 2021.
[5] Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. NOTE: Robust continual test-time adaptation against temporal correlation. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances Neural Information Processing Systems, 2022.
[6] Ian Goodfellow, Jean Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Commun. ACM, 63(1 (5(11):1391144 144, oct 2020.
Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil
[7] Yves Grandvalet and Yoshua Bengio. Semi-supervised learning by entropy minimization. In L. Saul, Y. Weiss, and L. Bottou, editors, Advances in Neural Information Processing Systems, volume 17. MIT Press, 2004.
[8] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for im- age recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[9] Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. In International Conference on Learning Representations, 2019.
[10] Dan Hendrycks and Kevin Gimpel. baseline for detecting misclassified and duitii--ist examples in neural networks. In International Conference on Learning Representations, 2017.
[11] Dan Hendrycks, Mantas Mazeika, and Thomas Dietterich. Deep anomaly detection with outlier exposure. Proceedings of the International Conference on Learning Representations, 2019.
[12] Junyuan Hong, Lingjuan Lyu, Jiayu Zhou, and Michael Spranger. MECTA: Memory-economic continual test-time model adaptation. In The Eleventh International Conference on Learning Representations, 2023.
[13] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In Francis Bach and David Blei, editors, Proceedings of the 32nd International Conference on Machine Learning, volume 37 of Proceedings of Machine Learning Research, pages 448 456, Lille, France, 07-09 Jul 2015. PMLR.
[14] Diederick P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In Interna- tional Conference on Learning Representations (ICLR), 2015.
[15] A. Krizhevsky and G. Hinton. Learning multiple layers of features from tiny images. Master's thesis, Department of Computer Science, University of Toronto, 2009.
[16] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):227)-22222 1998.
[17] Dong-Hyun Lee. Pseudo-label: The simple and efficient supervipe learning method for deep neural networks. In Workshop on challenges representation learning, ICMI., page 896, 2013.
11

---

[18] Kimin Lee, Honglak Lee, Kibok Lee, and Jinwoo Shin. Training confidence-calbraan clas- sifters for detecting -uttt--distribttionn samples. In International Conference on Learning Representations, 2018.
[19] Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. A simple unified framework for detect- ing out-of-distribution samples and adversarial attacks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett, editors, Advances in Neural Information Pro- cessing Systems, volume 31. Curran Associates, Inc., 2018
[20] Shiyu Liang, Yixuan Li, and R. Srikant. Enhancing the reliability of out-of-distribution image detection in neural networks. In International Conference on Learning Representations, 2018.
[21] Weitang Liu, Xiaoyun Wang, John Owens, and Yixuan Li. Energy- out-of-distribution detection. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21475-21475. Curran Associates, Inc., 2020.
[22] Ilya Loshchilov and Frank Hutter. SGDR: Stochastic gradient descent with warm restarts, In International Conference Learning Representations (ICLR), 2017.
[23] Torch maintainers and contributors. Torchvision: Pytorch's ' computer vision library. https://gi thwbbbtttttuuuttte//g// com/ woom/pyoorch/viiiio..o 2016.
[24] Yifei Ming, Ying Fan, and Yixuan Li. POEM: Out-of-distribution detection with posterior sampling. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song. Csaba Szepesvari, Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 15650-15665 PMLR, 17-23 Jul 2022.
[25] Sina Mohseni, Mandar Pitale, JBS Yadawa, and Zhangyang Wang. Self-supervised learning for generalizable out-of-distribution detection. Proceedings of the AAAI Conference Artificial Intelligence, 34(04):5216-522) Apr. 2020.
[26] Norman Mu and Justin Gilmer. Mnist-e: A robustness benchmark for computer vision. arXiv preprint 190X6 02337, 2019.
[27] Zachary Nado, Shreyas Padhy, D Sculley, Alexander D' Amour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction-tine batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963, 2020.
[28] Shuaicheng NIu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari. Gang Niu, and Sivan Sabato, editors, Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 16888-16905. PMLR, 17-23 Jul 2022.
[29] Shuaicheng Niu, Jiaxiang Wu. Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In The Eleventh International Conference on Learning Representations, 2023.
[30] Pau Panareda Busto and Juergen Gall. Open set domain adaptation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), Oct 2017.
[31] Joaquin Quiñooero-Cannee Dataset shift in machine learning. Mit Press, 2008
Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence.
[32] Joseph Redmon, Santosh Divvala, Ross Girshick. and Ali Farhadi. You only look once: Unified, real-time object detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
[33] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Nassir Navab, Joachim Hornegger, William M. Wells, and Alejandro F. Frangi, editors, Medical Image Computing and Conputer-Assisted Intervention MICCAI 2015, pages 234-241, Cham, 2015. Springer International Publishing.
12

---

[34] Kuniaki Saito, Donghyun Kim, Stan Sclaroff, and Kate Saenko. Universal domain adaptation through self supervision. In H. Larochelle. M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 16282-16292. Curran Associates, Inc., 2020.
[35] Kuniaki Saito, Shohei Yamamoto, Yoshitaka Ushiku, and Tatsuya Harada. Open set domain adaptation by backpropagation. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
[36] Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 11539-11551. Curran Associates, Inc., 2020.
[37] Roy Schwartz, Jesse Dodge, Noah A. Smith, and Oren Etzioni. Green ai. Commun. ACM, 63(12):54 63, nov 2020.
[38] Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Representations, 2021.
[39] Qin Wang, Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7201-7211, June 2022.
[40] Tong Wu, Feiran Jia. Xiangyu Qi. Jiachen T. Wang, Vikash Sehwag, Saeed Mahloujifar, and Prateek Mittal. Uncovering adversarial risks of test-time adaptation. arXiv preprint 1rXiv:2301.22576 2023.
[41] Jingkang Yang. Pengyun Wang. Dejian Zou, Zitang Zhou, Kunyuan Ding, WENXUAN PENG, Haoqi Wang, Guangyao Chen, Bo Li, Yiyou Sun, Xuefeng Du, Kaiyang Zhou, Wayne Zhang, Dan Hendrycks, Yixuan Li, and Ziwei Liu. Openood: Benchmarking generalized out-of- distribution detection. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 32598-32611. Curran Associates, Inc., 2022.
[42] Kaichao You, Mingsheng Long, Zhangjie Cao, Jianmin Wang. and Michael I. Jordan. Universal domain adaptation. In 2019 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2715-2724, 2019.
[43] Qing Yu and Kiyoharu Aizawa. Unsupervised out-of-distribution detection by maximum classifier discrepancy. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[44] Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pages 15922-15932, June 2023.
13

---

A
Experiment details
We conducted all experiments in the paper using three random seeds (0, 1, 2) and reported the average accuracies and their corresponding standard deviations. The experiments were performed on NVIDIA GeForce RTX 3090 and NVIDIA TITAN RTX GPUs. For single execution of SoTTA, the test-time adaptation phase consumed I minutes for SEPARIO-CCCIARRO
and 10 minutes for ImageNet-C.
A.1 Baseline details
In this study, utilized the official implementations of the baseline methods. To ensure consistency, we adopted the reported best hyperparameters documented in the respective papers or source code repositories. Furthermore,
present supplementary information regarding the implementation specifics of the baseline methods and provide a comprehensive overview of our experimental setup, including detailed descriptions of the employed hyperparameters.
SOTTA (Ours). We used ADAM optimizer [14], with a BN momentum of m 0.2, and learning rate of ( = 0.001 with single adaptation epoch. We set the HUS size to 64 and the confidence threshold Co to 0.99 for CIFAR 10-C (10 classes), 0.66 for CIFAR 100-C (100 classes), and 0.33 for ImageNet-C (1,000 classes). We set entropy-sharpner L2-norm constraint p 0.5 following the suggestion [4].
For PL [17], we only updated the BN layers following the previous studies [38, 39]. We set the learning rate as LR = 0.001 as the same as TENT [38].
PL.
TENT. For TENT [38], we set the learning rate LR 0.001 for CIFARIO-C and LR 0.00025 for ImageNet-C, following the guidance provided in the original paper. We referred to the official code for implementations.
LAME. LAME [1] relies on an affinity matrix and incorporates hyperparameters associated with We followed the hyperparameter selection specified by the authors in their paper and referred to their official code* for implementation details. Specifically, we employed the kNN affinity matrix with a value of k set to 5.
COTTA. COTTA [39] incorporates three hyperparameters: the augmentation confidence threshold Pth, restoration factor p. and exponential moving average (EMA) factor m. To ensure consistency, we adopted the hyperparameter values recommended by the authors. Specifically, we set the restoration factor to p 0.01 and the EMA factor to 0.999. For the augmentation confidence threshold, the authors provide guideline for its selection, suggesting using the 5% quantile of the softmax predictions' confidence the source domains. We followed this guideline, which results in Pth 0.92 for CIFARIO-C. Pth. = 0.72 for CIFAR100-C, and Pts = 0.1 for ImageNet-C. We referred to the official code for implementing CoTTA
EATA. For EATA [28], we followed the settings from the original paper. We set LR = 0.005/ 1.005/0.005, /0.00/00000/000 05.50000022 for C--.//CIAAROO--CCCCIIC where | is number of classes. We set cosine sample similarity threshold e = 0.4/0.4 44/0..40.05, trade-off parameter 3 1/1/2, 000, the moving average factor 0.1. We utilized 2,000 samples for calculating Fisher importance as suggested. We referred to the official code for implementing EATA.
entropy constant Fgg 0.4 x |n|
SAR. SAR [29] aims to adapt diverse batch sizes, and we chose typical batch size of 64 for fair comparison. We followed the learning rate as LR 0.00025, sharpness threshold p p= 0.5, and entropy threshold F, 0.4 x In| where |} is the total number of classes, as suggested in the original paper. Finally, we froze the top layer (layer4 for ResNet18) as the original paper, and SoTTA also follows this implementation. We referred to the original code? for implementing SAR.
4 httt:://ggthab..mmm/eeeeaaaaagg
14

---

RoTTA RoTTA [44] uses Adam Optimizer by setting learning rate as LR = 0.001 and 8 = 0.9. We followed the authors' hyperparameters selection from the paper. including -t-stttittic exponential moving average updating rate as 0.05, the Teacher model's s exponential moving average updating rate as V 0.001, timeliness parameter as X = 1.0, and uncertainty parameter as ), = 1.0. We referred to the original code for implementing RoTTA.
A.2 Target dataset details
CIFARIO-CCITARIOOCC
[9] serves as a widely adopted benchmark
for evaluating the robustness of models against corruptions [27, 36, 38, 39]. Both datasets consist of 50,000 training samples and 10,000 test samples, categorized into 10/100 classes. To assess the robustness of models, datasets introduce 15 types of corruptions to the test data, including Gaussian Noise, Shot Noise, Impulse Noise, Defocus Blur, Frosted Glass Blur, Motion Blur, Zoom Blur, Snow, Frost, Fog, Brightness, Contrast, Elastic Transformation, Pixelate, and JPEG Compression. For our experiments, we adopt the highest severity level of corruption, level 5, in line with previous studies [27, 36, 38, 39j. Consequently, the datasets consist of 150,000 corrupted test samples. To train our models, we employ the ResNet18 [8] architecture as the backbone network. The model is trained on the clean training data to generate the source models. We utilize stochastic gradient descent with a momentum of 0.9 and cosine annealing learning rate scheduling [22] for 200 epochs. The initial learning rate is set to 0.1, and a batch size 128 is used during training.
ImageNet-C. ImageNet-C is another widely adopted benchmark for evaluating the robustness of models against corruptions [1, 27, 36, 38, 39]. The ImageNet dataset [3] consists of 1,281,167
training samples and 50,000 test samples. Similar to CIFAR ImageNet-C applies the same 15 types of corruptions, resulting in 750,000 corrupted test samples. We utilize the highest severity level of corruption, equivalent to CIFAR 10-C. For our experiments, we employ a pre-trained ResNet18 [8] model from the Torch Vision library [23], which is pre-trained the ImageNet dataset [3] and is widely used as a backhone for various computer vision tasks.
A.3 Noisy dataset details
CIFAR100 (Near). CIFAR 100 [15] consists of 50,000/10,000 training/test data with 100 classes. We utilized training data without any corruption. We undersampled the dataset to 10,000 for the CIFAR1O-C and CIFAR CIFAR100-C target cases by randomly removing samples and used the entire training set (50,000) for the ImageNet-C target case.
ImageNet (Near). ImageNet [3] consists of 1,281,167/50,000 training/test data with 1,000 classes. We utilized test data without any corruption. We undersampled the dataset to 10,000 for the CIFAR100-C target case by randomly removing samples.
MNIST (Far). MNIST [16] contains 60,000/10,000 training/test data with 10 classes. We utilized test data without any corruption. We used the entire test set for the CIFAR 10-C//CIFAR 100-C target case, and oversampled the dataset by randomly resampling, which results in 50,000 samples that is equivalent to the size of each ImageNet-C target data.
Attack. We implemented the modified indiscriminate distribution invading attack (DIA) [40]. First, we duplicated the entire set of target samples and treated them as malicious samples. Subsequently, we randomly shuffled these duplicated samples within the original target sample set. During the adaptation phase, we injected perturbations into the malicious samples to increase the overall error rate on benign samples within the same batch. As result, we perturbed 10,000 samples (CIFARIO CIFARRI-CICIIAR 100-C) and 50,000 samples (ImageNet-CC to serve as attack samples. For CIFARIO-C/CII used hyperparameters of maximum perturbation constraint and attacking steps N = 10. For ImageNet-C, we used hyperparameters of maximum perturbation constraint 0.2, attack learning rate a = 1/255, and attacking steps N
100-C, we 0.1, attack learning rate = 1/255,
Uniform random noise (Noise). We generated a uniform random valued image in the scaled RGB range 00, 1). with the same height and width as the corresponding target dataset. We generated the same amount of noise samples as each target dataset.
'nttps://githuu htttp:://githubb.com/BBT-T-- www.//I-I////OTTT....
15

---

B Result details
Table 5: Classification accuracy (%) and their corresponding standard deviations on CIFARIO-C for 15 types of corruptions under five scenarios. Bold numbers are the highest accuracy. Averaged over three different random seeds.
Noise
Blur
Weather
Digital
Method
Goa
Shot
Imp.
Def.
Gla.
Mot.
Zoom
Snow
Fro.
Fog
Brit.
Cout.
Elis.
Pix.
JPEG
Avg.
Source
26.0
24.7
56.7
$2.0
64.5
78.0
67.0
74.1
91.5
33.0
76.6
73.2
57.7
+-3
+3.5
+103
+02
+20
=13
+76
-25
-05
-03
=15
-01
-06
-05
-10
Stats (27)
67.0
69.0
60.4
87.8
65.6
86.3
87.4
81.6
80.3
85.4
90.7
86.9
76.7
79.3
71.9
78.4
PL [17]
86.9
64.4
86.6
80.
78.
89.6
84.0
76.2
73.1
78.5
+13
+1.3
-04
-1
-09
-10
-18
-10
-63
-66
-01
-14
-61
TENT [38]
74.5
77.6
66.6
88.2
66.2
86.9
8R.8
83.7
81.3
86.0
91.1
86.9
77.9
82.7
76.7
81.0
Bcniga
LAME [1]
21.8
55.3
79.2
90.1
25.0
74.1
55.9
+36
4-00
---3
+57
-14
-4-4
+17
-63
-11
-67
-11
-66
-01
CoTTA [39)
76.9
78.6
72.3
88.2
70.9
86.8
88.1
85.4
83.4
86.1
91.3
84.9
79.2
83.0
79.9
82.2
EATA [28]
76.0
88.4
88.4
84.5
85.0
91.5
89.9
78.4
82.4
+67
+10
-01
+63
-02
+11
SAR [29]
68.3
69.7
58.9
87.8
628
86.5
81.4
81,6
80.3
R5.4
90.7
86.9
76.7
79.3
720
78.3
RoTTA [44]
67.4
58.3
87.2
62.4
85.8
87.3
81.2
76.9
85.3
901.
57.2
76.5
71.6
75.5
605
+1.1
+09
462
+12
405
+1
+28
466
SOTTA
75.0
77.5
68.8
88.8
70.7
87.5
89.0
85.4
$6.0
88.3
91.9
K3.9
79.8
K3.9
78.
K2.2
+15
Source
26,0
33.2
24.7
56.7
$2.0
67.4
64.8
78.0
67.0
74.1
91.5
$3.9
76.6
46.4
73.2
57.7
BN Stots (27)
64.9
58.2
84.1
85.1
85.3
76.5
70.3
76.6
-03
+09
+07
+06
-65
-61
-03
-66
-63
-10
-01
(17
63.2
63.5
51.7
81.6
58.4
78.4
85.3
79.3
79.7
80.3
80.3
79.5
75.1
70.9
60.3
73.4
TENT [78]
81.3
59.6
85.8
79.6
88.6
85.7
75.5
71.1
74.3
+36
+5.
+-1
+35
+11
+10
+11
+55
-10
14
+10
435
-06
Near
LAME [1]
24.3
31.6
19.9
53.9
53.2
65.9
62.5
79.0
69.5
73.
90.1
28.4
75.0
44.3
74.3
56.4
COTTA [39]
72.7
74.3
66.0
82.6
67.6
81.8
84.]
84.]
85.5
62.5
91.1
69.9
78.1
76.1
79.3
78.4
243
3447
1MA
2
211
203
S4T
204
BADA [28]
51.5
50.3
40.]
70.4
45.8
72.8
77.2
66.8
67.4
74.4
83.9
68.3
60.2
67.3
62.1
63.9
SAR [29
59.0
60,9
52.8
76.2
55.4
63.7
81.8
78.8
7k66
85.5
92.4
85.3
66.6
64.0
63.8
72.4
2154
214.
2144
2347
137
200
281
201
1122
233
RoTTA [44)
66.3
68.3
59.4
86.0
63.2
85.4
87.0
83.5
82.8
86.4
92.4
84.8
77.1
71.6
71,6
77.7
SoTTA
74,3
76.7
66.5
87.5
66.9
86.4
87.8
84.4
83.8
87.2
91.3
88.4
78.7
K2.4
78.0
81.4
+14
+0X
+1A
+4A
+61
-11
+06
-64
Source
26.0
33.2
24.7
56.7
52.0
67.4
66.8
76.0
67.0
76.1
91.5
33.9
76.6
46.4
73.2
57.3
133
133
142
127
122
112
122
12.5
100
112
210
HN Stats 1271
62.7
66.1
56.0
86.5
60.7
86.2
87.2
79.8
78.4
85.7
89.9
80.2
75.5
69.8
65.0
75.2
PLI1F1
55.2
54,1
46.3
63.2
69.3
80.0
83.0
76.8
75.3
80.6
67.6
74.6
70.5
66.8
63.6
69.8
422
1.3
1.32
140
124
210
112
11
21A
119
23A
215
TENT (28]
$1.6
57.4
43.7
84.8
45.5
83.3
85.3
80.4
75.1
83.5
68.9
80.8
75.5
72.1
66.7
71.2
LAME [1]
22.8
29.6
19.3
53.2
50.4
64.6
79.1
67.9
72.8
90.1
28.1
74.7
44.4
74.3
55.5
135
1337
142
21A
2014
217
SHT
240
215
211
20K
201
21B
204
CoTTA [39]
67.4
71.1
59.4
85.3
61.2
82.5
84.3
80.4
80.4
83.8
87.3
58.0
76.0
70.3
72.9
74.3
EATA [28]
40,0
46.4
34.9
73.5
35.
59.5
76.4
61.1
59.5
68.4
85.2
55.2
46.5
53.5
40.0
56.3
+23
230
211
290
212
203
SAR (29)
60.3
62.6
50.9
86.5
55.4
84.3
87.3
79.8
78.
85.7
80.9
8033
70.1
67.8
60.9
73.3
RoTTA [44]
67.3
69.8
61.1
88.1
66.0
86.6
88.0
82.0
78.8
86.2
91.2
61.7
77.5
79.6
73.0
77.1
-05
+10
+18
406
+11
-16
-04
-0X
-10
SoTTA
73,3
76.3
66.3
M0.5
68.3
M6.8
M8.3
84.1
84.2
N7.2
92.0
89.0
77.8
R3.8
77.8
81.6
Source
26.0
33.2
24.7
56.7
$2.0
67.4
64.8
78.0
67.0
74.1
91.5
33.9
76.6
46.4
73.2
57.7
BN Stats (27)
44.5
46.8
39.8
63.3
42.1
63.5
62.5
62.3
59.3
62.1
75.4
65.1
50.6
53.9
46.9
55.9
11.
2113
211
212
217
112
212
215
210
21A
21S
116
214
PL
50.1
61.4
55.1
73.1
51.9
71.7
72.5
71.3
69.1
60
81.3
72.1
60.9
67.4
60.1
66.3
TENT [38]
62.9
65.2
56.7
74.8
54.5
73.4
75.2
73.8
71.8
72.1
85.0
73.7
63.1
70.0
64.0
68.9
111
510
200
200
206
215
214
201
Attck
LAME [1]
21.0
28.0
17.3
52.7
$2.9
65.0
62.0
79.3
70.4
73.9
90.3
28.4
75.6
74.3
55.9
COTTA [39)
53.1
$7.0
49.7
67.9
55.9
69.7
71.3
7423
69.5
68.1
84.0
47.0
68.6
62.3
71.6
60.5
+1A
+13
+18
407
-16
400
414
419
+70
.19
-10
-16
-11
=15
EATA [28]
66.6
68.7
$7.9
75.2
57.2
74.7
75.5
75.1
75.5
73.8
83.5
76.0
64.3
73.5
67.8
70.9
SAR (29)
46.1
48.1
40.3
63.3
42.2
63.5
62.5
62.3
99.3
62.1
75.4
65.1
50.6
$3.0
47.6
56.3
+.5
-17
+13
+16
+18
+15
-10
-14
-15
-10
-15
RoTTA [44]
69.7
71.8
62.9
88.6
67.8
87.4
S8.
85.2
80.1
87.1
91.8
78.4
80.6
74.1
78.4
-55
SoTTA
78.2
80.8
72.3
MI.1
73,6
80.2
9L.3
87.4
86.2
89.3
92.9
87.8
81.3
86.6
81.0
84.5
+13
+111
+1.
+112
+11
+155
+1LS
-11
+BA
Source
26.0
33.2
24.7
56.7
$2.0
67.4
64.8
78.0
67.0
74.1
91.5
33.9
76.6
46.4
73.2
57.7
133
135
142
112
126
100
243
21>
201
106
210
BN Stats (27)
51.7
$3.9
45.5
52.7
41.5
51.0
55.1
62.8
63.8
53.8
76.9
55.8
46.8
564
548
[LT]
47.6
52.7
44.7
48.0
36,
49.4
54.1
61.9
56.5
50.9
77.1
45.3
43.
40.4
50.5
518
...
11%
156
-00
TENT [38]
54.0
57.1
36.7
48.9
28.3
50.5
51.0
64.0
64.7
43.7
38.4
5677
57.0
52.1
Noise
LAME [1]
21.8
28.6
18.5
51.6
$0.8
64.3
60.9
78.4
67.3
71.7
9XL5
27.0
75.1
43.0
73.4
$4.0
.35
+32
+.1
47%
-18
+3%
-15
-07
-00
-10
-06
COTTA [39]
60.4
60.3
52.4
52.0
430
404
423
10A
105
411
243
413
EATA [25]
42.2
41.0
33.2
32.7
25.0
21.9
36.3
$0.8
42.6
31.6
61.5
20.3
27.5
35.8
43.1
36.0
41.
A11
453
431
114
421
134
127
163
L113
233
422
141
283
-02
SAR [29]
57.5
77.1
50.0
111
A17
113
103
112
103
ROTIA|45
61.4
66.9
$6.1
80.1
59.1
798
82.3
79.7
76.7
71.8
91.2
69.0
72.3
73.1
728
73.6
193
114
105
JUA
240
112
265
165
SOTTA
51.
118
111
128
111
118
133
104
134
135
11k
213
114
16

---

Table 6: Classification accuracy (%) and their corresponding standard deviations on CIFARI00-C for 15 types of corruptions under five scenarios. Bold numbers are the highest accuracy. Averaged over three different random seeds.
Noise
Blur
Weather
Digital
Method
Gau
Shot
Imp.
1kef.
Gla.
Mot.
Zoom
Snow
Fro.
Flog
Heit.
Cont.
Elas.
Pix.
JPLG
Avg.
Source
10.6
7.2
34.9
19.6
44.1
41.9
46.3
34.2
41.1
67.3
18.5
50.4
-04
BN stats [27]
39.2
401.7
34.1
66.1
42.5
63.6
64.8
53.8
53.5
59.1
68.2
64.5
53.9
56.6
45.2
53.7
+137
-01
-33
+35
+00
-03
45.7
40.8
66.3
63.7
65.7
56.8
55.1
61.0
68.6
64.6
54.6
+63
TENT [18)
50.0
52.0
46.2
67.9
48.7
66.1
68.0
59.7
59.3
63.4
70.8
67.3
57.5
03.6
53.7
59.5
+055
+03
+1.5
+-5.5
w0.3
+32
+33
+02
+00
Heniga
LAME [1]
5.9
31.6
16.6
42.3
39.8
45.5
31.7
38.3
15.1
49.5
21.6
31.0
+33
COTTA [79]
47.5
48.5
45.2
64.0
46.4
61.7
62.8
55.3
56.1
56.7
68.1
58.4
54,3
000.2
53.4
55.8
KILS
A-4
+02
464
FATA [25]
35,0
31,5
21.6
23.6
34,7
21.1
14.7
412
492
SAR (29)
46.5
48.5
40.9
67.4
46.1
64.9
66.3
56.9
$6.4
61.2
69.8
66.8
56.1
60.3
50.8
57.3
1407
1-11
233
133
RoTTA [44]
43.1
48.7
612
+1.1
+10
+07
+05
+-11
SoTTA
52.0
53.4
45.0
68.8
49.1
66.7
69.0
61.7
601.2
64.7
72.2
66.4
58.6
64.1
55.0
60.5
+11
+13
+44
+55
Source
10.6
12.1
7.2
34,9
19.6
[1.1
41,9
46.3
34.2
4LT
67.3
18.5
50.4
24.9
44.6
33.2
413
2112
100
1:14
133
1:11
133
BN staes 127]
508
2006
2000
110
101
1315
133
101
PL
32.4
32.1
26.5
$8.4
33.5
56.9
58.8
51.5
50.5
$3.4
66.7
51.4
49.3
53.1
45.3
48.0
2:3
23.1
213
11
103
TENT [28]
2.5
2150
$21
E33
515
514
Near
LAME [3]
8.1
9.5
5.9
32.6
17.2
43.0
40.4
45.7
32.7
39.1
66.8
15.6
40.8
22.3
44.0
51.5
134
CoTTA [39)
36.9
$4.8
50,0
200
233
103
[28]
4.5
4.3
3.6
7.4
4.9
7.6
7.5
7.0
5.7
5.8
9.9
5.
6.6
5.7
57
61
SAR (29)
44.6
38.3
63.5
72,1
62.0
51.6
55.4
-01
-07
+04
+35
+33
-21
-B1
-01
RoTTA [44]
36.6
38.6
M0.5
64.0
38.1
61.9
63.7
55.1
50.3
59.4
68.2
26.6
52.7
52.3
44.3
49.4
SoTTA
40.4
64.8
65.8
70.8
65.8
53.4
57.1
-11
-67
+1.
+1.3
-43
+03
-06
-05
+11
-61
Source
10.6
7.2
34,9
19.6
46.3
18.5
44.6
33.2
213
21.2
200
246
1.14
205
294
stass [27]
32.5
34.4
27.9
59.1
34.2
56.3
59.6
48.4
48.6
53.5
64.1
51.0
47.6
46.9
37.0
468
2:11
201
[L
57.5
62.1
519
+3.
$10
+:27
+12
511
501
TENT [38]
17.3
16.9
13.9
$7.5
19.8
55.0
60.0
30.9
40.8
49.8
63.6
51.6
37.0
44.5
32.7
40.0
LAME [1]
5.9
31.2
17.0
38.5
44.9
14.9
43.3
308
+17
+1.
+10
+00
+13
+33
+31
-16
-01
-01
CoTTA (39)
32.1
34.5
28.6
47.2
12.7
49.8
51.1
45.9
46.7
49.3
56.7
20.8
44.0
46.4
418
42.4
EATA [28]
3,2
6.8
4.4
4.3
48
48
404
-05
406
-05
+53
+33
-04
SAR (29)
37.4
38.9
92.2
62.3
36.9
60.3
63.3
51.8
52.5
56.7
66.8
61.1
50.4
53.6
44.2
51.2
RoTTA [44]
39.3
35.2
63.7
66.2
26.3
55.2
41.0
49.8
+2.
+32
+03
+1.
.05
+13
.31
+35
-11
-11
-66
SOTTA
50.8
51.5
42.3
66.9
46.2
64.5
67.3
60.3
59.5
63,8
7007
68.6
55.8
62.5
54.0
59.0
Source
10.6
12.1
7.2
34,9
19.6
44.1
41.9
46.3
34.2
41.1
67.3
18.5
50.4
24.9
44.6
33.2
DN stass 127]
19.0
15.6
45.8
21.8
20.2
-06
+04
.04
[17]
33.8
34.8
29.0
43.8
20.5
44.4
45.1
41.9
40.1
30.8
53.7
30.8
36.0
43.5
35.3
30.3
TENT [78]
25.3
37.0
37.4
34.4
46.4
31.9
-00
+10
+10
+03
-06
+33
-11
+61
Attakk
LAME [1]
7.5
9.1
59
31.8
16.4
42.3
39.4
45.5
31.9
38.4
66.3
15.0
40.4
21.5
43.7
31.0
CoTTA [39]
34.4
34.5
29.9
42.5
38.4
50.6
43.4
30,8
37.2
+08
+06
606
+10
409
-04
+10
-03
+03
+31
-31
-38
-61
-61
-63
EATA [28]
2.2
2.0
2.5
4,4
1.7
3.7
3.8
3.1
2.8
3.5
15.1
2.9
2.5
3.4
27
337
SAR [20]
27.0
28.2
25.1
40.6
34.7
35.6
49.3
39.4
31.0
28.5
34.4
+05
+0G
463
K0G
600
-01
+33
x33
+33
.63
RoTIA [44]
40.5
41.7
35.9
66.2
43.5
64.]
65.5
54.5
49.5
59.8
68.3
25.7
54.7
56.6
46.2
51.5
SoTTA
54.3
55.6
47.6
69.6
$1.6
67.8
69.7
62.7
61.7
66.2
72.5
68.3
59.4
65.3
56.5
61.9
ANT
114
141
1412
111
194
SBA
Source
10.5
12.1
7.2
34,9
19.6
46.1
41.9
46.3
34.2
41.1
67.3
18.5
50.4
24.9
44.6
33.2
413
+122
AM0
105
A06
405
104
A32
4:4
A33
AS1
435
433
464
BN
25.5
25.5
20.8
28.2
22.0
28.4
31.0
30,7
32.6
24.5
44.2
25.8
26.3
29.4
29.A
28.3
PL
21.4
25.6
15.8
22.5
19.8
23.8
28.1
30.7
21.1
47.4
14.0
26.1
23.4
23.8
42A
43A
A2A
121
423
110
L33
133
A-A
423
100
TENT 1381
16.4
17.3
11.3
20.1
11.2
17.8
26.3
20.0
24.6
16.3
50.9
10.9
13.5
25.3
19.5
20.0
Nosse
LAME 131
7.9
9.2
6.1
32,0
16.5
42.5
39.8
45.0
31.9
38.8
66.2
15.1
49.8
21.7
43.6
31.1
11A
217
L10
147
A13
1-3
L33
133
L34
L23
182
181
CoTTA 139]
28.6
28.4
25.3
25.6
21.8
25.3
28.2
28.4
31.0
20.4
38.2
20.1
25.1
33.3
304
27.3
EADA 128]
2.5
2.7
2,3
2.6
2,0
2.0
2.8
1.9
2.4
2.0
2.7
2.2
3.0
2.1
2.6
24
115
30A
INS
148
143
141
134
133
E33
133
133
181
282
SAR (29)
37.5
37.8
29.3
38.8
26.4
38.9
42,4
41.2
42.7
36.4
57.2
32.1
32.9
40.7
37.7
3.
RoTTA [44]
36.6
37.9
31.2
61.9
40.4
60.7
61.9
51.7
45.7
55.9
66.5
25.5
51.6
53.0
44.5
48.3
310
213
514
110
203
134
S34
113
135
111
215
SoTTA
50.4
52.3
41.9
66.3
45.5
65.4
66.7
60.1
59.3
63.1
7007
65.6
55.9
6188
546
88.6
17

---

Table 7: Classification accuracy (%) and their corresponding standard deviations on ImageNet-C for 15 types of corruptions under five scenarios. Bold numbers are the highest accuracy. Averaged over three different random seeds.
Noise
Blur
Weather
Digital
Medsod
Gou.
Shot
Imp.
Def.
Gio.
Mec.
Zocms
Snow
Pro.
Fioe
Hrit.
Cont.
Elas.
Pix.
JPbG
Avg..
Source
1:2
1.5
1.0
11.4
11.2
17.6
10.9
16:5
14.3
$1.3
3.4
168
23.1
20.6
14.6
-00
-00
+00
-00
-00
-.30
-330
-338
-30
-338
-34
-30
-00
-80
-80
sbas [27]
13.0
14.1
13.4
11.7
12.8
23.1
33.3
29.1
28.1
40.3
57.7
11.9
38.4
43.8
36.4
27.1
PL. [17]
16.5
11.2
25.1
39.1
35.5
58.3
5.2
46-5
50.5
45.6
30.5
-12
-1.
+67
+07
+1.
+1.
+.3
-11
-13
-35
+33
+33
+35
+0-3
-B1
+01
TENT [18]
13.0
14.1
13.4
11.7
12.8
23.1
33.3
29.1
28.1
40.3
57.7
11.9
38.4
43.8
36.4
27.1
Seniga
LAME [1]
0.5
11.4
17.5
10.3
16.4
3.4
29.6
14.4
600
+00
400
600
+.00
+000
6.00
+33
+:3
+33
+-38
+330
+33
+80
-00
+00
COTTA [79)
17.7
19.0
18,0
15.7
17.4
30.6
39.0
34.0
32.4
46.9
59.3
18.7
43.1
49.8
42.2
32.2
RATA [25]
27.5
25.9
23.9
45.2
59.9
30.6
48.4
47.0
38.0
402
+01
604
+0.
+.0
+33
+33
+34
+333
+B1
+61
SAR (29)
24.5
26.4
26,5
21,0
21.6
33.3
41.1
3M.3
34.6
49.2
39.4
24.8
46.5
20.7
45.8
36,1
ReTTA [44]
15.1
16.5
15.5
13.1
14.2
25.5
36.1
31.8
28.9
44.3
59.5
15.6
41.6
46.8
40.3
29.7
200
100
2:37
1:31
232
100
200
SoTTA
29.2
31.8
29.8
26.2
27.6
37.9
44.7
42.8
37.9
52.3
60.1
24.1
50.3
53.4
48.7
39,8
Source
1.2
1.8
1.0
11.4
8.7
11.2
17.6
10.9
16.5
14.3
51.3
3.4
168
23.1
29.6
14.6
BN stzEs 1271
5.8
6.9
6.7
8.5
8.5
15.5
24.6
19.1
21.5
26.8
49.7
4.5
26.2
31.9
27.3
18.9
100
141
4.44
1.3.1
1.38
LBE
PLI171
0.5
0.7
0.5
2.2
1.6
4.4
7.2
3.8
3.6
11.4
35.6
0.6
6.5
18.7
57
6.9
TENT (38]
5.8
6.9
6.7
8.5
8.5
15.5
24.6
19.1
21.5
26.8
49.7
4.5
26.2
31.7
27.3
18.9
2013
2011
2000
5.144
2:11
LBD.
201
Near
LAME [3]
1.0
1.5
0.8
10.8
8,4
11.1
17.5
10.3
16.4
14.1
51.3
3.4
16.5
23.0
29.6
14.4
CoTTA [39)
7.8
7,4
11.6
11.1
23.1
30.5
24.6
24.8
36.0
53.7
5.8
31.8
41.4
34.0
23.3
310
241
2:144
2:3
1.31
203
EADA [28]
6.6
9.1
7.7
14.1
12.9
23.3
33.5
20.0
28.9
40.1
55,4
6.4
36.9
43.7
36.5
25.6
SAR (29)
6.7
10.2
8.T
15.9
13.5
28.1
37.0
32.9
28.2
44.6
56.8
1.8
40.8
47.7
41.6
27.6
+1a
+HA
14A
-1.3
RoTTA [44]
3.1
5,4
3.7
10.8
9.3
23.0
33.5
32.3
30.3
44.6
59.3
0.7
40.2
46.1
25.6
SoTTA
0.3
5.8
0.5
21.7
21.5
26.9
39,0
35,0
28.1
46.5
56.1
0.5
44.8
49.2
43.2
27.9
-01
+0.0
+02
+16
+0.
+01
+33
+31
-61
-6.
Source
1.2
1.8
1,0
11.4
8.7
11.2
17.6
10.9
16.5
14.3
$1.3
3.4
168
23.1
29.6
14.6
210
3100
100
100
140
1400
1400
5:00
1.88
8:38
1.3.
131
200
UN stass 127)
4.5
5.0
5.0
4.2
5.3
9.1
16.2
17.4
18.0
22.2
43.3
1.2
23.2
26.9
20.9
14.8
[LF]
0.4
0.5
0.5
0.7
1,0
1.4
3,4
2.2
2.8
34.7
0.2
63
12.3
4.3
5.1
20m
300
2.1
133
STA
203
TENT [28]
5.0
5.0
4.2
5.3
9.2
16.2
17.4
18.0
22.2
43.3
1.2
23.2
26.9
20.9
148
LAME [1]
0.7
1.1
0.5
11.4
8.5
11.1
17.5
10.3
16.3
14.0
51.3
3.3
16.5
23.0
20.6
14.4
600
5100
100
100
1:3
200
CoTTA (39)
5.1
4.5
4.2
6.2
10.7
18.9
21.6
76-0
30.0
48.2
0.9
27.9
27.0
17.6
EATA [28]
7.9
10.1
10.1
8.9
10.2
17.9
28.8
27.1
26.8
38.2
52.2
0.8
34.3
40.3
$2.6
23,1
-06
+05
+03
+05
+08
+13
-33
+31
+33
-B1
-61
SAR (29)
3.7
3.5
5.5
20.6
32.1
31.8
26.1
43.1
54.0
0.4
30.2
39.1
23.5
RoTTA [44]
13.9
15.5
16.2
12.1
12.8
25.0
35.8
33.6
20.5
45.8
59.4
3.0
41.9
47.0
41.0
20.2
-01
-01
+H1
-01
-11
+11
+32
+13
-83
+01
-63
SoTTA
26.9
29.5
27.3
22,3
23.6
35.8
42.2
40.8
35.5
50.7
58.4
1.6
46.8
36.1
Source
1.2
1.0
11.4
11.2
17.6
10.9
16.5
14.3
51.3
3.4
168
23.1
29.6
14.6
DN stats 127]
6.4
7.7
6.7
7.2
7.3
12.5
20.0
19.2
17.6
24.8
46.1
10.5
25.1
24.5
24.9
17.4
500
2011
100
101
203
[17]
7.8
6.3
4.0
10.5
22.2
20.0
9.7
79.8
48.6
1.9
34.5
70.8
36.0
18.1
TENT [18]
7.7
6.7
7.2
12.5
20.0
19.2
17.6
46.1
10.5
24.5
25.0
-05
Amack
LAME [1]
0.7
1.1
0.5
11.4
8.5
10.7
17.5
10.3
16.4
14.1
51.3
3.4
16.5
18.5
20.6
14.0
+00
+00
+80
+00
400
405
+35
+30
.86
COTTA [39]
17.2
18.6
17.3
14.5
15.0
27.0
32.7
31.9
28.3
40.2
52.0
18.9
38.6
38.7
+13
FATA [28]
15.3
17.5
15.5
13.9
12.9
22.0
28.8
25,0
24.3
35.5
49.2
17.3
35.7
39.1
36DD
26.1
+02
402
#02
+:1
+11
+3.8
+12
+B1
+61
SAR|29] [29]
35.8
2001
201
202
123
2:38
83.
205
210
RoTIA [44)
19.0
19.7
19.1
16.8
17.1
28.9
38.4
35.7
31.1
47.0
59.8
22,0
43.5
30.1
43.3
32.0
100
211%
1111
2112
2:32
2:1
2:11
212
SoTTA
1.63
111
1.14
261
161
Source
100
11.4
100
100
10.9
4.8
1335
400
BN stass 127]
7.0
7.5
7.4
5.1
6.0
7.5
11.9
12.4
11.4
10.7
34,5
4.4
16.3
23.9
25.7
12.8
484
100
1011
1400
101
4.8
100
PLJ1F1
1013
L15
180
TENT 1881
7.0
7.5
7.4
5.1
6.0
7.8
[1.8
12.3
11.3
10.6
34,5
4.5
16.3
23.9
25.6
12.
1000
1001
113
100
Nose
LAME 131
2000
2100
200
1000
200
2:50
2135
5.3
133
230
COTTA [39]
R.3
9.0
9.2
4.3
5.4
7.8
13.3
16.6
13.7
14.9
44.0
3.2
19.4
30.9
35.3
160
1000
243
2:11
2158
2:33
538
111
284
185
EATA [28]
2111
2012
203
293
234
233
283
281
SAR (29)
13.9
18.8
16.3
5.1
3.4
6.6
25.0
27.6
19.8
31.9
40.8
1.0
20.0
41.3
38.3
20.0
21.
103
113
231
111
205
11.3
1:1
131
RoTTA [44]
42.4
41.3
312
202
402
400
--14
133
515
561
SoTTA
27.5
30.4
28.3
26.5
27.3
37.7
43.6
42.4
36.9
51.6
50.2
2388
49.6
S3D
483
39,0
*43
161
18

---

Table 8: Classification accuracy (%) and their corresponding standard deviations on ablation study of individual components CIFARIO-C for 15 types of corruptions under five scenarios. Bold numbers are the highest accuracy. Averaged over three different random seeds.
Nosse
Blur
Weather
Digital
Metiod
Gan.
Shot
Imp.
Def.
Gila.
Mor.
Zoom
Fro.
Fog
Brit.
Conr.
Elas.
JPBG
Avg.
Source
26.0
33.2
24.7
56.7
52,0
67.4
64.8
76.0
67.0
74.1
91.5
33.9
76.6
26.4
73.2
57.7
1333
1353
1.43
2223
1.27
4:13
21.6
1.23
-008
11.5
1100
1.9.5
1:10
HC:
10.2
10.9
10.7
61.9
14.2
41.8
82.0
36.3
52.3
98.7
11.2
34.0
14.2
23.5
34.9
401
631.
A11.7
4.677
A1.3
4158
A.3.
A48
UC
26.0
42.0
10.5
82.4
59,0
79.3
82.5
77.5
76.5
81.5
86.3
73.2
73.3
74.9
71.4
66.4
+65
+1.5
-06
+1.
+.5
+1.1
Denign
HC + UC (HUS)
21.1
84.6
62.4
83.4
85.2
79.3
76.4
75.1
02.88
ESM
57.9
89.6
67.7
78.5
FSC ESM
69.0
57.7
59.
84.4
81.0
79.7
78.1
UC: FSM
74.9
77.1
08.2
98.7
71.0
87.4
89.1
85.0
34.0
57.8
92.0
86,2
79.8
78.0
82.2
HUS ESM (SoTTA)
75.0
77.5
68.8
85.8
57.5
59.0
34.0
58.2
91.9
83.9
79.8
78.3
$2.2
+65
+03
+1.5
+07
Source
26.0
33.2
34.7
56.7
$2.0
67.4
64.8
78.0
67.0
74.1
91.5
33.9
76.6
46.4
73.2
57.7
A33
1333
1:4
4227
A27
2.55
L...
203
A.S5
4.10
HC
10.3
10.5
10.1
12.6
12.3
11.9
16.7
16.1
14.4
14.5
16.6
11.2
19.7
13.2
14.0
13.6
+01
+35
+8.
-19
+1.4
-3.4
1.55
+0.7
+00
UC
48.3
20.8
73.3
50.4
73.1
74.9
71.0
61.5
71.77
N2.4
59.7
64.3
64.2
62.1
6.44
4.4
4.2
4.5
435
L1.
L44
Near
EEC + UC (IUSS)
+3.5
15.9
+33
73.8
-14
77.
-154
70.8
61.6
+3.
62.6
66.2
+53
61.7
-13
ESM
58.5
83.9
86.7
81.3
85.2
83.3
77.9
76.7
76,0
77.9
HC ESM
86.3
87.6
84.8
81.2
86.9
87.3
79.1
62.3
77.3
80.9
UC FSM
68.1
60.3
84.8
84.2
65.5
82.2
81.7
65.3
90.8
84.4
77.0
76.5
78.0
IIUS ESM (SoTTA)
66.5
87.5
96.4
87.8
83.8
57.2
91.3
88.4
78.7
75.0
$1.4
623
482
Seurce
26.0
33.2
24.7
56.7
52.0
67.4
64.8
78.0
67.0
74.1
91.5
33.9
76.6
46.4
73.2
57.7
+5.
8335
+1.1
+22
516
101
11.
106
+0.
HC
10.3
10.7
10.1
15.4
12.1
17.4
26.7
16.2
16.6
17.9
58.5
11.0
15.3
12.9
15.8
17.6
401
10.5
1007
183
1.5
1.254
103
1.1
UC
49.3
43.5
10.2
66.5
46.7
65.3
69.4
67.5
62.0
65.8
74.3
55.4
55.8
59.2
55.8
56.5
449
+62
44.4
433
43.
+111
+20
[EC + UC (IIUS)
26.2
11.9
70.0
49.2
72.0
77.3
72.8
70.2
75.2
84.2
53.9
62.9
65.2
66.3
58.4
+11
-34
-16
ESM
49.7
84.3
52.2
83.3
85.2
75.0
$5.2
78.8
71.8
66.6
72.8
+2.3
+1.1
40.
+25
HC ESM
55.2
82.1
82.2
82.5
70.4
76.6
69.0
+31
UC HSM
57.3
84.5
85.9
79.0
85.1
90.8
81.9
73.5
76.8
69.3
75.9
HITS + FSM (SeTTA)
84.2
89.0
77.8
77.8
A.1
Source
26.0
33.2
24.7
56.7
12.0
67.4
64.8
78.0
67.0
74.1
91.5
33.9
76.6
26.4
73.2
57.7
233
1353
1:4
22.1
X27
133
5.25
20.
11.5
211
HC
10.2
10.4
10.1
15.1
11.7
21.5
31.6
17.1
13.1
22.0
32.6
10.5
18.5
11.9
17.2
16.9
410
402
144
ALL.
435
10.55
3.3
13.5
4.45
10.
UC
44.1
47.3
10.1
K2.9
63.1
S1.7
$4.3
79.1
79.0
83.7
68.3
$1.0
73.2
78.5
73.3
70,0
1315
2.333
1.2
1224
155
212
115
19.5
L35
11.
135
Attack
HC + UC (HUR)
673
4.2.6
10.3
A84
-145
A16T
47.9
54.4
46.4
L144
4350
45.3
55.0
47.5
4230
42.6
4.2.5
36,5
-145
40.9
BSM
79.5
71.6
71.8
58.3
85.5
87.2
B5.4
79.8
-04
-05
-04
+1.
HC LSM
70.9
89.3
71.8
87.9
85.6
85.4
87.3
R5.
79.8
UC ESM
72.3
85.7
89.3
88.6
80.5
84.3
HUS ESM (SoTTA)
72.3
90.1
89,2
86.2
89.3
87.8
81.0
84.5
Source
26.0
33.2
24.7
56.7
$2.0
67.4
64.8
78.0
67.0
74.1
91.5
33.9
76.6
46.4
73.2
57.7
-3.
+3.5
-13
-11
+3.
+13
-16
+35
-16
+AA
-07
+10
HC
10.3
10.3
10.1
17.5
11.8
18.1
18.9
19.0
14.1
25.4
41.6
11,1
15.1
13.7
14.6
16.8
243
295
233
51S
$1.8
1000
51.5
193
UC
24.8
41.8
10.1
72.4
$3.0
70.0
74.1
69.9
60.9
66.6
77.9
67.7
64.2
67.2
63.2
59.5
4.5.
4
43.4
122
415
422
A1.2
48A
A1.3
413
A23
A32
Noise
HC + UC (HUS)
21.7
-114
40.5
10.7
74.0
+46
46.6
72.5
+43
78.1
+36
73.3
71.3
76.6
+1.3
81.1
112
40.9
+113
65.9
67.4
+27
63.3
440
58.9
+25
ESM
59.4
51.7
48,3
57.1
65.8
65.8
61.4
77.8
56.2
63.6
60.5
42.1
4.1
A157
437
A33
L4T
42A
A31
443
41L3
A63
453
HC HSM
58.0
748
67.6
70.8
70.1
68.7
+13
+34
+13
-15
-14
+33
+33
+23
UC LSM
64.7
62.2
82.2
84.6
10,1
82.6
90.1
79.8
73.6
76.9
77.7
HUS + ESM (SoTTA)
66.8
86.1
84.3
86.6
810
85.7
84,1
77.2
187
424
19

---

Table 9: Classification accuracy (%) and their corresponding standard deviations on ablation study of the size of Noise on CIFARIO-C for 15 types of corruptions. Bold numbers are the highest accuracy. Averaged over three different random seeds.
Nosse
Weather
Digital
Mathod
Giu
Shot
Emp.
Def
Giln
Met.
7ores
Snow
Fm..
Fogg
Bril.
Cont.
Flis.
Fix.
JPFG
Avg.
Source
26.0
33.2
24.7
56.7
52.0
67.4
64.88
75.0
67.0
74,1
91.5
33.9
76.6
46.4
73.2
57.7
+:35
+:3.
4.4
+1.
+277
A1122
+.6
+E4
+2.5
+133
6.1
467
40.
A.3
BN stats (27)
59.6
61.6
51.5
66.9
49.8
65.3
68.6
71.2
71.5
65.0
84.2
70.0
58.8
63.5
65.4
64.9
-0.
+03
+01
-13
-00
+01
+81
-65
+02
+05
+33
+11
+13
-05
+B1
PL117]
59.9
61.3
52.1
66.4
46.1
62.7
67.2
09.6
69.2
65.4
83.8
66.8
55.5
64.9
65.6
63.8
147
3:55
2114
542
11.
213
$20
51.2
5:44
230
541
TENT [38]
64.3
70.0
59.3
68.8
47.7
65.8
72.4
73.6
75.4
65.4
88.2
69.0
63.7
72.2
67.8
68.1
4:2
4.24
L42
4311
A30
45.
140
433
A33
5000
LAME [1]
22.0
28.9
18.8
52.2
51.2
64.9
61.5
78.9
68.0
72.3
90.3
27.6
75.3
43.6
73.8
55.3
+36
+1.1
+3.5
+0.5
+0.3
+13
+1.
+85
CoTTA [39]
68.5
53.1
62.1
64,5
86.1
70.2
67.4
41.
440
427
A35
423
A1.
EATA [28]
58.4
58.8
38.7
$7.9
64.5
76.0
$4.6
58.5
+0.
-9.1
+25
+0.
SAR [29]
65.0
65.5
RoTTA [44]
59.9
78.9
91.0
74.0
SOTTA
85,0
78.6
80.5
Source
26.0
33.2
34.7
56.7
$2.0
67.4
6488
78.0
67.0
74,1
91.5
33.9
76.6
46.4
73.2
57.7
+35
+3.5
+13
-31
-04
+1.
+1
407
-05
+10
RN stArs [27]
51.7
53.9
45.5
52.7
41.5
$1.0
55.1
62.8
63.8
53.8
76.9
55.8
46.8
54.8
56.4
548
6.3
210
=17
107
200
11
207
591
PL7]
47.6
52.7
44.7
48.9
36.1
19.4
54.1
61.9
56.5
50,9
77.1
45.2
43.1
49.4
59.5
518
190
1.34
112A
15.1
129
214
14.
1.33
148
143
185
TENT [38]
56.0
57.1
36.7
48.9
28.3
50.5
51.0
64.0
64.7
49.5
80.5
43.7
38.4
56.7
57.0
52.1
+67
+4.
45.2
+1.
+30
445
454
10000
LAME [1]
21.8
28.6
51.6
50.8
64.3
60.9
67.3
71.7
90.5
27.0
43.0
73,4
54.9
-35
-33
COTTA
60.4
47.3
41.6
44.1
47.7
79.0
60.2
548
+21
=04
+27
+24
+13
+13
EATA [28)
61.5
43.1
3600
-57
SAR
59.4
$7.9
77.1
59,8
RoTTA [44]
60.1
59.1
79.8
72.8
73.6
SOTTA
73.3
$6.1
84.3
85.7
91.1
77.2
$0.0
Source
26.0
33.2
34.7
56.7
52.0
67.4
64.8
78.0
67.0
74.1
91.5
33.9
76.6
46.4
73.2
57.7
4.3
4.42
4.7
4227
11.2
A.3
A.3
A.D
BN stats (27)
41.3
42.9
37.2
37.4
32.6
36.1
39.6
$2.0
52.7
10.6
65.0
37.4
33.9
46.1
41.4
42.5
197
4.1.3
21A
11.
11.5
13.
183
PL.(7]]
25.5
20.5
21.7
20.2
41.7
25,0
60.8
23.1
29.5
28.5
A11
L10
450
A57
A23
TENT [38]
21.5
18.5
17.8
63.8
19.9
25,0
25.0
-05
+13
-53
+14
20000
LAME [1]
21.7
50.
29.6
63.6
71.0
90.6
2666
73.3
54.3
CoTTA [39)
409
EATA [78)
22.3
15.1
SAR [29)
41.6
39.6
33.4
29.3
34.6
36.1
38.5
33.0
422
RoTTA [44]
62.5
64.5
75.
58,3
79.0
79.0
77.3
91.3
69.0
71.5
73.4
72,4
729
SOTTA
73.2
75.6
61.0
84.5
84.8
89.7
82.8
81.1
77.5
78.6
218
234
13A
11A
514
118
20

---

C Additional ablative studies
We conducted experiments to understand the sensitivity of our two hyperparameters: confidence
threshold (Co) and BN momentum (m). We varied Co and and reported the corresponding accuracy.
85.0
85.0
Classification accuracy (%)
82.5
62.5
Classification accuracy
80.0
80.0
77.5
77.5
75.0
75.0
72.5
Benign
Far
Noise
72.5
Benign
Far
Noise
Near
Attack
Near
Atlack
70.0
61
0.9
0.99
0.895
0.999
70.0
0.06
0.16
02
0.25
(a) Effect of hyperparameter Co.
(b) Effect hyperparameter m.
Figure 8: Effect of hyperparameters the model accuracy CIFARIO-C for 15 types of corruptions under five scenarios: Benign, Near, Far, Attack, and Noise. Averaged over three different random seeds.
Confidence threshold. Our result shows that the selection of Co shows similar patterns across different scenarios (Benign ~ Noise). The result illustrates a tradeoff; a low Co value does not effectively reject noisy samples, while a high Co value filters benign data, We found proper value of Co (0.99) that generally works well across the scenarios. Also, we found that the optimal Co depends primarily in-distribution data. Our interpretation is that setting different Co values for CIFARIO-C. CIFARI00 100-C, and ImageNet-C is straightforward as they have a different number of classes (10, 100, and 1,000), which leads to different ranges of the model's confidence.
BN momentum. Across the tested range, the variations in performance This finding indicates that choosing a low momentum value from within the specified range ((0.05, 0.3]) is adequate maintain favorable performance. Please note that setting high momentum would corrupt the result, which is implicated by the algorithms directly utilizing test-time statistics (e.g., TENT) suffering from accuracy degradation with noisy data streams (e.g., TENT: 81.0% 52.1% for Noise at Table 1).
found to be negligible.
D Further discussions
D.1 Theoretical explanation of the impact of noisy data streams
We provide a theoretical explanation of the impact of noisy data streams with common entropy minimization as an example. With the posterior distribution p of the model in terms of training data D and benign test data B in test-time adaptation:
frameworks [2, 7], we can express the
A B
log () = log (0))
H (yol2o).
The posterior distribution of model parameters depends on the prior distribution q and the average of entropy H of benign samples with a multiplier À. Here, we incorporate the additional noisy data stream N into Equation and introduce new posterior distribution considering noisy streams:
[N|
(og,( log (())
A B
H (3)
AN
Σ H(yn)
(8)
[N
21

---

Table 10: Average classification accuracy (%) and their corresponding standard deviations on ablation study of the effect of high- high-confidence uniform- class continual memory of SoTTA on CIFARIO-C. Bold numbers are the highest accuracy. Averaged over three different random seeds.
Benign
Near
Far
Attack
Noise
Avg
SoTTA (wo High-confidence)
82.2 #02
78.0 e.4
75.9 #0.
84.3
77.7 +07
79.6 +.2
SoTTA (w/ Uniform-class)
82.3 #.2
80.9 e06
74.9 224
83.5
68.7
78.0 +00
SoTTA (w/o Continual)
81.0 +0.
79.5 +03
75.5 +.8
84.4
65.7 +70
77.2 1.8
SoTTA
82.2 +.3
81.4 0.5
84.5
80.0
81.9 0.5
With Equation and Equation 8, we can now derive model parameter variations caused by noisy test samples:
logr(0|D, logr(01D,B,N)
AN
Σ (ymllz.)
N
Equation 9 implies that the (1) model adapted only from benign data and (2) model adapted with both benign and noisy data differ by the amount of the average entropy of noisy samples. This also suggests that high entropy from severe noisy samples would result in significant model drift in adaptation (i.e., model corruption).
D.2
Comparison with previous TTA methods
D.2.1 EATA and SAR
While SoTTA, EATA [28], and SAR [29] all leverage sample filtering strategy, the key distinction of input-wise robustness of SoTTA and EATA/SAR lies in three aspects: (1) Our high-confidence sampling strategy in SoTTA aims to filter noisy samples by utilizing only the samples with high confidence, while both EATA and SAR use a different approach that excludes a few high-entropy samples, particularly during the early adaptation stage. In our preliminary study, found that our method excludes 99. 99.9% of the noisy samples, whereas EATA and SAR exclude 33.55% of such samples. (2) While EATA and SAR adapt to every incoming low-entropy sample, SoTTA leverages uniform-class memory management approach to prevent overfitting. As shown in Figure 5b, noisy samples often lead imbalanced class predictions, and these skewed distributions could lead to an undesirable bias in p(y) and thus might negatively impact TTA objectives, such as entropy minimization. The ablation study in Table 10 shows the effectiveness of uniform sampling with 3. 3.9%p accuracy improvement. (3) EATA and SAR reset the memory buffer and restart the sample collection process for each adaptation. This strategy susceptible to overfitting due to a smaller number of samples used for adaptation and the temporal distribution drift of the samples. In contrast, our continual memory management approach effectively mitigates this issue by retaining high- confidence uniform-class samples in the memory, shown in Table 10.
We acknowledge that both SoTTA and SAR utilize sharpness- aware minimization proposed by Foret et al. [4]. However, we clarify that the motivation behind using SAM is different. While SAR intends avoid model collapse when exposed samples with large gradients, we aim to enhance the model's robustness to noisy samples with high confidence scores. As illustrated in Figure 6, we observed that entropy-sharpna minimization effectively prevents the model from overfitting to noisy samples. As result, while our algorithm led to marginal performance degradation in noisy settings (82.2% - 80.0% for Noise), EATA and SAR showed significant degradation (EATA 82.4% + 36.0% for Noise; SAR 78.3% + 58.3% for Noise).
awaree
D.2.2 RoTTA
Regarding our high- high-confidence uniform sampling technique, RoTTA [44] could be compared. First of all, RoTTA's objective is different from ours, RoTTA focused on temporal distribution changes of test streams without considering noisy samples. Similar to SoTTA, RoTTA's memory bank maintains recent high-confidence samples. However, RoTTA has no filtering mechanism for low-confidence samples, which makes RoTTA fail avoid noisy samples, especially in the early stage of TTA. In contrast, confidence-bar based memory management scheme efffectivly rejects noisy samples, and
22

---

Table 11: Average classification accuracy (%) of ODIN+TTA on CIFARIO-C. Bold numbers are the accuracy with improvement from normal TTAs. Averaged over three different random seeds.
Benign
Allack
Method
w/o ODIN
w/ ODIN
w/o ODIN
w/ ODIN
w/o ODIN
w/ ODIN
wio ODIN
w/ ODIN
wlo ODIN
w/ ODIN
Source
57.7
57.7 1.0
21.7
$7.7 10
57.7 18
57.7
57.7
57.7 1:
57.7 1.
57.7 L.
BN
78.2
78.2
76.5
76.5
+03
75.9
55.8
55.8
55.9
56.7
[17]
78.4
78.8 21.
23.1
74.3 106
71
71.6
66.5 11.1
66.5 1111
52.1
52.1 204
TENT [38]
81
=1.
74.5
76.1 :06
13.
74.7
69.0
69.1
54,4
56.2
56.1
26.7
56.2
54.9
$5.2
COTTA [39]
82.2
82.
78.
78.2 204
73
73.6
69.6
69.6
57.8
62.0
EATA
82.4
an3
63.9.04
69.2
56.3
59.9
70.9
70,9
36.0
50.8
SAR [29]
78.4
78.4
=2.8
72.8
75.7
76.0
56.2
16.2
58.7 20.
58.7 =03
RoTTA [44]
75.3 km2
75.3 AM7
77.5
77.0
77.0 aus
78.4 AMA
78.4 ABS
73.5 AM3
73.5
SOTTA
Classitcation accuracy (6)
70
50
30-
20-
ODIN (Benign)
(Near)
ODIN (Far)
(Attack)
(Noise)
we ODIN (Benign)
w/o ODIN (Near)
wio ODIN (Far)
we ODIN (Attack)
wro ODIN (Noisa)
0.1001
0.1000
1.10
0.1087
1009
0.1011
0.1015
11013
Figure 9: Effect of OOD threshold 8 on classification accuracy %% of TDNT+ CIFAR10-C. Averaged over three different random seeds.
thus it prevents potential model drift from the beginning of TTA scenarios. As result, our approach outperforms RoTTA noisy test streams (e.g., 5.4%p better than RoTTA on CIFARIO-C).
D.3 Comparison with out-of-distribution detection algorithms
We discussed the limitation of applying -uitof-distribution detection TTA in Section Still, are curious about the effect of applying out-of-distribution algorithms to our scenario. To this end, we conduct experiments using of the out-of-distribution algorithms, ODIN [20], in our noisy data streams. Specifically, we filtered OOD samples detected by ODIN and performed TTA algorithms on the samples left.
Note that similar to prior studies on OOD, ODIN uses a thresholding approach to predict whether sample is OOD. It thus requires validation data with binary labels indicating whether it is in- distribution or OOD to decide the best threshold 5. However, in TTA scenarios, validation data not provided, which makes it difficult apply OOD algorithms directly in scenario. We circumvented this problem using the labeled test batches to get the best threshold. Following the original paper, we searched for the best threshold from 0.1 to 0.12 with step size of 0.000001, which took over 20,000 times longer than the original TTA algorithm.
Table 11 shows that the impact of discarding OOD samples with ODIN is negligible, yielding only 0.3%p improvement in the average accuracy despite a huge computation cost. Also, Figure shows the high sensitivity of ODIN with respect to threshold hyperparameter 6, which implies that applying OOD TTA is impractical.
We conclude the practical limitations of OOD detection algorithms for TTA as follows: (1) OOD methods assume that model is fixed during test time, while a model changes continually in TTA. (2) As previously noted, most OOD algorithms require labels for validation data unavailable in TTA scenarios. Even using the same test dataset for selecting the threshold, the performance improvement was marginal. (3) Low performance possibly results from the fact that OOD detection studies are
23

---

built on the condition that training and test domains are the same, which differs from TTA's scenario. These collectively make it difficult to apply OOD detection studies directly to TTA scenarios.
D.4 Applying to other domains
While this study primarily focuses on classification tasks, there are other tasks where test-time adaptation would be useful. Here we discuss the applicability of SoTTA (1) image segmentation and (2) object detection, which are crucial in autonomous driving scenarios.
For image segmentation, when noisy objects are present in the imput. the model might produce noisy predictions on those pixels, leading to detrimental results. Extending SoTTA to operate at the pixel level would allow it to be compatible with the segmentation task while minimizing the negative influences of those noisy pixels on model predictions in test-time adaptation scenarios.
Similarly. SoTTA could be tailored to object detection's 's classification (recognition) task. For example, the context of the YOLO framework [32], SoTTA could filter and store grids with high confidence for test-time adaptation, enhancing detection accuracy. However, our current approach must address the localization task (bounding box regression) during test-time adaptation. Implementing this feature non-trivial and would require careful consideration and potential redesign of certain aspects of our methodology. Accurately localizing bounding boxes during test-time adaptation presents an exciting avenue for future research.
E License of assets
Datasets CIFAR 1O/CIFAR 100 (MIT License), CIFAR10CC/CIFARIOOCC (Creative Commons
Atrriuttonn 4.0 International), ImageNet-C (Apache 2.0), and MNIST (CC-BY-NC-SA 3.0).
Codes Torchvision for R (Apache 2.0), the official repository of CoTTA (MIT License), the official repository of TENT (MIT License), the official repository of LAME (CC BY-NC-SA 4.0), the official repository of EATA (MIT License), the official repository of SAR (BSD 3-Clause License), and the official repository RoTTA (MIT License).
24


Reviews:
[{"review_id": "WZzaSOROI4", "reviewer": "Reviewer_kiEt", "summary": "This paper proposes screening-out test-time adaptation which is claimed robust to non-interest samples. It filters out the impact of non-interest samples with a high-confidence uniform-class sampling. It proposes entropy-sharpness minimization to deal with large gradients. Experiments are completed on CIFAR10-C and ImageNet-C.", "questions": "1. Can it also work on segmentation and detection? Please discuss.", "limitations": "Please refer to \"weakness\".", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The proposed problem (screening-out test-time adaptation) is novel and sharp.\n2. This paper has a good presentation. For example, Fig.1 and Fig.2 are very clear.\n3. Entropy-sharpness minimization seems reasonable as discussed in Section 3.2.\n4. Experiments are completed on both small and large datasets including ImageNet-C. As shown in Tab.1, the improvement is good enough.", "weaknesses": "1. The problem is interesting but the proposed two methods are straightforward. I am a little worried about the technical novelty.\n2. The design of input-wise robustness is hard coding. Although some samples are bad for optimization, they may be essential for the perception.\n3. Only classification is completed. However, segmentation is also very important in autonomous driving scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and effort in providing us with positive comments. We respond to your question in what follows. Please also refer to the *global response* we posted together.\n\n---\n**Weakness 1. The problem is interesting but the proposed two methods are straightforward. I am a little worried about the technical novelty.**\n\nWe appreciate the reviewer's insightful comment. Regarding high-confidence uniform sampling, RoTTA could be compared with our method. Regarding the memory bank concept introduced in RoTTA, it is essential to note that the fundamental objective of RoTTA differs from our proposed method. RoTTA's memory bank primarily focuses on maintaining recent high-confidence samples. However, without a dedicated filtering method for low-confidence samples, RoTTA fails to avoid non-interest samples, especially in the early stage of test-time adaptation. In contrast, our confidence-based memory management scheme effectively rejects non-interest samples, and thus it rejects potential model drift from the beginning of TTA scenarios. As a result, our approach outperforms RoTTA in noisy test streams, as shown in Table 1 of our manuscript. \n\nRegarding the second component, we acknowledge that both SoTTA and SAR utilize sharpness-aware minimization proposed by Foret et al. [r1]. However, we clarify that the motivation behind using SAM is different. While SAR intends to avoid model collapse when exposed to samples with large gradients, we aim to enhance the model's robustness to non-interest samples with high confidence scores. As illustrated in Figure 6 in the manuscript, we observed that entropy-sharpness minimization effectively prevents the model from overfitting to non-interest samples.\n\n\n\n\n**Weakness 2. The design of input-wise robustness is hard coding. Although some samples are bad for optimization, they may be essential for the perception.**\n\nLow-confidence samples could be beneficial for the perception. However, we want to note the risks of model corruption associated with including such low-confidence samples in the selection. In our discussion of the effect of hyperparameters, as detailed in the global response #2, we observed that lowering the confidence threshold (C0) resulted in performance degradation (e.g., 81.0% (C0=0.99) -> 77.7% (C0=0.7) in Noise scenario; see Figure 8 in the attached PDF). This observation supports the need for a cautious approach in dealing with low-confidence samples. In light of your feedback, we will incorporate this discussion of the trade-off between the perception and risks of model corruption in our filtering strategy into the manuscript.\n\n\n\n**Weakness 3. Only classification is completed. However, segmentation is also very important in autonomous driving scenarios. / Question 1. Can it also work on segmentation and detection? Please discuss.**\n\nWe appreciate your insightful comment. While our work primarily focuses on classification tasks, we acknowledge the importance of image segmentation and object detection, particularly in autonomous driving scenarios.\n\nFor image segmentation, when non-interest objects are present in the input, the model might produce noisy predictions on those pixels, leading to detrimental results. Extending our method to operate at the pixel level would allow it to be compatible with the segmentation task while minimizing the negative influences of those noisy pixels on model predictions in test-time adaptation scenarios. \n\nSimilarly, our method could be tailored to object detection's classification (recognition) task. For example, in the context of the YOLO framework, our method could filter and store grids with high confidence for test-time adaptation, enhancing detection accuracy. However, our current approach must address the localization task (bounding box regression) during test-time adaptation. Implementing this feature is non-trivial and would require careful consideration and potential redesign of certain aspects of our methodology. Accurately localizing bounding boxes during test-time adaptation presents an exciting avenue for future research. \n\nWe sincerely appreciate the reviewer's insightful comment and will incorporate this discussion into our manuscript.\n"}, {"review_id": "nyRr0IdyLy", "reviewer": "Reviewer_gePD", "summary": "This article presents a new Test-Time Adaptation (TTA) scenario, wherein the model is adapted to noisy test streams. To address the challenges posed by this scenario, the paper introduces the Screening-out Test-Time Adaptation (SoTTA) algorithm, which leverages input-wise and parameter-wise robustness. The effectiveness of SoTTA is demonstrated through extensive comparison experiments conducted on TTA benchmarks.", "questions": "1. What would be the outcome if CSTU and ESM were combined? Would the performance surpass that of SoTTA?\n2. It is acknowledged that in real-world scenarios, the non-interest scenes mentioned in the experimental section may occur simultaneously. Have any relevant experiments been conducted to address this?\n", "limitations": "Actually, the problem setting is interesting to some extent, however, the techniques in this paper are similar to several compared methods, which may limit the novelty of this paper.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. This work proposes a novel test time adaptation setup, which considers the noisy data (non-interest) in real world application during test phase, whose motivation is convincing.\n2. The significant performance gain shows SoTTA addresses the challenges under noisy test stream well.\n", "weaknesses": "1. The initial component of the method is inspired by the memory bank concept introduced in RoTTA, albeit in a simplified form. The second component, pioneered by SAR[1], represents a novel approach within the TTA domain. However, it is worth noting that the methodology section may lack originality.\n2. The sensitivity analysis and selection criterion for key hyperparameters, specifically $m,C_0$ , are not included in the current study.\n3. The motivation of this paper stems primarily from experimental findings, and it would be advantageous to analyze the motivation or methods from a theoretical perspective.\n4. The paper contains several typographical errors, and it would be valuable to thoroughly revise it.\n\n[1] Towards stable test-time adaptation in dynamic wild world\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and effort in providing us with positive comments. We respond to your question in what follows. Please also refer to the *global response* we have posted together.\n\n---\n**Weakness 1.**\n\nWe appreciate the reviewer's insightful comment. Regarding the memory bank concept introduced in RoTTA, it is essential to note that the fundamental objective of RoTTA differs from our proposed method. RoTTA's memory bank primarily focuses on maintaining recent high-confidence samples. However, without a dedicated filtering-out method for low-confidence samples, RoTTA fails to avoid non-interest samples, especially in the early stage of test-time adaptation. In contrast, our confidence-based memory management scheme effectively rejects non-interest samples, and thus it rejects potential model drift from the beginning of TTA scenarios. As a result, our approach outperforms RoTTA in noisy test streams, as shown in Table 1 of our manuscript. \n\nRegarding the second component, we acknowledge that both SoTTA and SAR utilize sharpness-aware minimization proposed by Foret et al. [r1]. However, the motivation behind using SAM is different. While SAR intends to avoid model collapse when exposed to samples with large gradients, we aim to enhance the model's robustness to non-interest samples with high confidence scores. As illustrated in Figure 6 in the manuscript, we observed that entropy-sharpness minimization effectively prevents the model from overfitting to non-interest samples.\n\n**Weakness 2.**\n\nThank you for your valuable suggestion. We include the sensitivity analysis of the hyperparameters at global response #2. \n\n\n**Weakness 3.**\n\nWe appreciate your suggestion. The current manuscript highlights the practical issues of noisy data streams in real-world TTA scenarios from experimental findings and proposes a methodology to address this problem effectively.\n\nHere, we provide the theoretical motivation for handling noisy data streams in TTA scenarios. With the Bayesian-learning-based frameworks [r2, r3], we can express the posterior distribution of the model in terms of training ($D$) / benign test data ($B$) in test-time adaptation: \n\nEquation 1.\n$\\log p(\\theta | D, B) = \\log q(\\theta) - \\frac{\\lambda_B}{|B|} \\sum_{b=1}^{|B|} H(y_b | x_b)$\n\nThe posterior distribution of model parameters depends on the prior distribution ($q$) and the average of entropy ($H$) of benign samples with a certain weight ($\\lambda$).\n\nHere, we incorporate the additional noisy data stream ($N$ into Equation 1 and introduce the new posterior distribution for our target scenario with noisy streams:\n\nEquation 2.\n$\\log p(\\theta | D, B, N) = \\log q(\\theta) - \\frac{\\lambda_B}{|B|} \\sum_{b=1}^{|B|} H(y_b | x_b)  - \\frac{\\lambda_N}{|N|} \\sum_{n=1}^{|N|} H(y_n | x_n)$\n\nWe can now derive model parameter variations for benign and non-interest test samples:\n\nEquation 3.\n$\\log p(\\theta | D, B) - \\log p(\\theta | D, B, N)= \\frac{\\lambda_N}{|N|} \\sum_{n=1}^{M} H(y_n | x_n)$\n\n\nEquation 3 implies that the (1) model adapted only from benign ($B$) and (2) model adapted with both benign ($B$) and noise ($N$) differs by the amount of the average entropy of noisy (non-interest) samples. The equation also suggests that a high entropy from severe noise would result in a significant model drift in adaptation.\n\nWe appreciate the reviewer's constructive feedback. We will incorporate and discuss theoretical analysis further in our final manuscript. \n\n\n\n**Weakness 4.**\n\n\nThank you for your careful examination of our paper. We apologize for the typographical errors in the initial submission, and we will thoroughly revise the manuscript.\n\n\n\n**Question 1.**\n\nHere, we compare the original SoTTA with the CSTU+ESM version (under the same setting as Table 1 in our manuscript).\n\n| | Benign | Near | Far | Attack | Noise | Avg |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|\n| SoTTA | 82.9±0.4 | 81.4±0.5 | 81.6±0.5 | 84.5±0.3 | 81.0±1.5 | 82.3  |\n| CSTU+ESM | 83.0±0.2 | 81.0±0.0 | 78.5±0.2 | 83.7±0.2 | 78.7±0.9 | 81.0 |\n\nThe above table shows that CSTU shows comparable accuracy to SoTTA (HUS+ESM) without non-interest samples (Benign). However, CSTU shows 2.3x higher performance degradation with Noise samples (- 4.3%p) than SoTTA (-1.9%p). This result originates from CSTU's structure of only maintaining the \"recent high-confidence samples.\" CSTU does not discard any low-confident samples when the memory is not full, which is susceptible to harmful samples in the early adaptation stage.\n\n\n**Question 2.**\n\nOur methodology filters non-interest samples, which could be easily extended to simultaneous non-interest scenes. We expect to further develop robust TTA methods in more complex settings, such as mixed-domain, shuffled-class scenarios [r4].\n\n\n\n[r1] Foret, Pierre, et al. \"Sharpness-aware Minimization for Efficiently Improving Generalization.\" International Conference on Learning Representations. 2020.\n\n[r2] Brahma, Dhanajit, and Piyush Rai. \"A Probabilistic Framework for Lifelong Test-Time Adaptation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n\n[r3] Yves Grandvalet and Yoshua Bengio. \"Semi-supervised learning by entropy minimization.\" Advances in neural information processing systems, 17, 2004.\n\n[r4] Marsden, Robert A., Mario Döbler, and Bin Yang. \"Universal Test-time Adaptation through Weight Ensembling, Diversity Weighting, and Prior Correction.\" arXiv preprint arXiv:2306.00650 (2023).\n"}, {"review_id": "7iVOTZesjD", "reviewer": "Reviewer_6Z6q", "summary": "This paper studies a practical problem of test-time adaptation where non-interest testing samples may appear and mislead the adaptation. This problem is quite serious in practical applications, and the problem setting is relatively novel. To address this problem, the authors propose the SoTTA method, which solves this problem in two aspects. For input-wise robustness, SoTTA filters out the non-interest samples with a high-confidence uniform-class memory buffer. For parameter-wise robustness, entropy-sharpness minimization is adopted to ensure that the landscape is smooth during the adaptation. The proposed method is evaluated on two benchmark datasets under one standard setting and four robust settings with different types of non-interest samples. The results show that SoTTA gives state-of-the-art performance compared to existing TTA methods.", "questions": "1. The experiment shows that RoTTA outperforms existing TTA methods even in standard settings. Can the authors explain why this phenomenon occurs? Intuitively, the techniques mentioned in this article are all aimed at addressing the problem of non-interesting samples. What factors enable them to achieve better performance when non-interesting samples do not exist?\n2. Can the authors provide some sensitivity analysis regarding the threshold C0? It is very important for the actual effect of filtering out non-interesting samples. An adaptive scheme for setting C0 would also be acceptable.\n3. One naive way to solve the proposed setting is to adopt an out-of-distribution process before test-time adaptation. How does this naive method work in the proposed setting?\n\n**[IMPORTANT]** I would like to raise my score if you successfully address my concerns in the Weakness and Question sections. Overall, this paper is interesting and I appreciate it. However, it is currently slightly below the level of acceptance.", "limitations": "The authors have properly discussed the limitations.", "rating": 5, "confidence": 5, "soundness": 1, "presentation": 3, "contribution": 2, "strengths": "1. This paper studies a novel problem setting of test-time adaptation, where non-interest testing samples may appear and mislead the adaptation. This is a relatively novel and practical problem in real applications.\n2. The overall idea of this paper makes sense. This paper tackles the harmful impact of non-interest with a filter and robust loss function. The experiments in the main paper also show that the proposed method gives state-of-the-art performance.", "weaknesses": "1. First, this paper ignores one common TTA benchmark, i.e., CIFAR100. Additionally, the results of ImageNet-C have been hidden in the appendix. These key results should be presented in the main paper with comprehensive analysis.\n2. One TTA algorithm, EATA [1], adopts a similar sample selection strategy during adaptation. This method should be taken into consideration in the experiment parts because it has the potential to handle the problem of non-interest samples.\n3. This paper does not provide any discussion about the selection and sensitivity of threshold C0. Intuitively, the optimal C0 depends on both in-distribution data and out-of-distribution data, and it is not easy to decide in real applications.\n4. There is more related work that can be discussed to further improve this paper. For test-time adaptation, the above-mentioned EATA method [1] and recent advanced TTA methods [2-4] about robustness and evaluation should be considered. For out-of-distribution detection, methods [5-6] that can be efficiently optimized in an unsupervised manner should be taken into consideration.\n\n**Reference**\n\n[1] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, Mingkui Tan: Efficient Test-Time Model Adaptation without Forgetting. ICML 2022: 16888-16905\n\n[2] Hao Zhao, Yuejiang Liu, Alexandre Alahi, Tao Lin: On Pitfalls of Test-Time Adaptation. ICML 2023\n\n[3] Zhi Zhou, Lan-Zhe Guo, Lin-Han Jia, Dingchu Zhang, Yu-Feng Li: ODS: Test-Time Adaptation in the Presence of Open-World Data Shift. ICML 2023\n\n[4] Tong Wu, Feiran Jia, Xiangyu Qi, Jiachen T. Wang, Vikash Sehwag, Saeed Mahloujifar, Prateek Mittal: Uncovering Adversarial Risks of Test-Time Adaptation. ICML 2023\n\n[5] Zhi Zhou, Lan-Zhe Guo, Zhanzhan Cheng, Yu-Feng Li, Shiliang Pu: STEP: Out-of-Distribution Detection in the Presence of Limited In-Distribution Labeled Data. NeurIPS 2021: 29168-29180\n\n[6] Jiangpeng He, Fengqing Zhu: Out-Of-Distribution Detection In Unsupervised Continual Learning. CVPR Workshops 2022: 3849-3854", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and effort in providing us with positive comments. We respond to your question in what follows. Please also refer to the *global response* we have posted together.\n\n---\n**Weakness 1.**\n\nIn our current manuscript, the CIFAR100 dataset acted as one of the non-interest scenarios (\"Near\") as it has similar characteristics to CIFAR10 and ImageNet, as detailed in Section 4 of our manuscript. However, we acknowledge the importance of the CIFAR100 benchmark. We will conduct experiments on the CIFAR100 dataset and include the results in the final manuscript.\n\nFor your concerns regarding ImageNet-C, we put its result in the appendix due to space limitations. We will restructure the evaluations section to incorporate the ImageNet-C result in the main paper.\n\n\n**Weakness 2.**\n\nThank you for your valuable suggestion. We include the results of EATA and discussions in global response #1. To summarize, we found that EATA shows significant accuracy degradation with the presence of non-interest samples (e.g., 82.4% -> 36.6% for Noise). \n\n\n**Weakness 3.**\n\nThank you for your valuable suggestion. We include the sensitivity of the hyperparameter C0 and discussions in global response #2.\n\n\n**Weakness 4.**\n\nWe want to highlight that our problem setting and approach differ from those studies. EATA [1] is primarily concerned with preventing catastrophic forgetting, which differs from our goal of progressively adapting to new test samples of interest. We have experimented with EATA in global response #1. DIA [4] 's primary focus is designing an attack method for TTA, which we use as the attack scenario in our experiment (Section 4 of our manuscript). TTAB [2] and ODS [3] are tailored to address distributional shifts but do not consider non-interest samples. The focus of OOD methods [5, 6] is the detection of OOD samples within the same domain, which is not suitable for TTA scenarios. We have included the experimental results and limitations of OOD detection methods in response to your Question 3 below. We will include discussions on these studies in our final manuscript.\n\n\n**Question 1.**\n\nWe assume the reviewer asked about our algorithm (SoTTA), not RoTTA.\n\nOur interpretation of this result is two-fold. First, our high-confidence uniform-class sampling strategy filters not only non-interest samples but also benign samples that would negatively impact the algorithm’s objective. This implies that there exist samples that are more beneficial for adaptation, which aligns with the findings that high-entropy samples harm adaptation performance [1]. Second, entropy-sharpness minimization helps ensure both robustness to non-interest samples and generalizability of the model by preventing model drifts, leading to performance improvement with benign samples. This is also consistent with SAR [r1], which argues that the entropy-sharpness minimization is robust to large gradients, which could harm the adaptation performance.\n\n\n**Question 2.**\n\nThank you for your valuable suggestion. We include the sensitivity of the hyperparameter C0 and discussions in global response #2.\n\n\n**Question 3.**\n\nFollowing your suggestion, we conducted additional experiments using one of the famous OOD detection algorithms, ODIN [r2], in our noisy data streams. Specifically, we filtered OOD samples detected by ODIN and performed TTA algorithms on the samples left.\n\nSimilar to prior studies on OOD, ODIN uses a thresholding approach to predict whether a sample is OOD. It thus requires validation data with binary labels indicating whether it is in-distribution or OOD to decide the best threshold. However, in TTA scenarios, validation data is not provided, which makes it hard to apply OOD algorithms directly in our scenario. We circumvented this problem using the labeled test batches to get the best threshold. Following the original paper, we searched for the best threshold from 0.1 to 0.12 with a step size of 0.000001, which took over 20,000 times longer than the original TTA algorithm.\n\nThe result (Table 8 in the one-page PDF) shows that the impact of discarding OOD samples with ODIN (with-ODIN) is negligible, yielding only a 0.331% improvement in the average accuracy despite a huge computation cost. \n\nThere exist practical limitations of OOD detection algorithms for TTA. (1) OOD methods assume that a model is fixed during test time, while a model changes continually in TTA. (2) As previously noted, most OOD algorithms require labels for validation data unavailable in TTA scenarios. Even using the same test dataset for selecting the threshold, the performance improvement was marginal. (3) Low performance possibly results from the fact that OOD detection studies are built on the condition that training and test domains are the same, which differs from TTA’s scenario. These collectively make it difficult to apply OOD detection studies directly to TTA scenarios. We discussed these limitations in Section 5 of our manuscript.\n\nWe will include the whole result in our appendix in our final manuscript.\n\n\n[r1] Niu, Shuaicheng, et al. \"Towards Stable Test-time Adaptation in Dynamic Wild World.\" ICLR ‘22\n\n[r2] Liang, Shiyu, Yixuan Li, and R. Srikant. \"Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks.\" ICLR ‘18\n"}, {"review_id": "gmdoGA8PRr", "reviewer": "Reviewer_HUhd", "summary": "The authors point out that model may suffer from non-interest samples while TTA. Existing TTA methods are not robust to these samples. To address these issues, the authors proposed a methods called SoTTA with two key components, input-wise robustness via high-confidence uniform-class sampling and parameter-wise robustness via entropy-sharpness minimization.", "questions": "See comments above.", "limitations": "See comments above.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The authors focus on TTA in the wild and point out that TTA with non-interest would lead to performance degradation.", "weaknesses": "The proposed method seems to be not novel.\n\nA) The proposed “Input-wise robustness via high-confidence uniform-class sampling” method excludes low-confidence samples in TTA. Such samples filtration strategies have been explored in ETA [Efficient Test-Time Model Adaptation without Forgetting].\n\nB) The proposed “Parameter-wise robustness via entropy-sharpness minimization” method introduces entropy-sharpness minimization, which have been explored in SAR [Towards Stable Test-time Adaptation in Dynamic Wild World].\n\nIt would be better for the authors to clarify the differences between the proposed methods and the TTA methods I mentioned above.\n\nIn the experimental results, I found the proposed always outperforms SAR. But in my understanding, the main components of these two methods are almost the same. Both of them include low-confidence sample filtration strategy and entropy-sharpness minimization. So I have no idea why the proposed method can yield much better results.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your time and effort in providing us with positive comments. We respond to your question in what follows. Please also refer to the *global response* we have posted together.\n\n---\n\n**Weakness 1. It would be better for the authors to clarify the differences between the proposed methods and the TTA methods I mentioned above.**\n\nWhile SoTTA (ours), EATA, and SAR all leverage sample filtering strategy, the key distinction of input-wise robustness of SoTTA and EATA/SAR lies in two aspects: (1) characteristics of the samples filtered out and (2) memory management strategies. \n\nFirst, our high-confidence sampling strategy in SoTTA aims to filter non-interest samples by utilizing only the samples with high confidence, while EATA and SAR use the same algorithm that excludes a few high-entropy samples, particularly during the early adaptation stage. As evidence, we show that our method excludes 99.98% of the non-interest samples, whereas EATA and SAR exclude 33.55% of such samples. \n\nSecond, while EATA and SAR adapt to every incoming low-entropy sample, SoTTA leverages a uniform-class sampling approach to prevent overfitting. As shown in Figure 5, non-interest samples often lead to imbalanced class predictions, and these skewed distributions could lead to an undesirable bias in p(y) and thus might negatively impact TTA objectives, such as entropy minimization. The ablation study in Table 2 of our paper shows the effectiveness of uniform sampling with a 9.8% accuracy improvement.\n\nWe acknowledge that both SoTTA and SAR utilize sharpness-aware minimization proposed by Foret et al. [r1]. However, we clarify that the motivation behind using SAM is different. While SAR intends to avoid model collapse when exposed to samples with large gradients, we aim to enhance the model's robustness to non-interest samples with high confidence scores. As illustrated in Figure 6 in the manuscript, we observed that entropy-sharpness minimization effectively prevents the model from overfitting to non-interest samples.\n\nIn conclusion, while our algorithm results in marginal performance degradation in noisy settings (82.9% -> 81.0% for Noise), EATA and SAR show significant degradation: EATA 82.4% -> 36.6% for Noise and SAR 78.3% -> 58.3% for Noise. The detailed results of SoTTA and SAR are included in the manuscript, and the result of EATA is included in the global response #1.\n\n\n\n\n**Weakness 2. In the experimental results, I found the proposed always outperforms SAR. But in my understanding, the main components of these two methods are almost the same. Both of them include low-confidence sample filtration strategy and entropy-sharpness minimization. So I have no idea why the proposed method can yield much better results.**\n\n\nWe summarize the key difference between SoTTA and SAR in the comments above. To conclude, It is precisely due to the robustness of our method against non-interest samples that SoTTA outperforms SAR. By reducing the influence of non-interest samples and considering class imbalance through uniform-class sampling, our approach mitigates the risk of model corruption caused by biased learning from a few non-interest samples. It results in better overall performance, as demonstrated in our experimental results.\n\nWe thank the reviewer for pointing out this, and we will revise our manuscript to clarify the difference between SoTTA and SAR.\n\n\n[r1] Foret, Pierre, et al. \"Sharpness-aware Minimization for Efficiently Improving Generalization.\" International Conference on Learning Representations. 2020.\n"}]

Author rebuttal:
## Global Response

Dear Reviewers, 

We sincerely appreciate your efforts and time in reviewing our manuscript.

The contribution of our work lies in investigating the crucial yet unexplored challenge of test sample diversity in real-world scenarios. Notably, we unveil that existing TTA algorithms suffer from significant performance degradation with such noisy streams. As a solution, we propose SoTTA that is robust to the sample diversity and outperforms the state-of-the-art baselines.


We appreciate the reviewer's constructive and insightful comments on our paper. To answer your concerns and questions, we have written the responses with the following additional experiments and clarification:


Additional experiments on EATA and RoTTA to clarify the differences between the methods and our approach (6Z6q, HUhd, gePD)
Sensitivity analysis of the hyperparameters (gePD, 6Z6q)
Clarification of the effectiveness of our input-wise robustness method compared with other existing approaches (kiEt, gePD)
Clarification of the difference between SoTTA and SAR (gePD, HUHd)
Theoretical analysis regarding the motivation for handling noisy data streams in TTA scenarios (gePD)
Additional experiment to compare our method with an OOD detection method (6Z6q)
Clarification of our evaluation scenario with CIFAR100 (6Z6q)
Comparison between our method and relevant TTA methods to clarify the focus of our work (6Z6q)
Discussion of the possibilities of applying our methods in broader scenarios such as image segmentation and object detection (kiEt)

Additionally, we summarize two major issues as follows:

### 1. Comparison with EATA [c1] (Reviewer 6Z6q and HUhd)

Two reviewers (6Z6q and HUhd) identified EATA [c1] as a critical baseline for its similarity in sample selection with our approach. In response, we evaluated EATA under two conditions on our noisy data stream scenario with CIFAR10-C: (1) EATA, where we applied the method directly to the noisy data stream, encompassing Fisher importance calculation and model adaptation in the presence of non-interest samples, and (2) EATA-Clean, where we gave EATA an advantage by excluding the non-interest samples when calculating the Fisher information matrix. We ensured that the experimental settings remained consistent with those described in our original submission, as noted in Table 1 of our manuscript.

The experiment results are presented in the table below:

| | Benign | Near | Far | Attack | Noise | Avg |
|---|:---:|:---:|:---:|:---:|:---:|:---:|
| SoTTA | 82.9±0.4 | 81.4±0.5 | 81.6±0.5 | 84.5±0.3 | 81.0±1.5 | 82.3 |
| EATA-Clean | 82.4±0.2 | 64.4±1.5 | 57.6±2.6 | 70.9±0.7 | 36.6±1.2 | 62.4 |
| EATA | 82.4±0.2 | 63.9±0.4 | 56.3±0.5 | 70.9±0.6 | 36.0±0.8 | 61.9 |

From the results, we observed a significant accuracy degradation for EATA in the presence of non-interest samples, dropping from 82.4% to 36.6% when confronted with noisy data (Noise). Our proposed SoTTA algorithm utilizes high-confidence sampling, effectively excluding a more significant number of non-interest samples. SoTTA filtered out 99.98% of non-interest samples, whereas EATA only excluded 33.55% in the memory-filling stage.

In addition, EATA calculates the Fisher importance based on a few initial test-time samples (e.g., 2000 samples [c1]) to calculate parameter importance and prevent catastrophic forgetting. This approach becomes problematic as including non-interest samples in the initial sample set corrupts the Fisher importance values, resulting in a degradation of accuracy (EATA-Clean 62.4% -> EATA 61.9%).



### 2. Effect of hyperparameters C0, m (Reviewer gePD and 6Z6q)

In response to the reviewers' (gePD and 6Z6q) request for the sensitivity analysis of hyperparameters, we conducted experiments to present the results for two specific hyperparameters: confidence threshold (C0) and BN momentum (m). Notably, we varied C0 within the range of [0.7, 0.999] and m within [0.05, 0.3] and reported the corresponding accuracy. Please refer to the detailed results in the attached PDF file.


Confidence threshold (C0) (Figure 8a in the one-page PDF):

Our result shows that the selection of C0 shows similar patterns across different scenarios (Benign ~ Noise). The result illustrates a tradeoff; a low C0 value does not effectively reject non-interest samples, while a high C0 value filters benign data. We found a proper value of C0 (0.99) that generally works well across the scenarios. Also, we found that the optimal C0 depends primarily on in-distribution data. Our interpretation is that setting different C0 values for CIFAR10 and ImageNet is straightforward as they have a different number of classes (10 vs. 1000), which leads to different ranges of the model’s confidence. 


BN momentum (m) (Figure 8b in the one-page PDF):

Across the tested range, the variations in performance were found to be negligible. This finding indicates that choosing a low momentum value from within the specified range ([0.05, 0.3]) is adequate to maintain a favorable performance. Please note that setting a high momentum would corrupt the result, which is implicated by the algorithms directly utilizing test-time statistics (e.g., TENT) suffering from accuracy degradation with noisy data streams (e.g., TENT: 81.0% -> 52.1% for Noise at Table 1 in the manuscript).


We will incorporate these findings and discussions into the final manuscript. Thank you again for your thoughtful review, and we are open to any further suggestions or questions you may have.

[c1] Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, Mingkui Tan: Efficient Test-Time Model Adaptation without Forgetting. ICML 2022: 16888-16905


Meta-review: This work on test-time adaptation focuses on adaptation in the presence of noisy/irrelevant/adversarial data: it contributes an experimental protocol for this setting, comprehensive experiments, and a technically straightforward but effective method (SoTTA). Four expert reviewers with backgrounds on adaptation/generalization, open world modeling, and data-efficient learning mainly vote for acceptance (kiEt: 7, HUhd: 5, 6Z6q: 5) with one vote for rejection (gePD: 4). The authors provided a rebuttal, three of the four reviewers replied, and scores and two of three borderline scores were raised (6Z6q, HUhd). SoTTA shares common elements with recent more robust adaptation methods (especially RoTTA and SAR), but it does differ in (arguably minor) details that nevertheless improve results. This is demonstrated in comparison with strong and up-to-date baselines (including recent methods like SAR). At the same time, the experiments are informative about the more "open world" setting, which is an important direction of progress for test-time adaptation. The AC sides with acceptance in alignment with the majority of reviewers and the positive direction of the scores after rebuttal. The AC encourages the authors to incorporate the feedback from reviewers, and in particular emphasizes the inclusion of the sensitivity analysis (gePD, 6Z6q) and the discussion of TTA methods that are closely related in technique or setting (gePD, 6Z6q: EATA, ODS, ...). Although some of these papers are concurrent, referencing and discussing such work is more informative to the reader and makes for a more connected community.

Note: the AC asks the authors to reflect on their choice of "non-interest" terminology, and consider aligning with existing terminology such as "open world" or "distractor" samples, so that this work is more accessible to the existing audiences for this topic.
Decision: Accept (poster)