SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Taesik Gong†* Yewon Kim‡* Taeckyung Lee‡* Sorn Chottananurak‡ Sung-Ju Lee‡
†Nokia Bell Labs ‡KAIST
taesik.gong@nokia-bell-labs.com
{yewon.e.kim,taeckyung,sorn111930,profsj}@kaist.ac.kr
Test-time adaptation (TTA) aims to address distributional shifts between training and testing data using only unlabeled test data streams for continual model adaptation. However, most TTA methods assume benign test streams, while test samples could be unexpectedly diverse in the wild. For instance, an unseen object or noise could appear in autonomous driving. This leads to a new threat to existing TTA algorithms; we found that prior TTA algorithms suffer from those noisy test samples as they blindly adapt to incoming samples. To address this problem, we present Screening-out Test-Time Adaptation (SoTTA), a novel TTA algorithm that is robust to noisy samples. The key enabler of SoTTA is two-fold: (i) input-wise robustness via high-confidence uniform-class sampling that effectively filters out the impact of noisy samples and (ii) parameter-wise robustness via entropy-sharpness minimization that improves the robustness of model parameters against large gradients from noisy samples. Our evaluation with standard TTA benchmarks with various noisy scenarios shows that our method outperforms state-of-the-art TTA methods under the presence of noisy samples and achieves comparable accuracy to those methods without noisy samples. The source code is available at https://github.com/taeckyung/SoTTA.
Deep learning has achieved remarkable performance in various domains [6, 8, 33], but its effectiveness is often limited when the test and training data distributions are misaligned. This phenomenon, known as domain shift [31], is prevalent in real-world scenarios where unexpected environmental changes and noises result in poor model performance. For instance, in autonomous driving, weather conditions can change rapidly. To address this challenge, Test-Time Adaptation (TTA) [1, 5, 29, 38, 39, 44] has emerged as a promising paradigm that aims to improve the generalization ability of deep learning models by adapting them to test samples, without requiring further data collection or labeling costs.
While TTA has been acknowledged as a promising method for enhancing the robustness of machine learning models against domain shifts, the evaluation of TTA frequently relies on the assumption that the test stream contains only benign test samples of interest. However, test data can be unexpectedly diverse in real-world settings, containing not only relevant data but also extraneous elements that are outside the model's scope, which we refer to noisy samples (Figure 1). For instance, unexpected noises can be introduced in autonomous driving scenarios, such as dust on the camera or adversarial samples by malicious users. As shown in Figure 2, we found that most of the prior TTA algorithms showed significantly degraded accuracy with the presence of noisy samples (e.g.,  $81.0\% \rightarrow 52.1\%$  in TENT [38] and  $82.2\% \rightarrow 54.8\%$  in CoTTA [39]).
Figure 1: Unlike prior assumptions (Clean TTA), real-world test streams could include unexpected noisy samples out of the model's scope (Noisy TTA), such as glare, fallen leaf covering the lens, unseen objects (e.g., a flamingo), and noise in autonomous driving scenarios. The accuracy of existing TTA methods degrades in such cases.
Figure 2: Average classification accuracy  $(\%)$  of existing TTA methods and our method (SoTTA) on CIFAR10-C. The performance of existing methods degrades when noisy data are mixed into the test stream (Noisy) compared with the original assumption (Clean). Higher is better.
To ensure the robustness of TTA against noisy samples, an intuitive solution might be screening out noisy samples from the test stream. Out-of-distribution (OOD) detection [10, 11, 18, 19, 20, 21, 24, 25, 43] is a representative method for this, as it tries to detect whether a sample is drawn from the same distribution as the training data or not. Similarly, open-set domain adaptation (OSDA) [30, 35] and universal domain adaptation (UDA) [34, 42] generalize the adaptation scenario by assuming that unknown classes are present in test data that are not in training data. However, these methods require access to a whole batch of training data and unlabeled target data, which do not often comply with TTA settings where the model has no access to train data at test time due to privacy issues [38] and storing a large batch of data is often infeasible due to resource constraints [12]. Therefore, how to make online TTA robust under practical noisy settings is still an open question.
In this paper, we propose Screening-out Test-Time Adaptation (SoTTA) that is robust to noisy samples. SoTTA achieves robustness to noisy samples in two perspectives: (i) input-wise robustness and (ii) parameter-wise robustness. Input-wise robustness aims to filter out noisy samples so that the model will be trained only with benign samples. We achieve this goal via High-confidence Uniform-class Sampling (HUS) that avoids selecting noisy samples when updating the model (Section 3.1). Parameter-wise robustness pursues updating the model weights in a way that prevents model drifting due to large gradients caused by noisy samples. We achieve this via entropy-sharpness minimization (ESM) that makes the loss landscape smoother and parameters resilient to weight perturbation caused by noisy samples (Section 3.2).
We evaluate SoTTA with three common TTA benchmarks (CIFAR10-C, CIFAR100-C, and ImageNet-C [9]) under four noisy scenarios with different levels of distributional shifts: Near, Far, Attack, and Noise (Section 2). We compare SoTTA with eight state-of-the-art TTA algorithms [1, 17, 27, 28, 29, 38, 39, 44], including the latest studies that address temporal distribution changes in TTA [1, 28, 29, 39, 44]. SoTTA showed its effectiveness with the presence of noisy samples. For instance, in CIFAR10-C, SoTTA achieved  $80.0\%$  accuracy under the strongest shift case (Noise), which is a  $22.3\%$  improvement via TTA and  $6.4\%$  better than the best baseline [44]. In addition, SoTTA achieves comparable performance to state-of-the-art TTA algorithms without noisy samples, e.g., showing  $82.2\%$  accuracy when the best baseline's accuracy is  $82.4\%$  in CIFAR10-C.
Contributions. (i) We highlight that test sample diversity in real-world scenarios is an important problem but has not been investigated yet in the literature. We found that most existing TTA algorithms undergo significant performance degradation with sample diversity. (ii) As a solution, we propose SoTTA that is robust to noisy samples by achieving input-wise and parameter-wise robustness. (iii) Our evaluation with three TTA benchmarks (CIFAR10-C, CIFAR100-C, and ImageNet-C) show that SoTTA outperforms the existing baselines.
(a) Benign.
(b) Near.
(c) Far.
(d) Attack.
(e) Noise.
Figure 3: Five test sample scenarios considered in this work: Benign, Near, Far, Attack, and Noise.
Figure 4: Overview of SoTTA. SoTTA achieves input-wise robustness via high-confidence uniform-class sampling (HUS) and parameter-wise robustness via entropy-sharps minimization (ESM).
Test-time adaptation. Let  $\mathcal{D}_S = \{\mathcal{X}^S, \mathcal{Y}\}$  be source data and  $(\mathbf{x}_i, y_i) \in \mathcal{X}^S \times \mathcal{Y}$  be each instance and the label pair that follows a probability distribution of the source data  $P_S(\mathbf{x}, y)$ . Similarly, let  $\mathcal{D}_{\mathcal{T}} = \{\mathcal{X}^{\mathcal{T}}, \mathcal{Y}\}$  be target data and  $(\mathbf{x}_j, y_j) \in \mathcal{X}^{\mathcal{T}} \times \mathcal{Y}$  be each target instance and the label pair following a target probability distribution  $P_{\mathcal{T}}(\mathbf{x}, y)$ , where  $y_j$  is usually unknown to the learning algorithm. The covariate shift assumption [31] is given between source and target data distributions, which is defined as  $P_S(\mathbf{x}) \neq P_{\mathcal{T}}(\mathbf{x})$  and  $P_S(y|\mathbf{x}) = P_{\mathcal{T}}(y|\mathbf{x})$ . Given an off-the-shelf model  $f(\cdot; \Theta)$  pre-trained from  $\mathcal{D}_S$ , the (fully) test-time adaptation (TTA) [38] aims to adapt  $f(\cdot; \Theta)$  to the target distribution  $P_{\mathcal{T}}$  utilizing only  $\mathbf{x}_j$  given test time.
Noisy test samples. We define noisy test samples to represent any samples that are not included in the target data distribution, i.e.,  $\tilde{\mathbf{x}} \notin \mathcal{X}^T$ . We use the term noisy to distinguish it from out-of-distribution (OOD), as TTA typically aims to adapt to OOD samples, such as corrupted ones. Theoretically, there could be numerous categories of noisy samples. In this study, we consider five scenarios: Benign, Near, Far², Attack, and Noise. Figure 3 shows examples of these scenarios. Benign is the typical setting of TTA studies without noisy samples, Near represents a semantic shift [41] from target distribution, Far is a severer shift where covariate shift is evident [41], Attack refers to intelligently generated adversarial attack with perturbation [40], and Noise refers to random noise. We focus on these scenarios to understand the impact of noisy samples in TTA as well as the simplicity of analysis. Detailed settings are described in Section 4.
Problem and challenges. Prior TTA methods assume test samples are benign and blindly adapt to incoming batches of test samples. The presence of noisy samples during test time can significantly degrade their performance for those TTA algorithms, which has not been explored in the literature yet. Dealing with noisy samples in TTA scenarios is particularly challenging as (i) TTA has no access to the source data, (ii) no labels are given for target test data, and (iii) the model is continually adapted and thus a desirable solution should apply to varying models. This makes it difficult to apply existing solutions that deal with a similar problem. For instance, out-of-distribution (OOD) detection studies [11, 19, 20, 43] are built on the assumption that a model is fixed at test time, and open-set domain adaptation (OSDA) methods [30, 35, 42] require labeled source and unlabeled target data for training.
2We borrowed the term Near and Far from the OOD detection benchmark [41].
Methodology overview. To address the problem, we propose Screening-out Test-Time Adaptation (SoTTA), whose overview is described in Figure 4. SoTTA achieves robustness to noisy samples in two perspectives: (i) input-wise robustness via high-confidence uniform-class sampling that avoids selecting noisy samples when updating the model (Section 3.1), and (ii) parameter-wise robustness via entropy-sharpness minimization that makes parameters resilient to weight perturbation caused by noisy samples (Section 3.2).
3.1 Input-wise robustness via high-confidence uniform-class sampling
Our first approach is to ensure input-wise robustness to noisy samples by filtering out them when selecting samples for adaptation. As locating noisy samples without their labels is challenging, our idea is based on the empirical observation of the model predictions with respect to noisy samples.
Observation. Our hypothesis is that noisy samples have distinguished properties from benign samples due to the distributional shifts, and this could
(a) Confidence distribution.
(b) Predicted class distribution.
Figure 5: Model predictions on benign (CIFAR10-C) and noisy samples.
be observable via the models' prediction outputs. We investigate two types of features that work as proxies for identifying benign samples: (i) confidence of the samples and (ii) predicted class distributions. Specifically, we compared the distribution of the softmax confidence (Figure 5a) and the predicted class distribution (Figure 5b) of benign samples with noisy samples. First, the confidence of the samples is relatively lower than that of benign samples. The more severe the distribution shift, the lower the confidence (e.g., Far is less confidence than Near), which is also in line with findings of previous studies that pre-trained models show higher confidence on target distribution than out-of-distribution data [10, 21]. Second, we found that noisy samples are often skewed in terms of predictions, and this phenomenon is prominent in more severe shifts (e.g., Noise), except for Attack, whose objective is to make the model fail to correctly classify. These skewed distributions could lead to an undesirable bias in  $p(y)$  and thus might negatively impact the TTA objective, such as entropy minimization [38].
Algorithm 1 High-confidence Uniform-class Sampling (HUS)
Input: test data stream  $\mathbf{x}_t$  , memory  $M$  with capacity  $N$    
for test time  $t\in \{1,\dots ,T\}$  do   
 $\hat{y}_t\gets f(\mathbf{x};\Theta)$    
if  $C(\mathbf{x};\Theta) > C_0$  then  $\triangleright$  Sampling confident data   
if  $|M| <   N$  then Add  $(\mathbf{x}_t,\hat{y}_t)$  to  $M$    
else   
 $\mathcal{V}^*\gets$  the most prevalent class(es) in  $M$    
if  $\hat{y}_t\notin \mathcal{V}^*$  then Balancing classes   
Randomly discard  $(\mathbf{x}_i,\hat{y}_i)$  from  $M$  where  $\hat{y}_i\in \mathcal{V}^*$    
else   
Randomly discard  $(\mathbf{x}_i,\hat{y}_i)$  from  $M$  where  $\hat{y}_i = \hat{y}_t$    
Add  $(\mathbf{x}_t,\hat{y}_t)$  to  $M$
Solution. Based on the aforementioned empirical analysis, we propose High-confidence Uniform-class Sampling (HUS) that avoids using noisy samples for adaptation by utilizing a small memory. We maintain confident samples while balancing their predicted classes in the memory. The selected samples in the memory are then used for adaptation. We describe the procedure as a pseudo-code code in Algorithm 1. Given a target test sample  $\mathbf{x}$ , HUS measures its confidence. Specifically, we define the confidence  $C(\mathbf{x};\Theta)$  of each test sample  $\mathbf{x}$  as:
$$
C (\mathbf {x}; \Theta) = \max  _ {i = 1 \dots n} \left(\frac {e ^ {\hat {y} _ {i}}}{\sum_ {j = 1} ^ {n} e ^ {\hat {y} _ {j}}}\right) \text {w h e r e} \hat {y} = f (\mathbf {x}; \Theta). \tag {1}
$$
We store the sample if its confidence is higher than the predefined threshold  $C_0$ . In this way, we maintain only high-confidence samples in the memory used for adaptation and thus reduce the impact of low-confidence noisy samples.
Furthermore, while storing data in the memory, we balance classes among them. Specifically, if the predicted class of the current test sample is not in the most prevalent class in the memory, then HUS randomly replaces one random sample in the most prevalent class with the new sample. Otherwise, if the current sample belongs to the most prevalent class in the memory, HUS replaces one random sample in the same class with the current one. We can maintain classes uniformly with this strategy, which is effective for not only filtering out noisy samples but also removing class biases among samples when used for adaptation, which we found is beneficial for TTA. We found these two memory management strategies not only effectively reduce the impact of noisy samples for adaptation but also improve the model performance in benign-only cases by avoiding model drifting due to biased and low-confidence samples (Section 4).
With the stored samples in the memory, we update the normalization statistics and affine parameters in batch normalization (BN) [13] layers, following the prior TTA methods [29, 38, 44]. This is known as not only computationally efficient but also showed comparable performance improvement to updating the whole layers. While we aim to avoid using noisy samples for adaptation, a few noisy samples could still be stored in the memory, e.g., when they are similar to benign samples or outliers. To be robust to temporal variances of the samples in the memory, we take the exponential moving average to update the BN statistics (means  $\mu$  and variances  $\sigma^2$ ) instead of directly using the statistics from the samples in the memory. Specifically, we update the means and variances of BN layers as: (i)  $\hat{\mu}_t = (1 - m)\hat{\mu}_{t-1} + m\mu_t$  and (ii)  $\hat{\sigma}_t^2 = (1 - m)\hat{\sigma}_{t-1}^2 + m\sigma_t^2$ , where  $m \in [0,1]$  is a momentum hyperparameter. We describe updating the affine parameters in Section 3.2.
3.2 Parameter-wise robustness via entropy-sharpness minimization
Our second approach is to secure parameter-wise robustness to noisy samples by training the model in a way that is robust to noisy samples. Our idea is based on the observation that the parameter update is often corrupted with noisy samples, and this could be mitigated by smoothing the loss landscape with respect to parameters.
Observation. While most existing TTA algorithms utilize the entropy minimization loss [5, 38, 44], it could drift the model with high gradient
(a) Without ESM.
(b) With ESM.
Figure 6: Cumulative accuracy  $(\%)$  of benign samples (CIFAR10-C) and gradient norm of noisy samples (Noise) as the adaptation proceeds. The dotted line refers to the source model accuracy.
samples [29]. We observed that adaptation with noisy samples often hinders the model from adapting to benign samples. Figure 6 shows an example of this phenomenon. Specifically, test samples consist of  $10\mathrm{k}$  benign samples and  $10\mathrm{k}$  noisy samples (Noise), which are randomly shuffled. We computed the cumulative accuracy for the benign test data and the gradient norm of noisy samples at each step. As shown in Figure 6a, the gradient norm of noisy samples dropped rapidly, indicating that the model is gradually adapting to these noise data. This leads to a significant accuracy drop for benign samples. The key question here is how to prevent the model from overfitting to noisy samples. Figure 6b shows the result with entropy-sharpness minimization (ESM) that we explain in the following paragraph. With ESM, the gradient norm of noise data remains high, and the accuracy for benign samples improved after adaptation as intended.
Solution. To make the model parameters robust to adaptation with noisy samples, the entropy loss landscape should be smoother so that the model becomes resilient to unexpected model drift due to noisy samples. To that end, we jointly minimize the naive entropy loss and the sharpness of the entropy loss and thus make the loss landscape robust to model weight perturbations by large gradients from noisy samples. Specifically, we replace the naive entropy minimization  $E$  with the
entropy-sharpeness minimization (ESM) as:
$$
\min  _ {\Theta} E _ {S} (\mathbf {x}, \Theta) = \min  _ {\Theta} \max  _ {| | \epsilon | | _ {2} \leq \rho} E (\mathbf {x}; \Theta + \epsilon), \tag {2}
$$
where the entropy-sharppiness  $E_{S}(\mathbf{x},\Theta)$  is defined as the maximum objective around the weight perturbation with L2-norm constraint  $\rho$ . To tackle this joint optimization problem, we follow sharpness-aware minimization [4] similar to [29], which originally aims to improve the generalizability of models over standard optimization algorithms such as stochastic gradient descent (SGD). We repurpose this optimization to make the model robust to noisy samples in TTA scenarios.
Specifically, assuming  $\rho \ll 1$ , the optimization can be approximated via a first-order Taylor expansion:
$$
\epsilon^ {*} (\Theta) \triangleq \underset {| | \epsilon | | _ {2} \leq \rho} {\arg \max } E (\mathbf {x}; \Theta + \epsilon) \approx \underset {| | \epsilon | | _ {2} \leq \rho} {\arg \max } \epsilon^ {T} \nabla_ {\Theta} E (\mathbf {x}; \Theta). \tag {3}
$$
The solution for this approximation problem is given by the classical dual norm problem:
$$
\hat {\epsilon} (\Theta) = \rho \operatorname {s i g n} \left(\nabla_ {\Theta} E (\mathbf {x}; \Theta)\right) | \nabla_ {\Theta} E (\mathbf {x}; \Theta) | ^ {q - 1} / \left(\left| \left| \nabla_ {\Theta} E (\mathbf {x}; \Theta) \right| \right| _ {q} ^ {q}\right) ^ {1 / p}, \tag {4}
$$
where  $1 / p + 1 / q = 1$ . We set  $p = 2$  for further implementation, following the suggestion [4]. By substituting  $\hat{\epsilon} (\Theta)$  to the original entropy-sharps minimization problem, the final gradient approximation is:
$$
\nabla_ {\Theta} E _ {S} (\mathbf {x}, \Theta) \approx \nabla_ {\Theta} E (\mathbf {x}, \Theta) | _ {\Theta + \hat {\epsilon} (\Theta)}. \tag {5}
$$
In summary, we calculate the entropy-sharps minimization objective via two steps. First, at time step  $t$ , it calculates the  $\hat{\epsilon}(\Theta_t)$  with previous parameters and entropy loss. It generates the temporal model with the new parameters:  $\Theta_t + \hat{\epsilon}(\Theta_t)$ . Second, based on the temporary model, the second step updates the original model's parameters with the approximation in Equation 5. By putting it together with HUS (Section 3.1), parameters are updated by:
$$
\Theta_ {t} = \Theta_ {t - t _ {0}} - \eta \nabla_ {\Theta} E (\mathbf {x}, \Theta) | _ {\mathbf {x} = \mathrm {H U S} _ {C _ {0}} (t)}, \Theta = \Theta_ {t - t _ {0}} + \dot {\epsilon} \left(\Theta_ {t - t _ {0}}\right), \tag {6}
$$
where  $\eta$  is the step size and  $t_0$  is the model adaptation interval. For simplicity, we set the model adaptation interval the same as the memory capacity, i.e.,  $t_0 = N$ . In the experiments, we use  $N = 64$ , which is one of the most widely-used batch sizes in TTA methods [36, 38, 44].
This section describes our experimental setup and demonstrates the results in various settings. Please refer to Appendix A and B for further details.
Scenario. To mimic the presence of noisy samples in addition to the original target test samples, we injected various noisy datasets into target datasets and randomly shuffled them, which we detail in the following paragraphs. We report the classification accuracy of the original target samples to measure the performance of the model in the presence of noisy samples. We ran all experiments with three random seeds (0, 1, 2) and reported the average accuracy and standard deviations in Appendix B.
Target datasets. We used three standard TTA benchmarks: CIFAR10-C, CIFAR100-C, and ImageNet-C [9] as our target datasets. All datasets contain 15 different types and five levels of corruption, where we use the most severe corruption level of five. For each corruption type, the CIFAR10-C/CIFAR100-C dataset has 10,000 test data with 10/100 classes, and the ImageNet-C dataset has 50,000 test data with 1000 classes. We use ResNet18 [8] as the backbone network. We pre-trained the model for CIFAR10 with training data and used the TorchVision [23] pre-trained model for ImageNet.
Noisy datasets. Besides the target datasets (Benign) mentioned above, we consider four noisy scenarios (Figure 3): CIFAR100 [15]/ImageNet [3] (Near), MNIST [26] (Far), adversarial attack (Attack), and uniform random noise (Noise). As both CIFAR10-C and ImageNet-C have real-world images for object recognition tasks, CIFAR100 would be a near dataset to them. For CIFAR100-C, we select ImageNet [3] for a near dataset. We select MNIST as a far dataset because they are not real-world images. We referred to the OOD benchmark [41] for choosing the term Near and Far and the datasets. For the adversarial attack, we adopt the Distribution Invading Attack (DIA), which is an
adversarial attack algorithm designed for TTA [40]. We inject the small perturbations to malicious samples to increase the overall error rate on benign data in the same batch. For the uniform random noise, we generate pixel-wise uniform-random images with the same size as the target images. We set the number of noisy samples equal to each target dataset as default, and we also investigated the impact of the number of noisy samples in the following ablation study (Figure 7). In cases where the number of noisy samples is different from the target datasets, we randomly resampled them. To ensure that the learning algorithm does not know the information of noisy samples beforehand, the pixel values of all noisy images are normalized with respect to the target dataset.