PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Authors: Jing Ma
Hanlin Li
Xiang Xiang
Venue: ICML 2025 poster (2025)
Keywords: Test-Time Adaptation
Malicious Sample Purification
Primary area: general_machine_learning->transfer_multitask_and_metalearning
TL;DR: Rather than selecting and discarding malicious samples for test-time model adaptation, why not purify them into benign ones?

Abstract: Test-Time Adaptation (TTA) enables deep neural networks to adapt to arbitrary distributions during inference. Existing TTA algorithms generally tend to select benign samples that help achieve robust online prediction and stable self-training. Although malicious samples that would undermine the model's optimization should be filtered out, it also leads to a waste of test data. To alleviate this issue, we focus on how to make full use of the malicious test samples for TTA by transforming them into benign ones, and propose a plug-and-play method, PTTA. The core of our solution lies in the purification strategy, which retrieves benign samples having opposite effects on the objective function to perform Mixup with malicious samples, based on a saliency indicator for encoding benign and malicious data. This strategy results in effective utilization of the information in malicious samples and an improvement of the models' online test accuracy. In this way, we can directly apply the purification loss to existing TTA algorithms without the need to carefully adjust the sample selection threshold. Extensive experiments on four types of TTA tasks as well as classification, segmentation, and adversarial defense demonstrate the effectiveness of our method. Code is available at https://github.com/HAIV-Lab/PTTA.

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Jing Ma
Hanlin Li
Xiang Xiang 122
Abstract
Data Stream
t == 2
t t=
Test-Time Adaptation (TTA) enables deep neural networks to adapt to arbitrary distributions dur- ing inference. Existing TTA algorithms generally tend to select benign samples that help achieve robust online prediction and stable self-training. Although malicious samples that would under- mine the model's optimization should be filtered out, also leads to waste of test data. To allevi- ate this issue, we focus on how to make full use of the malicious test samples for TTA by trans- forming them into benign ones, and propose a plug-and-play method, PTTA. The core of our solution lies the purification strategy, which retrieves benign samples having opposite effects the objective function to perform Mixup with malicious samples, based on saliency indicator for encoding benign and malicious data. This strategy results in effective utilization of the in- formation in malicious samples and an improve- ment of the models' online test accuracy. this way, we can directly apply the purification loss to existing TTA algorithms without the need to carefully adjust the sample selection threshold. Extensive experiments on four types of TTA tasks as well as classification, segmentation, and ad- versarial defense demonstrate the effectiveness of our method. Code available https: / //github. Ceco//HAIV-Lb//BA
Sample Selection
X
X
+ ptTa
Test Time Adaptation
Benign Samples
Malicious Samples
Purified Samples
Figure I: test-time model adaptation using the proposed PTTA.
Illustration of purifying malicious test samples for
of pretrained models. TTA approaches break free from the curse of i.i.d. assumption and explore the adaptive potential of widely used models in unseen scenarios. A series of works have been proposed to achieve the stable and robust prediction of Deep Neural Networks (DNNs) in complex and dynamically changing environments (Wang et al., 2021; Niu et al., 2022a; Lee et al., 2024; Chen et al., 2024).
Since TTA algorithms seek to avoid human supervision and intervention, most of them adopt self-supervised tasks, such as Entropy Minimization (Wang et al., 2021) and Con- sistency Learning (Wang al., 2022). This leads to the amplification of models' biases and catastrophic forgetting of original knowledge during continuous adaptation (Zhang et al., 2024: Niu et al., 2022a). In response to this challenge, recent studies have shifted their focus towards selecting be- nign samples useful for TTA algorithms from the test data (Niu et al., 2022a: Lee et al., 2024; Zhang et al., 2024). However, existing approaches incorporate sample selection criteria filter and discard portion of test samples that could otherwise be utilized. This practice results in data waste for TTA tasks, which grapple with data scarcity is- sues. Simultaneously, previous TTA methods encounter a trade-off between stable optimization and comprehensive use of test samples. This trade-off necessitates the cautious selection of hyperparameters and thresholds of the criteria.
1. Introduction
Test-Time Adaptation (TTA) is way that goes beyond large-scale and heavy pre-training, and utilizes unlabeled test samples enhance the online prediction performance
Equul contribution, co-first author. NNational Key Lab Multi- Spectral Info. Intelligent Processing Tech., School of Artificial Intelligence and Automation, Huazhong University of Science and Tech. (HUST), Wuhan, China. 'Peng Cheng National Lab, Shen- zhen, China. "Scoool of Computer Science and Technology, HUST, China. Correspondence to: Xiang Xiang <xex @ whwst.cdu.cnt
Proceedings of the ndnd International Conference Machine Learning, Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
45
feature, and logit levels. The saliency indicator is also an efficient retrieval metric for retrieving benign samples in memory bank or within the mini-batch. Based on this indi- cator, we design a pipeline of benign sample retrieval and malicious sample purification, as shown in Fig. 1. Specifi- cally, we employ existing sample selection criteria for TTA to distinguish benign and malicious samples (Niu et al., 2022a: Lee et al., 2024; Zhang et al., 2024). Then we re- trieve the counterpart from benign samples that has the most opposite effect on the objective function compared to a malicious sample. Two matched samples are Mixuped to calculate the purification loss. In this way, we effectively use the malicious samples that should have been filtered out, thereby improving the utilization rate of test data. We demonstrate the effectiveness of PTTA and its universality for different DNNs through extensive experiments, includ- ing episodic, single, continual, and lifelong TTA tasks, as well as image classification, semantic segmentation, and adversarial defense tasks.
40
50
35
Accuracy
30
Accuracy
40
PTTA
25
Noddapi
30
20
20
-H0-
Tent
15
PTTA
10
Nodelagt
10
40
70
100
10
40
70
100
Proportion
Proportion
Figure 2: We vary the proportion of selected benign (low-entropy) samples and plot the test accuracy curve of ResNet50 (Ieft) and ViT-B/16 (right).
To illustrate this issue, we conduct a pilot experiment to empirically demonstrate the deficiency of sample selection. Specifically, we employ ResNet50 and ViT-B/16 pretrained on ImageNet- I to conduct experiments on continual TTA task constructed from ImageNet-C. The results, shown in Fig. 2, reveal that the threshold for benign sample selection controls the testing accuracy of DNNs. A minor change in the threshold will lead to severe performance degradation, e.g., 36% accuracy drop when proportion changes from 0.6 to 0.7 in Fig. 2 (right). High sensitivity to thresholds limits the scalability of existing TTA algorithms.
In summary: 1) cious test samples for TTA and propose a saliency indicator to encode benign and malicious samples effectively; 2) We put forward plug-and-play solution, PTTA, to purify mali- cious samples by using retrieved benign samples that have the opposite impact on the objective function. It improves the utilization rate of test data and enhances models' test accuracy, while reducing sensitivity to the thresholds; and 3) We conduct extensive experiments to demonstrate the generality of PTTA across various tasks and models, as well as its expandability on existing TTA algorithms.
We analyze the hazards of learning mali-
Although the filtered malicious samples' may undermine the stability of TTA algorithms, they nonetheless reflect the distribution of the test data. If it is possible to transform ma- licious samples into benign ones and subsequently make full use of them, the optimization efficiency can be enhanced. Motivated by adversarial purification (Tang & Zhang, 2024; Shi et al., 2021), we aim to utilize benign samples to purify the malicious ones. Instead of adding noise to the pixels, we retrieve for a malicious sample the counterpart that has an opposite impact on the objective function. We employ the simple Mixup technique (Zhang et al., 2018) to accom- plish malicious sample purification, due to its advantages of improving the calibration of DNNs and reducing over- confident predictions. We also demonstrate the benefits of our proposed Purification for Test-Time Adaptation (PTTA) in Fig. 2. We find that the performance of existing TTA algo- rithm is significantly improved by PTTA, and its sensitivity to the threshold is reduced. In addition, the test accuracy of the model grows as the increase the utilization rate of test samples, as shown in Fig. 2 (right).
2. Preliminaries
Notations. Considering the C-way classification, we define the label space Y E Rº and the sample space X E R. Following the assumptions of existing TTA methods, we assume that all unlabeled samples x E X arriving at test time could be categorized into one of the known classes in the label space, i.e., y(z) E Y. We define a Deep Neural Network f and its parameters 0. The TTA task follows the paradigm of online learning. That is, at any time t, only Nos unlabeled samples x, = f2 ) are available. The predicted output of the model f for sample x is defined as logits z f(x)= = [=1,..,,cc] Generally, Softmax function a--) is applied to compute the probability vector p of [pI..... Pc]. A widely used approach is to decompose the DNN f into feature extractor & and a classifier The n-dimenssonal features extracted by o are called latent code defined as h = d(x) = [h..... h..)
In this paper, we propose a plug-and-play solution to purify malicious test samples. To accurately encode the impacts of benign and malicious samples on TTA, we propose saliency indicator, which measures the contribution of an individual test sample to the objective function at the pixel,
Entropy Minimization. Shannon entropy (Shannon, 1948) is often used measure the confidence of model predictions. Wang et al. (2021) establish connection between entropy and the error rate, and show that more confident predictions
Different from defining malicious samples in adversarial de- fense those attacked by adding noise, we define malicious sam- ples as those filtered out by sample selection criteria. Essentially, both definitions refer to the samples that undermine TTA methods.
2

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
are all-in-all correct. On top of this, Entropy Minimization is taken the optimization objective function for TTA:
75
ResNet50
ViT-B/16
Accuracy
50-
Ny
min
[S(un))86) (T 0) (5 0),
25
Nbs
==
(1)
1-10%%
10-20%
20 30%
30-46%
40-30%
50-600%
60--00%
70 70-90%
S0 900-90%
90. 100%
where is the time step. Minimizing the conditional entropy reduces the overlap of output class distributions, leading to the low-density separation between classes.
40
Proportion of Entropy
60-
ResNet50
Sample Selection Criteria. Not all test samples are helpful for TTA. Therefore, selecting informative benign
50-
ViT-B/16
40
Accuracy
samples is crucial, which is denoted as an indicator function ((S(x)) > } for reweighting in Eq. 1, where S(-) is scoring function for each sample, and So is preset thresh- old. A series of selection criteria has been proposed. ETA (Niu et al., 2022a) selects diverse samples with low entropy, ie.. (x))) < Ento] n [[Cos( (ars), m² m' < <e). De (Lee et al., 2024) introduces the PLPD and add criterion, Le., (PLPD(()))) > r). CPL (Zhang et al., 2024) proposes the intra- and inter-instance label selection, ie., 1 E MinSize( (il {iاE) Ei (x.))) 2 +})} U 1(j (1m) > These sample selection criteria divide test data into benign and malicious samples, and then filter out the malicious ones, resulting in data waste. In this paper, we will demonstrate how to use purification to turn malicious samples into benign ones so as to make full use of test data.
30
20
10
1.5
2.0
2.5
3.0
3.5
4.0
4.5
5.0
Average Entropy
Figure 3: We test the accuracy of top-p% low-entropy sam- ples (top), and visualize the relationship between the aver- aged prediction entropy and the test accuracy of DNNs for 15 types of out-of- -ittr--dittiitttonn domains (down).
(Sor)pp
ically treated as hyperparameters) in the criterion, where slight variations lead to substantial performance degradation. Through these observations, it is intuitive to believe that malicious samples exist in an uncertain ratio within mini- batches, and existing sample selection criteria fail to com- pletely eliminate them from test data. This constitutes an- other form of malicious sample hazards that fundamentally constrains the practical applicability of TTA algorithms.
3. Method
3.1. Malicious Sample Hazards
Instead of designing criteria to detect and exclude malicious samples, we focus on how to transform malicious samples into benign ones. We name this transformation the malicious sample purification and propose PTTA, a plug-and and-play method, to improve existing TTA algorithms for test data utilization. The pipeline is shown in Fig. 4.
Vanilla Test-Time Adaptation (TTA) algorithm treats all the test samples equally. However, according to the model's predictions, samples with high conditional entropy are less accurate than those with low entropy. We verify the relation- ship between the classification accuracy and the conditional entropy of ResNet50 and ViT-B/16 pre-trained on ImageNet- IK for 15 types of out-o--distribution samples in ImageNet- C, shown in Fig. 3 (top). Applying Entropy Minimization for high-entropy samples does not bring benefits to TTA algorithms. Instead, it will undermine the stability of DNNs, especially when test data distribution has a large shift from the source training distribution. As shown in Fig. 3 (down), DNNs inherently have relatively high conditional entropy for test domains with low classification accuracy.
3.2. Saliency Indicator
In this subsection, introduce an indicator for encoding benign and malicious samples, motivated by adversarial purification. The concept of adversarial purification stems from adversarial defense (Tang & Zhang, 2024; Shi et al., 2021). The purpose of purification is to add noise y of a specified magnitude & at the pixel level of an image x to minimize the loss of the defense objective function, which is defined as 7 = arg minl/yllese C(fe (z + 77), (z))). The noise y is generally calculated as the partial derivative with respect to image 2,, i.e., 7 È sign(V,_[Joll (2)), v(x)). This first-order derivative C. called saliency information, is used to judge how pixels in the image affect the model's final prediction, and it also indicates the importance of each pixel (Simonyan et al., 2014) Another interpolation is that
simple solution is sort the test samples (within a mini- batch) in ascending order according to the conditional en- tropy, and then take the top-p% low-entropy samples, called benign samples, to apply Entropy Minimization. This sam- ple selection criterion can effectively improve the perfor- mance of vanilla TTA algorithm, as shown in Fig. 2. How- ever, this success exhibits high sensitivity to thresholds (typ-
3

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Test Samples
Saliency Indicator & Distance:
D2 Crnine(V,<Eaulddee)) (=fzu(ff(r,)))
Loss Function
Benign Sample Retrieval:
Model
Listal = Liaa + OeC
sj argmax (F 18gSNaa
Purified Samples
Malicious Sample Purification:
Benign Sample Selection
z/g = Az + (1 Ajmj, Bi Aùj + (1 X)øī
Test Samples
Sample Store & Retrieval
Renign Samples
Memory Bank or In-Batch
Forward
Benign Sample Retrieval
Purified Samples
Purification
Forward
Figure 4: The pipeline of Purification for Test-Time Adaptation (PTTA). We purify malicious test samples using benign samples retrieved from memory bank or within the mini-batch, based on the saliency indicator we proposed for encoding benign and malicious data. The purified samples are used to calculate the purification loss Cpur. which directly inserted into the original objective function Lua of existing TTA methods.
it measures which individual pixels need to be changed the most in order minimize the objective function. Conse- quently, samples contributing similarly to the objective func- tion exhibit aligned directions of their saliency information. Therefore, the saliency information of Entropy Minimiza- tion (a widely adopted objective functiin TTA methods) with respect to test sample v, i.e., Von (fo())) can effectively used for purification. We further demonstrate the flexibility in choosing objective functions in Sec. 4.3.
ages, it reduces the interference of irrelevant pixels. Mean- while, the features possess semantic information and more closely related to the model's final decisions.
Logit-saliency indicator employs V C as the saliency information. Vi can be directly calculated without the need of backpropagating gradients, i.e.,
V V=Cem = - - ())
O Op.
(4)
Intuitively, the logit-level saliency represents the changes in the logits output of the model after optimization with respect the sample x. It directly encodes the impact of test samples on the model's final decisions.
However, instead of directly superimposing noises on pixels for purification, we utilize noises to encode the gradient di- rection objective functions for test samples. Specifically, these unit noise vectors (with a magnitude of 1.0) indicate the direction in which the entropy of model's predictions decreases the fastest in the sample space. Since gradients corresponding to benign samples x+ and those correspond- ing to malicious samples x= have opposite directions, i...,
We conduct ablation study on these three indicators, as shown in Table 1. The logit-saliency indicator generally ex- hibits better performance and lower computational overhead. Meanwhile, we also compare with adversarial purification methods, which directly superimpose noises to malicious samples via FGSM (Tang & Zhang, 2024) or SOAP (Shi et al., 2021). Adversarial purification methods fail to im- prove and even impair the performance of TTA algorithms.
Cosine(V VV (fo(xx) V.LTm((Jo (x)xx)))))))) < O. (2)
Therefore, we employ the saliency information as an indica- tor for encoding benign and malicious samples. Meanwhile, we calculate the cosine distance of the saliency indicator to retrieve two samples r; and z; that have the most opposite contributions to objective functions as follows:
3.3. Purification for Test-Time Adaptation
In this subsection, we elaborate on how perform malicious sample purification and scale it plug-and-play method to existing TTA algorithms. Put it briefly, we mainly adopt the Mixup technique (Zhang et al., 2018) to merge benign and malicious samples, which serves as a data augmenta- tion to make up for the discarding of malicious samples by sample-selection
Daa = 1 Cosine( V (f vaLem(fo(zz)))
(3) Besides using the gradients of pixels, the gradients of fea- tures and logits can also provide saliency information. We investigate these three indicators the following.
TTA algorithms. While simple to implement, Mixup has been shown improve the cali- bration of DNNs, i.e., the predieted Softmax probabilities can better indicate the actual likelihood of correct predic- tion, and reduce the over-confident predictions of DNNs on out-of -ut-o--distribution data (Thulasidasan et al., 2019).
Pixel-saliency indicator uses Lent as the saliency infor- mation. It encodes the contribution to objective functions in the high-dimensional sample space, which preserves spatial or temporal information. But it is vulnerable to pixels that are irrelevant to the model prediction.
Feature-salient indicator exploits VaLem as the saliency information. The feature extractor compressive encodes im-
Benign Sample Retrieval. Next, we focus on determining the retrieval scope and retrieving using saliency indicator.

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Table 1: We demonstrate the generality of PTTA across various sample selection criteria. Meanwhile, we compare the performance of PTTA with different saliency indicators, types of retrieval, and sample purification scopes. We count and compare the number of forward (#FWds) and backward (#Bwds) propagations for different methods.
SINGLE
CONTINUAL
METHODS
ETA
DEYO
CPL
AVG.
ETA
DEYO
CPL
AVG.
#WWDS
#BWDS
COMPARISON WITH EXISTING PURIFICATION (PURIE) METHODS.
BASE ALGORITIM
48. 2e0.04
48. 6.0.11
820.04
46.5
62.2
62.2
33
60.0
Nbstch
Naatch
PURIF. FGSM
01
47.0--.0
4..
45.0
61.7-0as
62.2-0.19
56.4-0.8
60.1
Noutch X
Noutch X 2 Nortch X 6
PURIE W/ SOAP (5 ITER)
47.4
47.7a0.25
61.610.8
43.9
57.0.748
64.1a
49.1
56.7
Nhatch X 6
ABLATION STUDY OF DIFFERENT SALIENCY INDICATORS (IND.).
PIXEL--SAL SALL IND. SALL IND. + LOLII--ALL IND.
49.0se.m
50.0.o.0e
43.0.0.0s
47.3
65.6.08.2
66.0an.o
59.3...s
63.6
Nbatch X 2
Nostch X 2
49.000.04
49. 920.2
43.420005
47.5
65 65.45000
55.8n.02
58. 28..00.06
63.3
Nautch X 2
Nostch X 2
49.5g0.00
49. 10
44.6
48.0
65.3-0.06
65, 8.0.01
59.6.0.0
63.5
Nontch X 2
Noatch
ABLATION STUDY OF DIFFERENT
TYPES OF RETRIEVAL. (RET.).
ID RET. (Ni.. = 1A)
2.0.25
48, 320.22
1a..1
46.5
64.4
63.3-0.49
58. 28.2
62.0
Noutch X 2
Noutch
+OO RET. (Nms = 1k)
49.5.0.00
49.94a
44.6+0
48.0
65.3-0.06
65.8-0.01
59.6-0.a
63.5
Nintch X 2
Nhatoh Naatoh
+ +IB RET. (Nab 0)
49.3e0.o
50.0s2.44
43.6
47.6
65.5
65.8
59. L00.0
63.5
Naatch X 2
ABLATION STUDY OF DIVERENN SAMPLE PURIFICATION SCOPES.
+ PURIE. ALL SAMPS. PURIF, MALL SAMPS.
49.5e0.00
49.9 49.9
44.6.0.0 44.3.0.0 os
48.0
65.3
65.8sn.01
59. 19.66.n.1
63.5
Nhatek x 2
Naatoh
40.02
49. 09.00.0.8
47.7
7.0.0
65.340.06
59. 0..00.. 0.
63.0
Noutch x 2
Nautck
In-distribution (ID) retrieval suggests that the samples within source training distribution could be used as benign samples to purify out-of-distribution samples at test time, Specifically, we randomly choose Nub ID samples, stored in a fixed memory bank {æ,zūū; V Lem (Iz)) "-ر where their probabilities and saliency informations are computed offline. We then calculate the saliency distance D.. (Z;, Iry) between a online test sample and ID samples, and select one ID sample with the largest distance to perform Mixup.
where i is the model's predicted probability for x;. We set x= 1/(K + 1), where к decides that the top- samples with the largest saliency distance are retrieved from the scope. We uniformly set K = 1 and conduct an ablation study on the value of K in Sec. 4.4.
Built upon the original objective function Lua of existing TTA algorithms, we add purification loss as
1 No. C
6pur
Nos ΣEwigh log (o((t()t)))
Out-of duttof-distribution (OOD) retrieval. However, model fo will gradually adapt to the test data distribution in TTA tasks, while forgetting the source data (Niu et al., 2022a). At time step I, utilizing the benign test samples (x, J Jizto within a historical period [t - a, t] is beneficial to current model. Therefore, we define the memory bank as a first-in-first-or queue and update it with x," We retrieve benign samples in the memory bank by using the saliency distance.
Note that we purify all test samples in the mini-batch for Cpar- considering the performance improvement brought by greater amounts of augmented data. We experimentally demonstrate that it benefits our proposed PTTA in Table 1.
Overall, the total loss function is defined as
In-batch (IB) retrieval retrieves within benign samples xf in the mini-batcha at time step i without additional memory bank. One drawback is that it limits the retrieval scope.
Eaoial Lus + atCpurs
(8)
where is hyperparameter to balance the two loss func- tions, and we conduct an ablation study on in Sec. 4.4.
We compare three retrieval methods in Table 1. Refer to Appendix E.2 for detailed ablation studies.
4. Experiments
Malicious Sample Purification. For a test sample Fi, we retrieve a matched benign sample within a scope {1y))/m.
Nms.
4.1. Setups
We describe the benchmarks used in the experiments, the baselines for comparison, and the implementation of meth- ods in the following. Refer to Appendix C for details.
arg max Dea (ea(21,2,)) 15J
(5)
where 2; is employed to purify the test sample xy. The purified sample ziy and its pseudo-label are obtained by using the Mixup technique (Zhang et al., 2018) follows:
Benchmarks. We focus on episodic, single, continual, and lifelong tasks, which are widely concerned in Time-Time Adaptation research. In episodic task, a single batch of test samples is used to optimize the model, and then the updated model makes predictions for current batch. After
z = Ax; + (1 - -A)a
Vig = Xū; + (1 - A)0G.
5

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Table 2: Experimental results (top-1 classification accuracy (%) on the lifelong TTA task.
RouND R-1
METHODS
R-2
R-3
R-4
R-5
R-6
R-7
R-8
R-9
R-10
AVERAGE
NOADAPT
31.6
31.6-000
31.6-0.0
31.6-000
31,6:0.0
31.6+0.0
31.6.a.
31.6x0.c
31 11.6x0.0 on
31.6-000
31.6-000
TENT
8. 8.1
0.0.0000
0.1
0. 1.0000
a.1
2.100000
0.1
1.100000
0.1.0000
0.940.01
-PTTA
60.0an.8
31.8
0.1.0.00
0. 1.1.0
0.1an.o
0.100
0.1am0p
0. La0o
0.
0.1at
9.3a0.02
COTTA
45.9
40. 57
37. 37.041.5
34, 83
32.0kom
31 31.0ka.m
30.5
30. 20..9
30. 6.0
34. 57
PTTA
46.3
42. 8:0.s
09.2.0.0
810.01
38. 44
38, 2.. 03
0.00... 05
41.2.002
SOTTA
59.5.m.22
7
61. 1an.1s
61.5a0.13
61.526.19
61.628.15
61.7a0.15
61. 61.8pn.12
61 61.920.20
61.3
-PTTA
61.3am..9
62. b.n.2
63. Lanse
63.5am227
63.74o.a
63..4
63,9u0.15
63. 83.9
64.0wi24
SAR
60. 0.
61. 1-000
61 0z
61.6-om
61. 7:
61.8
61.7
61. 6.o.ce
59. 39.3.0.20 26
60. 0-
61. 1-000
PPTA
61 6:0.81
63. 0.000
63.4 03.40000
63. 02
63.850.2
63.928.01
64. O29.02
64 020.02
6..1s0.00
64. 1s0.02
63.5a00:
ETA
62. 2wner
59. 13.13
55.4an39
46. 64
31.5a
29.5a25
28.2
26:2.
16.1
25.1az.
39.0a15.2
PTTA
65.3.0.00
65.3.m
65. L-om
64.9kam
64.7.om
64.61a.m
64.4.0.0
64.3
64. 1.0.0
6..8.0
BATA
62. 62.5.0.4
39
37
61.7
61. 4s
61, 44044
31.3.
61. 21.220..8
0s0.42
0.0
61.62
PTTA
64.7a0.31
65.02
65.0an.39
65.1an.4
65.0s0.
65.0a8.45
65 65.020.43
65.020.3
65.0a0:1
65. 45.0..
DEYO
62. 02.000.50
43. Lasse
32. 122.0
25. 2
17. 17.9km07
16.8
15. 15.9
3.5.x
0. Lam
0. 1a000
21.7.15s
PTTA
65.8.a8
66.0.001
66.0-0ai
65.9.005
65.9.a
m
65.7
65.5 07
64.9
61.6
65.3-0
CPL
55.5
$7.3 11
58. Ls
58.4
58.5.0m
58.5
58.7.0.1
58. 38.8.0
58. 13
PTTA
59.7.0.02
61 1..50.0
62, 020.0
62.3-002
62.6.0.0
62.7:0.0
62.9t0.0
63.
63.0s0.00
63. 10.01
63.2-0.0
62.3-00)
Table 3: Experimental results (top-1 classification accuracy (%) on the single and continual TTA tasks for ImageNet-C (left) and other ImageNet variants (right). We highlight the highest accuracy in bold and the second best as underline.
RESNETSO
VIT-B/16
IMAGENET
METHODS
SINGLE
CONTL.
SINGLE
CONTL.
METHODS
-1K
-A
-V2.
-R.
-S.
AvG.
NOADAPT
18.2s0.00
18.2.000
31.62000
31,620.00
NOADAPT
78.020.00
23.920.00
66.320.00
43.220.00
18.2an.co
45. 920.00
TENT
42.820.04
9.930.22
51.3
14.2a105
TENT
82. 0.a.ss
29. 29.0u0.r7
70.3.0.m
17.6.0.0
2.0x0.1
40.2
PTTA
44.940
33
52.
60.1
PTTA
82. 020.00
28.720.15
70.420.19
19.12
2.5
40.528.3
COTTA
32.91om:
33.3
38.5.n69
42.9
COTTA
79. 6
26. 850.64
68.320.16
53.9t0.58
13.8
48.5s1.71
PPTAA
54.0c0.33
37.6:0
48, 05
51.3
PTTA
81.0.0.00
26. 260..
1a.
22.3
50.3
SOTTA
41.5iom
30.6.0m6
57.1
59.5.m.1
SOTTA
82, 2,0.00
29. 6s0.5
70.4
$7.4
40. 8.n.30
56. 1aa.1
PTTA
42.92035
37.020.9
58.
61.320.5
PITA
82.
30.1.0.12
70.5w0.0
57. a...s
40.9 10.9an.1
S6.2sa.1
SAR
44.0s5.15
40.020:1
56. 19
60.020..4
SAR
8 020.00
30.0a0.14
70.3
43.3a0.18
631.51
49.623.36
+ PTTA
45.5
46, 26
58, 04
11..000.0
PTTA
82, 1.0.05
30.5.0.3
70.5.0.17
43.2.0.00
24.3
50.1 150.19
ETA
43.7-0m6
63. 03
62. 2.0.00
ETA
82.3
29. 29.8a1.9
71.3
56.520.50
45, 15.9
57. 17..2
PTTA
49.5
48. 48.5a00s
64.4
65.3
PTTA
82.7.0.02
32.742.47
71.720.0 06
61.4.0.0.
47.450.c
29..2
EATA
48. 020.00
48.42003
61. 51
62.2.0.53
EATA
1...00
31.4
70.8.0.06
57.7a0.11
44.
57.2 57.2sa.15
PTTA
49.1
50.3ane4
62.7
64.7a0.11
PTTA
81.9
30.8.0..2
1.0.1
59.9s0.3
45. 6.0.56
57.9
DEYO
48. 18.6
9.2.0.25
64.2
22.250.55
DEYO
82. 12..1
39. 40.34
71. 8.0.5
59.5.0.75
40. 8+1.30
58. 14..1
PTTA
49.9 10
95.7
65.32001
65.8s9.01
PTTA
82,
40.7.0.3
71.920.12
61.0m0.41
42, 433.8
59.750.57
CPL +PTTA
42 44.61c.0s
12.3-015
52, os
55.5
CPL
770.17
27.7
69.3.a 37
55. 1.0.11
39.5.0.17
54.5.0.07
45.3an
54.6.0
59.7am.0
PTTA
81.250.12
27.420.32
69.5e00.4
56.0
39.3
54.750.0
that, model's parameters are reset to the source. For single and continual tasks, the model is iteratively updated in static and dynamically changing environments respectively. Fur- thermore, lifelong task extends the dynamic environment indefinitely, set as 10 rounds and a total of 150 corruptions. Following previous TTA methods, we employ ImageNet-C, CIFAR100- (Hendrycks Dietterich, 2018), ImageNet (Deng et al., 2009) and its variants: -A (Hendrycks et al., 2021b), -V2. (Recht al., 2019), -R. (Hendrycks et al., 2021a), -S. (Wang et al., 2019) to construct these tasks. ImageNet-C contains 15 types of corruptions applied to the original ImageNet validation images, each having 5 sever- ity levels. We exploit the most severe level (5-th level) for experiments. The same applied to CIFARI00- ImageNet variants represent the natural distribution shifts caused by natural adversarial samples, image sources, artistic rendi-
tions, and black-and-white sketches. Beyond image classifi- cation, we also consider the semantic segmentation task and employ CarlaTTA dataset (Marsden et al., 2024a) for exper- iments. TarlaTTA includes 5 gradually changing scenarios: day2night, clear2fog, clear2rain, dynamic, and highway.
Baselines. We employ Tent (Wang et al., 2021) and TPT (Shu et al., 2022), two simple and general TTA methods, as baselines. Meanwhile, we compare with state-of-the-art (SOTA) camplle-elccttonnbaieee ETA (Niu et al., 2022a), De (Lee et al., 2024), and CPL (Zhang et al., 2024). Since our proposed PTTA is plug-and- play solution, we apply PTTA to existing TTA methods and achieve superior performance improvements. Additionally, we compare with previous SOTA TTA approaches such as SAR (Niu al., 2023), MEMO (Zhang et al., 2022), SoTTA
TTA algorithms, including
6

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Table 4: Experimental results (top-1 classification accuracy (%)) for test time prompt tuning on the episodic TTA task.
Table 6: Error Rate (%) of adversarial attacked samples under the indiscriminate and instant attack scenario.
IMAGENET -22
METHODS
TENT
ETA
DEYO
CPL
AVG.
METHODS
-KK
A
R.
8.
AvG.
CIFARIO0-C
CLIP-RN5O
BN ADAPT
13.3
52.4.0.34
24.2
54,9.0.2
53.7a..0
ZERO-SHOT
2..2-0.0 21. 00 51 $1.4-0.00 56.2-0.0 00 33.4-0.0 o0) 44. 200.00
PTTA
12.0io
51.5-0.3
32.310
14.610.1
52.611.0
TPT
21.9e0.0 9.9. 36 52. 1=0.29 57. 0.0.34 33.3-0.05
14.6
MEDBN
46.610.4
45.9.n.14
48.7.001
47.2s0.m
47.1a..1
PTTA (ID)
59.1.0a1
16 12.4.0.9 58.0.0.10
33.5
PTTA (OOD)
59.
22.5 52.3an.77 57.9.0.12
33.4
45.0s0.14
PTTA
45.7
45.3.0.0
16.610.10
47.1
46.2.0.:
TPT (COOP)
62.8 22. 22.620.24
55, 160.16 56.0zn.02 02.00. 34,2a0.00
ImageNNet-C
PTTA (ID)
02.6-0.06
23. 13.9-0.1 1a 45.4400.4 57. 37.550.02 34.5-0
46.8-o.0
BN ADAPT
69.420.11
23.020
62.020
1at.0
65. 32..21111
PTTA (OOD)
62.7.00 23.5 55.4.0.2 17.1.1.01 34.3.0.0
46.6.0.46
PTTA
65.4
12.00
64,2 50
63.5e1.0
CLIP-VET-R//D
ZERO-SHOT SHOT
66.7.0. 47.910.00 60.9 74.0.0.0 16.1.0.00 59.1
TPT
26.2.0.s 48. la0.2 59.9an.11 74.3 45.8a0.11 58. 58.920.0
Tent
4 PTTA (ID)
PTTA (OOD)
66. 1=0.0 48. 750.3 00.0e0.88 75, 120.08 45.7.0.12
66.620.8 49.1.0.0 60. 00.00 75.3
45.920.10
59. 40..0
59.
100.07
Motion
Glass [25.7-29 61
26..9-30.6
Defoc
PTTA wlo Samp, Sel.
PTTA w/ Samp. Sel.
TPT (COOP)
71.3-000 49, 6...663 63. 33.92..12 75, 0.0.11 47.5-.007 61. 61.5eo.0
[40.0-44.0] 0]]
Imppll
PTTA (ID)
71.1-002 49.7-0.0
64.1-0.8 73.5-0.0
47.2-0
61.1 61.1.0.0
(29.9-33.7]
PTTA (OOD)
71.2.0.8 50.020.30 63, 92n.19 75.6.0.0 47.620.00 61.6x0.05
[48, 1-5133
Shot [30.3-32.6]
Table 5:
Experimental results for semantic segmentation on
Snow [45.9-49.0]
episodic and continual TTA tasks. We report the mloU.
Gauss.
128.5-32.4]
DAV2NIGHT
CLEAR2FOG
CLeAi
[39.7-43.6]
Frost
DYNAMIC
HIGHWAY
JPEG
[51.2-53.7]
Fog [56.2-59.0]
METHOSS
Pixel [57.2-59.9]
NOADAPT
08.4 00
00
46 60
28.7 28.740..0 33.618.0
Brit.
COTTA
61.31007
56.74007
70.810.11
46.410.o
[66.2 41 Comtt
Elastic (53.66666)666))
MEMO-EPL.
61.1
71.74001 65.1
00.3
35.344.17
[25.2-42.3]
+ PTTA
61,4,n0
S5.3.002
1400
50.910.:
35.310.11
Figure 5:
Comparison of selecting benign samples and using all samples as candidates for purifying malicious samples.
TENT-CONTL.
61.5a001
56.020o2
70. 920.2
40.4...01
32. 02.0.
PTTA
62.32003
26
02.5
02.6
33.7.0.01
(Gong et al., 2023), CoTTA (Wang et al., 2022), and EATA (Niu et al., 2022a). Since CPL is designed for unsupervised domain adaptation, we modify it for TTA tasks.
2016), ViT (Dosovitskiy et al., 2021), CLIP Models (Rad- ford et al., 2021), and DeepLab- V2 (Chen et al., 2017). We run experiments on random seeds and report the average accuracy and the standard deviation.
Implementations. Considering that the pipeline of PTTA encompasses sample selection and malicious sample purifi- cation. We mainly focus on the latter purification strategy. Therefore, we employ existing sample selection criteria, in- cluding those proposed in ETA, YO,,, and CPL, refer to Sec. 2. Meanwhile, we also demonstrate that even without using sample selection, the saliency indicator can accurately be used for retrieval, refer to Sec. 4.3. We default to using the logit-saliency indicator due to its better performance and lower computational overhead. We build a memory bank for OOD retrieval, setting it as a first-nn-irst-ouu queue and limiting its maximum length to 1, 000. Although In in-Batch retrieval does not require memory bank, OOD retrieval is used by default as it is more suitable for diverse TTA tasks and existing TTA methods. Uniformly, we set K = for Mixup and conduct an ablation study of K in Sec. 4.4. We set a = 3.0 for sample-selection sete--eoo-eeeeeeeeeeeeeeeeeeeeed TTA methods and a = 1.0 for selection TTA methods. We employ various DNNs for experiments, including ResNet (He et al.,
4.2. Main Results
We compare the performance Tent, CoTTA, SOTTA, SAR, ETA, EATA, De,O, and CPL on the lifelong TTA task. Ad- ditionally, we incorporate our proposed PTTA into these methods, as shown in Table 2. We find that PTTA can alle- viate the catastrophic forgetting of semple-selecctio-bbseeee TTA algorithms in lifelong tasks. For example, the accuracy drop from the 1-st to 10-th round is improved from 13..1% to 1.22% for ETA and ETA+PTTA. Although using the prior information of in-distribution data (similar EATA) can also mitigate forgetting. The application of PTTA fur- ther enhances the utilization ratio of test data, improves the calibration of DNNs, and reduces over-confident predictions. Therefore, EATA+PTTA maintains the model performance without degradation, i.e., a 0.3% accuracy increase from the 1-st to 10-th round. The same conclusion is verified again in CPL+PTTA with another 3.5% accuracy increase.
7

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
90.100%
90-100%
10
90.
100%
80-90%
8:55
6:91
6:55
8.82
8:16
8:68
11:01
10:00
0.14
80-50%
597
8:3
0/5
8.33
0.77
8.20
8:43
0154
0144
0.25
80-90%
0911
0.91
0:44
024
0.51
0.7%
0.44
1:24
70-90%
1.
0.99
6990
6.09
0.65
8:31
8:35
08.55
11:00
5.47
70-30%
0.55
656
8:68
0.51
8.75
0:89
0/43
1154
0.03
70-80%
085 1.55
0.0
631
0.1
0/7
93
6:14
0.71
60-70%
60-70%
60-70%
50-60%
0.00
0.07
0.90
0.67
0.51
50-40%
3.55
6:00
8:55
0.75
0.70
50-60%
0.54
031
830
0.78
40-50%
3.48
6194
0.902
8:56
8:39
03:55
01:52
40.50%
9:97
6.95
0:90
0..9
8:57
8:34
0.51
0771
9.71
40-50%
8:00
8.18
0.99
0,00
0:74
651
0:51
0.55
30-40%
30%40%
30-40%
20-30%
100
0.07
0:9
6:55
0.92
20-30%
8:98
6:00
20-30%
0.77
0.11
0.80
0.81
8.81
10-30%
1.0
8:49
0.09
6:08
0.91
6:57
8:57
0.96
0.95
8:55
10-20%
0:80
9.99
6:08
6:08
0.97
89
1.6
0.0
000
0.22
10-20%
0:35
0.54
631
8:1
0.45
0.84
0.54
630
6:00
0.95
0-10%
0-10%
0-10%
0-10%%
10-231%
20-30%
30-40%
4n-90%
50-40%
60- 11-
70-807%
50-99%
90- 100%
0-10%%
10-21%
20-10%
30-40%%
40-80%
5i-4076
60-70%
70-807s
80--4%
ges 1007
60
0-10%
10-20%6
20-300%
30-4175
40 90-50%
58-607
660-70%
78-800%
H0-99%
90- 119%
80
10
40
10
(a) sogiiny indicator
(b) Seature-salienge indicator
(c) Pixel-saliency indicator
Figure 6: The Saliency Distance (normalized to the range of 0 ~ 1) among test samples sorted in the ascending order of
prediction entropy and split by percentages. A good indicator satisfies Deo (xt (xt,x,t)) > Da (xt ,x>) > x
Accuracy (%)
45
49.5
50
Accuracy (%)
60
65
65
44
Jaw (Oleiia
Jax. (baget
40
Accuracy
Accuracy
49.0
REAcSinde
ETA Castaall)
Accuracy
Accuracy
DvTO (light))
1PTA Degti
64
Accuracy
44
(PLig
Seurraa 
58
48.5
-ITTA (ooabral)
63
49
PITA (Coctall
63
43
20
43
1k
4k
100k
1k
4k
100k
1k
4k
100k
1k
4k
100k
Max. Length of MB
Max. Length of MB
Max. Length of MB
Max. Length of MB
Figure 7: Ablation study of different maximum lengths of memory bank (MB).
We conduct experiments on single and continual TTA tasks using ResNet and ViT to verify the generality of PTTA across different DNN architectures in Table 3. The single task emphasizes optimizing the model within a static target distribution with limited samples. Therefore, PTTA utilizes malicious samples that should otherwise be filtered out, im- proving the model's optimization efficiency and avoiding sub-optimal state. Applying PTTA leads to an average accu- racy increase of 1.5% for ResNet50 and 1.2% for ViT-B/16. Meanwhile, PTTA addresses the performance degradation of ResNet50 in dynamically changing environments, caused by the weak robustness of ResNet50 to distribution shifts, and achieves improvements of 37.0% and 26.5% for Tent and YO respectively. Experiments validate the above conclusions, referred to the Table 15.
4.3. Extended Analysis
Effect of Saliency Indicator. We compare the logit-, feature-, and pixel-saliency indicators mentioned in Sec. 3.2 by visualizing the saliency distance Dsa among test sam- ples, as shown in Fig. 6. We hope that all test samples can be matched with benign samples for Mixup. Therefore, good saliency indicator should satisfy Dea (at, ,x+) > Daa (x ,x > (xa ,-), where x+ and x:" represent benign and malicious samples respectively. A typical exam- ple is shown in Fig. 6 (a). In conclusion, the sogit-saliency indicator can better encode benign and malicious samples than the other two, incurs lower computational overhead, and is more suitable for purifying malicious samples. Addi- tionally, the results of COTTA+PTTA Tables 2 and show that the teacher- student consistency loss function can also be efffectivly applied to the saliency indicator, demonstrating the flexibility in choosing objective functions.
CIFAR100-CC also
We also conduct single and episodic TTA experiments on datasets representing natural distribution shifts, namely Im- ageNet and its variants, as shown in Table and 4. PTTA achieves an additional accuracy improvement of up to 2.0% over existing TTA methods. Meanwhile, on average, PTTA increases the zero-shot accuracy of CLIP models by 1.8% for CLIP-RN50 and 1.4% for CLIP- VIT/B-V. Beyond image classification, we also validate PTTA on semantic segmentation that simulates practical autonomous driving requirements. The experimental results, as shown in Table 5, indicate that PTTA achieves an mloU improvement of up to 1.8% in scenarios of day-to-night and foar-toorainy//ooggy weather changes, well urban-to- toihhhwway transitions. The efficiency of PTTA is analyzed in Appendix C.4.
Necessity of Benign Sample Retrieval Considering the usage of Mixup technique in PTTA, we compare with the vanilla Mixup method (Zhang et al., 2018), which stores historical samples in a first-in-first-out memory queue (max- imum length of 1, 000) and randomly selects samples from it to mixup with mini-batch samples. Results in Table 18 show that vanilla Mixup provides significantly weaker per- formance improvements to base TTA algorithms than PTTA. This demonstrates the necessity of benign sample retrieval.
Unnecessary of Sample Selection Criteria. Sample se- lection is employed to screen benign samples for purifying
8

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Table 7: Ablation study on retrieving top- K benign samples for malicious sample purification.
Different Size of Memory Bank. The size of Memory Bank (MB) determines the scope of henign sample retrieval and the required additional storage overhead. In Fig. 7, we compare the impacts of different MB sizes, where max- imum length of 0 indicates that we use In-Batch retrieval. The results show that a small MB, such as 1, 000 by default, is appropriate, while an extremely large MB is not suitable for TTA tasks in dynamically changing environments.
SINGLE
METHODS
K =
K- 2
K =
AVG.
TENT
42. 42.8.0.04
42.8+0.04
42. 04
02.8
+ PTTA
44.920.05
44.7=0.04
44.5a0.03
44.7.0.22
ETA
48.2.0.4
48. 48.2.00.4
48. 48.2.0.04
18.240004
+ PTTA
49.020.05
49.120.09
49. 120.06
40.1
DEYO
18.6
48.6.
48. 6
Comparison of Retrieving Top- Benign Samples. The K determines the number of benign samples used for Mixup and the linear interpolation coefficient À = 1/(K + 1). As depicted in Table 7, PTTA is insensitive to the value of K. However, as the value of K approaches infinity, it will dilute the proportion of malicious data information in purified samples. To maximize the utilization of test data, we set 62.2.0.07
48.6 6
+ PTTA
49.3
49.3
49.4.0.00
49.3.0.03
CPL
42.8.0.04
42.8.0.04
42. 24
42.8.0.0
+PTTA
44.6x0.06
44.4.0.05
44. 1 10...
44.4an.25
CONTINUAL
TENT
14.2x10.5
14.2s10.s
14.2st0.5
14.2=10.5
+ PTTA
60.1s
59.7a0.0
59.420.05
59.7.0.32
ETA
62.2.0.07
62.2
62.2.0.07
K = 1 as the default.
PTTA
64.7a0..4
64.3.0.07
63.9a0.07
64.2.n.43
DEYO
62.2.a.35
35.2
22.2.0.3
62.2...35
+ PTTA
65.1
64.920.01
64.420.32
64.920.21
5. Conclusion
CPL
55.5
55.5
55.5
15.5
In this paper, we show that malicious samples severely de- grade the performance of TTA methods and highlight the inherent difficulty of perfectly detecting them from test data. Instead of selecting and filtering out malicious samples, we propose plug-and-plae solution, PTTA. It retrieves benign samples, exerting the most opposite influence on the objec- tive function based on saliency indicator, to purify mali- cious samples. PTTA improves the utilization rate of test data, delivers more stable and robust test-time predictions of DNNs, and reduces existing TTA algorithms' sensitivity to sample selection thresholds. We further discuss the ne- cessity of benign sample retrieval and the unnecessary of sample selection criteria. PTTA achieves superior perfor- mance in four types of TTA tasks, as well as in classification, segmentation, and adversarial defense tasks.
+ PTTA
59.6.0.01
59. 10..00
58.440.04
59.0sn.58
Accuracy (%)
50.0
-1900-4
.a- ETAiSkalo 7TSAASogg
Accuracy
Accuracy
-9.5
DVYO (Siglo (TDg
Accuracy
-0- (TArCasina))
DLYO (Cunizaa) PrTAACCriiaa
48.5
PTSA (Gntzad)
Value of a
Value of a
Figure 8: Ablation Study of a in Eq. 8.
malicious samples. However, we find that an excellent saliency indicator can retrieve correctly matched benign samples from candidates mixed with both benign and ma- licious samples. Thus, PTTA can be applied without the assistance of sample selection, and meanwhile, avoid the impact of selection thresholds. The experimental results in Fig. 5 indicate that Tent+PTTA exhibits comparable perfor- mance with or without sample selection. It further validates the effectiveness of the logit-saliency indicator.
A Acknowledgements
This work was supported by the 111 Project Compu- tational Intelligence and Intelligent Control under Grant B B180 1024, the Foundation for Outstanding Research Groups of Hubei Province under Grant A0252 the project of Peng Cheng Lab under Grant PCL2023 and the Natural Science Fund of Hubei Province under Grant 2022CFB823.
Adversarial Defense. We also verify the performance of PTTA under the indiscriminate and instant attack scenarios, as shown in Table 6. The classification error rate of attacked samples is reduced by applying PTTA on BN Adapt (Schnei- der et al., 2020) and MedBN (Park et al., 2024). It indicates that PTTA can be effectively used for adversarial defense.
Impact Statement
4.4. Ablation Studies
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
Ablation Study of a in Eq. 8. is introduced to balance the original TTA objective function Lua and the purification loss we proposed Cpar- As shown in Fig. 8, we compare the impacts of a on applying PTTA to existing TTA methods. Since sample-seless www---eeeeecneeeeeeeeeeeee based TTA methods assign greater weights to lower-entropy samples, a larger is appropriate.
9

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
References
Goldblum, M., Fowl, L.., Feizi, S., and Goldstein, T. Adver-
sarially robust distillation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pp. 3996- 4003, 2020.
Chen, G., Niu, S., Chen, D., Zhang, S., Li, C., Li, Y., and Tan, M. Cross-device collaborative test-time adaptation. In The eightt Annual Conference on Neural Infor- mation Processing Systems, 2024.
Gong, T., Kim, Y., Lee, T., Chottananurak, S., and Lee, S.-J. Sotta: Robust test-time adaptation on noisy data streams. Advances in Neural Information Processing Systems, 36: 14070-14093, 2023.
Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K., and Yuille, A. L. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. IEEE transactions on pattern analysis and machine intelligence, 40(4))834-848, 2017.
Goodfellow, I. J., Shlens, J., and Szegedy, C. Explain- ing and harnessing adversarial examples. arXiv preprint arXiv: 141..5522 2014.
Cong, T., He, X., Shen, Y.. and Zhang, Y. Test-time poi- soning attacks against test-time adaptation models. In 2024 IEEE Symposium on Security and Privacy (SP), pp- 1304-1324. IEFE, 2024.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learn- ing for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Croce, F.. Andriushchenko, M., Sehwag. V., Debenedetti, E., Flammarion, N., Chiang, M., Mittal, P., and Hein, M. Robustbench: a standardized adversarial robustness benchmark. In Thirty-fifth Conference on Neural Informa- tion Processing Systems Datasets and Benchmarks Track, 2021.
Hendryeks, D. and Dietterich, T. Benchmarking neural network robustness to common corruptions and perturba- tions. In International Conference on Learning Represen- tations, 2018.
Hendrycks, D.,, Basart, S., Mu, N., Kadavath, S., Wang, F., Dorundo, E., Desai, R., Zhu, T., Parajuli, S., Guo, M., et al. The many faces of robustness: critical analysis of out-of-distriburs generalization. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 8340-8349, 2021a.
Deng, J., Dong, W., Socher, R., Li, L.-J,, Li, K., and Fei-Fei, L. Imagenet: A lcage-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248 255. Icce, 2009.
Deng, Q. Niu, S., Zhang, R., Chen, Y., Zeng, R., Chen, J., and Hu, X. Learning to generate gradients for test-time adaptation via test-time training layers. In Proceedings of the AAAI Conference Artificial Intelligence, vol- ume 39, pp. 16235, 2025.
Hendrycks, D., Zhao, K., Basart, S., Steinhardt, J., and Song, D. Natural adversarial examples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1 15262-15271, 2021b.
Krizhevsky, A., Hinton, G., et al. Learning multiple layers of features from tiny images. 2009.
Deng. Z., Chen, Z., Niu. S., Li, T., Zhuang, B., and Tan, M. Efficient test-time adaptation for super-resolution with second-order degradation and reconstruction. Advances in Neural Information Processing Systems, 36:74671- 74701, 2023.
Lee, J., Jung, D., Lee, S., Park, J., Shin, J., Hwang, U., and Yoon, S. Entropy is not enough for test-time adap- tation: From the perspective of disentangled factors. In The Twelfth International Conference on Learning Repre- sentations, 2024.
Dosovitskiy, A., Ros, G., Codevilla, F., Lopez, A., and Koltun, V. Carla: An open urban driving simulator. In Conference on robot learning, pp. 1-16. PMLR, 2017.
Lin, W., Mirza, M. J., Kozinski, M., Possegger, H., Kuehne, H., and Bischof, H. Video test-time adaptation for action recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22952- 22961, 2023.
Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly. S., Uszkoreit, J., and Houlsby, N. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Ma, J. Improved self-training for test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 23701-23710, 2024.
Gao, J., Zhang, J., Liu, X., Darrell, T., Shelhamer, E., and Wang, D. Back to the source: Diffusion-driven adaptation to test-time corruption. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11786-11799 1176, 2023.
Madry, A., Makelov, A., Schmidt, L., Tsipras, D... and
Vladu, A. Towards deep learning models resistant to adversarial attacks. In International Conference on Learn ing Representations, 2018
10

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Marsden, R. A., Döbler, M., and Yang. B. Introducing intermediate domains for effective self-training during test-time. In 2024 International Joint Conference Neural Networks (IJCNN), pp. 1-10. IEEE, 2024a.
Shu, M., Nie, W. Huang, D.-A., Yu, Z., Goldstein, T. Anandkumar, A., and Xiao, C. Test-time prompt tuning for zero-shot generalization in vision-language models. Advances in Neural Information Processing Systems, 35: 14274-14289, 2022.
Marsden, R. A., Döbler, M., and Yang, B. Universal test- time adaptation through weight ensembling, diversity weighting, and prior correction. In Proceedings of the IEEE/CVF Winter Conference on Applications of Com- puter Vision, pp. 2555 2555-256,,22 2024b.
Simonyan, K., Vedaldi, A., and Zisserman, A. Deep inside convolutional networks: visualising image classification models and saliency maps. In Proceedings of the Interna- tional Conference on Learning Representations (ICLR). ICLR, 2014.
Niu, S., Wu, J., Zhang, Y. Chen, Y., Zheng, S., Zhao, P., and Tan, M. Efficient test-time model adaptation with- out forgetting. In International conference on machine learning, pp. 16888-166 16905, PMLR, 2022a.
Su, Z., Guo, J., Yao, K., Yang, X., Wang. Q., and Huang, K. Unraveling batch normalization for realistic test-time adaptation. In Proceedings of the AAAI Conference Ar- tificial Intelligence, volume 38, pp. 15136-15144, 2024.
Niu, S., Wu, J., Zhang, Y., Xu, G., Li, H., Zhao, P., Huang, J., Wang, Y., and Tan, M. Boost test-time performance with closed-loop inference. arXiv preprint arXiv:2203.10853 2022b.
Szegedy, C. Intriguing properties of neural networks. arXiv preprint arXiv: 131.6199,,, 2013.
Tan, M., Chen, G., Wu, J., Zhang, Y., Chen, Y., Zhao, P., and Niu, S. Uncertainty caliibaed test-time model adaptation without forgetting. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
Niu, S., Wu, J., Zhang, Y., Wen, Z., Chen, Y., Zhao, P., and Tan, M. Towards stable test-time adaptation in dynamic wild world. In The Eleventh International Conference on Learning Representations, 2023.
Tang, L. and Zhang, L. Robust overfitting does matter: Test- time adversarial purification with fgsm. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24347-24356, 2024.
Niu, S., Miao, C., Chen, G., Wu, P., and Zhao, P. Test-time model adaptation with only forward passes. In Forty-first International Conference on Machine Learning, 2024.
Park, H., Hwang, J., Mun, S., Park, S., and Ok, J. Medbn: Robust test-time adaptation against malicious test sam- ples. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 5997- 6007,2024.
Thulasidasan, S., Chennupati, G., Bilmes, A., Bhat- tacharya, T., and Michalak, S. On mixup training: Im- proved calibration and predictive uncertainty for deep neural networks. Advances in neural information process- ing systems, 32, 2019.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Wang. D., Shelhamer, E.,, Liu, S., Olshausen, B., and Darrell, T. Tent: Fully test-time adaptation by entropy minimiza- tion. In International Conference on Learning Represen- tations, 2021.
Wang. G., Ding, C., Tan, W., and Tan, M. Decoupled prototype learning for reliable test-time adaptation. IEEE Transactions on Multimedia, 2025.
Recht, B., Roelofs, R., Schmidt, I.., and Shankar, V. Do imagenet classifiers generalize to imagenet? In Interna- tional conference on machine learning, pp. 5389-5400. PMLR. 2019.
Wang, H., Ge, S., Lipton,
7., and Xing, E. Learning robust global representations by penalizing local predic- tive power. Advances in Neural Information Processing Systems, 32, 2019.
Schneider, S., Rusak, E., Eck, L., Bringmann, O., Bren- del. W., and Bethge, M. Improving robustness against common corruptions by covariate shift adaptation. Ad- vances in neural information processing systems, 33: 11539-11551, 2020.
Wang, Q., Fink, O., Van Gool, L., and Dai, D. Continual test- time domain adaptation. In Proceediiggs of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 7201-7211, 2022.
Shannon, C. E. A mathematical theory of communication. The Bell system technical journal, 27(3):379 423, 1948.
Wen, Z., Niu, S., Li, G., Wu, Q., Tan, M., and Wu, Q. Test-time model adaptation for visual question answering with debiased self- supervveerssssss.. IEEE Transactions on Multimedia, 26:2137-214 2023.
Shi, C., Holtz, C., and Mishne, G. Online adversarial purifi- cation based on self supervied learning. In International Conference on Learning Representations, 2021.

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Wu, T., Jia, F., Qi, X., Wang, J. T., Sehwag. V., Mahloujifar, S., and Mittal, P. Uncovering adversarial risks of test-time adaptation. Proceedings of Machine Learning Research, 202:37456-37499 2023.
Yang, Z., Xu, Z., Zhang, Hartley, R., and Tu, P. Ad- versarial purification with the manifold hypothesis. In Proceedings of the AAAI Conference Artificial Intelli- gence, volume 38, pp. 16379-16387, 2024.
Yuan, Y., Xu, B., Hou, L., Sun, F., Shen, H., and Cheng, X. Tea: Test-time energy adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 23901-23911 23911, 2024.
Zeng. R., Deng, Q.. Xu, H., Niu, S., and Chen, J. Ex- ploring motion cues for video test-time adaptation. In Proceedings of the 31 ACM International Conference on Multimedia, pp. 1840-1850, 2023.
Zhang, H., Cisse, M., Dauphin, N., and Lopez-Paz, D. mixup: Beyond empirical risk minimization. In Interna- tional Conference on Learning Representations, 2018.
Zhang, J., Wei, Q.. Liu, F., and Feng, L. Candidate pseu- dolabel learning: Enhancing vision-language models by prompt tuning with unlabeled data. International Con- ference Machine Learning, pp. 2024.
PMLR,
Zhang, M., Levine, S., and Finn, C. Memo: Test time robustness via adaptation and augmentation. Advances in neural information processing systems, 35:38629-386422, 2022.
Zhou, K., Yang. J., Loy, C. C., and Liu, Z. Learning to
prompt for vision-language models. International Jour- nal of Computer Vision, 13009):2337-2344, 2022.
12

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
APPENDIX
Contents
A Related Work
14
A.1 Test-Time Adaptation
14
A.2 Adversarial Attack and Defense
14
More Design Details of PTTA
15
B.1 Proof of Logit-Salinncy Indicator
15
B.2 Pseudo Code
16
More Implementation Details
16
C.I Benchmarks
16
C.2 Experimental Protocols
17
C.3 Method Implementations
18
Efficiency of PTTA
19
More Experimental Results
20
D.I Lifelong Test-Time Adaptation
20
D.2 Single & Continual Test-Time Adaptation
20
D.3 Episodic Test-Time Prompt Tuning
21
D.4 Semantic Segmentation
22
D.5 Adversarial Defense
23
Additional Diseussions
23
E.1 Effect of Saliency Indicator
23
E.2 ID, OOD, and IB Retrieval
24
E.3
Necessity of Benign Sample Retrieval
25
E,4
Unnecessary of Sample Selection Criteria
25
E.5
Ablation Study Eq.
26
E.6
Different Size of Memory Bank
26
E.7
Comparison of Retrieving Top-K Benign Samples
27
E.8
Single-Sample Test-Time Adaptation
27
E,9
Purification via FGSM, SOAP, and DDA
27
E.10
Saliency Distance vs. Direct Cosine Distance
29
E. Process of Continual Test-Time Adaptation
29
F Limitations and Future Work
29
13

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
A. Related Work
A.1. Test-Time Adaptation
TTA is a task of online adapting Deep Neural Networks (DNNs) pretrained on the source domain to arbitrary target data distributions while achieving performance improvement at test time. Unlike classical Unsupervised Domain Adaptation, TTA does not assume the stationarity of the target distribution. Instead, it conducts online learning in dynamic environment that changes over time (Wang et al., 2022). The optimization objective of TTA is maximize the adaptation of current batch in the data stream while preventing the catastrophic forgetting of DNNs. Recent research on TTA focuses utilizing the self- suff-suueee paradigm to update part of parameters in DNNs under the condition that source data and its statistics are unavailable (Lee et al., 2024; Wang et al., 2022; Niu et al., 2022a; Wang et al., 2021). Tent (Wang et al., 2021) connects the distribution shift and prediction error to the entropy. More severe shifts and more errors all-in-all lead to higher entropy. On this basis, entropy minimization the sole loss function to update Batch Normalization layers DNNs. Following Tent, EATA (Niu et al.. 2022a) shows that high-entropy samples lead to noisy gradients that could disrupt the model. Therefore, EATA employs an active sample selection criterion to identify reliable and non-redundant samples for TTA. Additionally, De (Lee et al., 2024) points out that entropy alone is an unreliable confidence metric, and proposes applying an object-destructive transformation to filter predictions that are more grounded in the object's 's shape. CPI. (Zhang et al., 2024) is another approach developed for fine-tuning vision-language models. It leverages intra- and inter-instanee label selection based on a confidence score matrix to obtain appropriate candidate pseudo-labels for unlabeled data.
Unlike leveraging sample selection criteria to filter malicious samples, some TTA methods aim to devise general optimization strategies that utilize all test samples which achieving significant performance improvements, such as FOA (Niu et al.. 2024), ROID (Marsden et al., 2024b), MGTTA (Deng al., 2025), and EATA-C (Tan et al., 2025). Some studies investigate enhancing TTA performance by improving the model adaptation process, such as improving entropy loss (Yuan et al., 2024; Wang et al., 2025), refining batch normalization (Su et al., 2024; Park et al., 2024), correcting the pseudo-labels (Ma, 2024), and optimizing the inference process (Niu et al., 2022b). We demonstrate that the purification strategy effectively reduces malicious content in all test samples. Mcanwhile, we show that the PTTA serves as a versatile plug-and and-play method compatible with existing TTA approaches.
Other than improving TTA methods, series of studies focus on the security vulnerabilities of TTA algorithms (Wu et al., 2023; Cong et al., 2024). Distribution Invading Attack (Wu et al., 2023) is proposed based on the insight that the predictions of benign samples can be affected by malicious samples within the same batch. These malicious samples can easily be exploited by adversaries to create poisoned samples, which are injected into the test batch to make DNNs misclassify benign and unperturbed test data. Cong et al. (2024) show that as few as 10 poisoned samples can degrade the performance of TTA by 35%. Inspired by these findings, we propose an effective purification strategy for malicious samples to make full use of test data, avoid sample filtering, thus further improving the performance of existing TTA methods.
Furthermore, the TTA framework has been widely adopted across various Artificial Intelligence tasks, including image super-resolution (Deng et al., 2023), video classification (Lin et al., 2023; Zeng et al., 2023), visual question answering (Wen et al., 2023), etc.. Notably, the malicious sample hazard is inherently persistent across these tasks. Extending PTTA to these tasks in practical application scenarios holds potential for yielding additional benefits.
A.2. Adversarial Attack and Defense
Adversarial Attack aims to generate adversarial samples that are indistinguishable from natural data, by applying small yet intentionally worst-case perturbations, and manipulate the input to fool DNNs into producing incorrect output. The best practice is to utilize the gradient direction of image pixels with respect to maximizing the objective function, which measures the contribution of each pixel to the success of the attack, to generate adversarial noise for one-step or multi-step attacks (Szegedy, 2013; Wu et al., 2023; Cong et al., 2024). Adversarial Attacks have inspired the research community to focus on how train robust models so that there no adversarial samples or least an adversary cannot easily find them (Madry et al., 2018). These defense methods include adversarial training (Goodfellow et al., 2014), adversarial distillation (Goldblum et al., 2020), and adversarial purification (Shi et al., 2021; Tang & Zhang, 2024; Goodfellow et al., 2014). SOAP (Shi et al., 2021) exploits an auxiliary self-supervised task to perform gradient descent for image pixels to purify adversarial samples at test time. TPAP (Tang& & Zhang, 2024) uses the Fast Gradient Sign Method (FGSM) (Goodfellow et al., 2014) as a way to purify unknown adversarial perturbations to obtain correctly classified purified samples. Yang et al. (2024) introduce the manifold hypothesis to adversarial defense and propose an adversarial purification method based on manifold learning and variational inference. In this paper, we focus on solving the systematic issues that affect the performance
14

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
bottleneck of TTA. We are motivated by the idea of image purification and propose utilize benign samples purify malicious samples. Instead of noise, find that image Mixup (Zhang al., 2018) can be more effectively and efficiently applied as a part of the purification strategy for test-time model adaptation.
B. More Design Details of PTTA
B.1. Proof Logit-Saliency Indicator
Given a sample x, the predicted output model fo for a is defined as logits z = fa(x) = [z....., zc]. where C' is the number of classes. The predicted probability p (ff(()) Pi.....e is obtained by applying the Softmax function a()), L..,
exp(z+)
P1 C exp(zz)
(9)
Eg
The partial derivative of p. with respect to 2; and æj (i + j) is
0p:
exp(2i)
Ej exp(2z) exp(zi) exp(2;)
=p; -pi
0z:
ES exp(2;)))
(10)
0p:
- exp(as) exp(zz) pp; Ps.
0zj
(2j exp(z;))²
According to the formula of conditional entropy, Lem = Ei Pi log Pi, the partial derivative of LEm (f(()) with respect to z; is
@L(m(fo(x))
0 3 Pj logp;
Ozi
Iz:
Op, log Ps
0p; log
Ozi
Ozi
j4i
0p:
i llggr p
Opj
i log Pj
log pi 0zi
Pi
log P. Oz
P3
Oz,
Izi
jti
0p:
Op:
0p
Opj
log Pi Zz.
log Pi Zz.
+
Zz.
0z.
(11)
iti
jz
(1 + log p..)
Σ (1 + (og
J
C
(1+ log p:)p: +
Σ
(1 + log (-pp)))
p.(1 logp,) - Pi Σp, (1 + log
Since we have + log = 2, + (1 - log E5 exp(22)), where Q log ES exp(2;) is independent of index Then,
15

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
the Eq. 11 can be rewritten as
(Lrm(Jo(x))
C
C
Pizj +P Pi Σ Pjzj P:
02,
e Ps2s + + Σnj))
P:a. Pt Σ
3
(12)
c
Pia - Pi Σ Pj
c 21 Σ
Pi
Next, we use vectors to express the gradient of Entropy Minimization as
V Cia = -
(DCmm(fo(x))
(L
p-pp2. Op
(13)
Oz.
Ozc
B.2. Pseudo Code
Algorithm 1 Purification for Test-Time Adaptation (PTTA)
Input: Source model Jo with parameters 0. Test samples x f=1 ;= Diess arrived at time step 6. A memory bank M. Benign samples {2t Vū selected by basic TTA methods. Hyperparameters Learning rate n7.
Output: Predictions ýt = {(ss) 1: for test samples Xx.
for x, E Diest do
Compute predictions y. fe(x;) and logit-saliency indicator V Incorporate selected benign samples {att, vea V (= into M; Retrieve xi from M using saliency distance D.n (x),
11 (Eg. 4)
// (Eg. 3 and 5)
Generate purified image dt, and its pseudo-label yiy using Mixup;
// (Eq. 6)
Compute total loss Liotal Lita ccCpa((at)) )
// (Eg. 7)
Update f with 0 +- f V Lumati
end
C. More Implementation Details
C.1. Benchmarks
CIFAR-100-C & ImageNet-C (Hendrycks & Dietterich, 2018) respectively comprise corrupted variants of the validation images from CIFAR-100 (Krizhevsky et al., 2009) and ImageNet (Deng et al., 2009). These datasets include 15 types of corruptions: Gaussian noise (Gauss.), Shot noise (Shot), Impulse noise (Impul.), Defocus blur (Defoc.), Frosted Glass blur (Glass), Motion blur (Motion), Zoom blur (Zoom), Snow (Snow), Frost (Frost), Fog (Fog), Brightness (Brit.), Contrast (Contr.), Elastic (Elastic), Pixelation (Pixel), and JPEG (JPEG), Each corruption consists of levels of severity. Our experiments and evaluation concentrate on the most severe corruption level (5-th level).
ImageNet-IK(Deng al., 2009) is widely used and highly influential dataset for evaluating the performance of deep learning algorithms. It encompasses 1, 281, 167 images distributed across 1, 000 different object categories. In Test-Time Adaptation (TTA), the val set is often used to evaluate the performance of TTA algorithms on i.i.d. data and to measure the importance of parameters (e.g., EATA). We use the validation images to construct an in-distribution (ID) memory bank.
ImageNet-YY V2 (Recht et al., 2019) is an independent test set that consists natural images. These images are gathered from diverse sources. It encompasses 10, 000 images which are distributed across 000 ImageNet categories.
16

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
ImageNet-A (Hendrycks al., 2021b) serves challenging test set. It is composed of "natural adversarial examples" that misclassified by standard ResNet50. This set contains 7, 500 images belonging to 200 ImageNet categories.
ImageNet-R (Hendrycks et al., 2021a) is a dataset that collects images of ImageNet categories but in the form of artistic renditions. In total, it has 30, 000 images, and these images cover 200 ImageNet categories.
ImageNet-Sketch (Wang et al., 2019) is a dataset made up of black-and-whit sketches. It is collected independently from the original ImageNet validation set. Altogether, it includes 50, 000 images that span 1, 000 ImageNet categories.
TarlaTTAAMaaseese et al.. 2024a) is a synthetic autonomous driving dataset for exploring progressive test time adaptation in urban scene segmentation. It is created based on CARLA (Dosovitskiy et al., 2017), open source simulator for autonomous driving research. It contains 5 progressive test sequences evolved from the sunny-noon source domain, namely day2night, clear2fog, clear2rain, dynamic, and highway.
C.2. Experimental Protocols
Episodic Test-Time Adaptation. In the episodic task, TTA algorithms are required to optimize the model using the test samples within a single mini-batch. The updated model is then utilized to predict the current mini-batch Subsequently, the model's 's parameters are reset to the source in preparation for learning the next batch of samples. We mainly use CLIP models (Radford et al., 2021), I), including CLIP-RN50 and CLIP-ViT-B/16, for the episodic tasks. Instead of updating the parameters of CLIP models, we follow TPT (Shu et al., 2022) to update the learnable text prompt. We conduct episodic tasks constructed on ImageNet and its variants. We utilize two distinct methods for initializing the text prompts: 1) the default prompt, "a photo of {CLASS)", and 2) the prompt optimized through CoOP (Zhou et al., 2022). We select top-10% low-entropy test samples for TTA. We employ the Adam W with learning rate of 0.005 and batch size of 64. We run experiments on 3 random seeds and report the mean and standard deviation of the accuracy.
Single Test-Time Adaptation. We use 15 types of corruption in ImageNet-C and CIFAR100-C, as well as ImageNet and its variants to construct single tasks. A corruption or a variant represents a static distribution shift relative to ImageNet- IK. The model parameters are continuously updated within a static distribution without being reset. On ImageNet-C. we use ResNet50 and ViT-B/16, which are sourced from the torchvision library and the timm² library, respectively. For ResNet50, consistently use the SGD with learning rate of 0.00025, momentum of 0.9, batch size of 64, and no weight decay. For ViT-B/16, we consistently use the SGD with a learning rate of 0.001, a momentum of 0.9, a batch size of 64, and no weight decay. In experiments with a batch size of 1, the learning rates for ResNet50 and V/T-B/16 are adjusted to 0.00025 /64 and 0.001 0.001/64, except for Yo and SAR, the learning rates are set to 0.00025 /22 for ResNet50 and 0.001 /32 for ViT-B/16. On ImageNet-R, ImageNet-Ve V2, ImageNet-A, and SmageNeet-Skethh we conduct experiments using ViT-B/16. We employ the SGD with a learning rate of 0.005, a momentum of 0.9, a batch size of 64, and no weight decay. On CIFAR 100-C, we employ Wide ResNet-40 (WRN-40) with pre-trained weights from the RobustBench (Croce et al., 2021). All methods consistently use the SGD with learning rate of 0.0005, momentum of 0.9, batch size of 64, and weight decay. We run experiments on 3 random seeds and report the mean and standard deviation of the accuracy.
Continual Test-Time Adaptation. The continual task considers a dynamically changing environment, which is constructed with 15 types of corruptions concatenated in the order of Gaussian noise- - Shot noise -> Impulse noise -> Defocus blur + Frosted Glass blur + Motion blur + Zoom blur - Snow + Frost - Fog + Brightness - Contrast + Elastic + Pixelation > JPEG. The model is continuously optimized in such environment, and its parameters are never reset. We construct the continual tasks on Netge and CIFAR consecutive corruptions. On ImageNet-C, we use ResNet50 and ViT-B/16. For ResNet50, we consistently use the SGD with a learning rate of 0.00025, a momentum of 0.9, batch size of 64, and no weight decay. For ViT-B/16, we consistently use the SGD with a learning rate of 0.001, a momentum of 0.9, a batch size of 64, and no weight decay. In experiments with a batch size of 1, the learning rates for ResNee50 and ViT-B/16 are adjusted to 0.00025 /64 and 0.001 /64 except for DeYO and SAR, the learning rates are set to 0.00025/32 for ResNet50 and 0.001/33 /32 for ViT-B/16. On CIFAR100-C, we use Wide ResNet-40 (WRN-40) with pre-trained weights from the RobustBench (Croce et al., 2021). All methods consistently use the SGD with a learning rate of 0.0005, a momentum of 0.9, batch size of 64, and no weight decay. We run experiments on 3 random seeds and report the mean and variance of the accuracy.
and we report the Top-1 classification accuracy
15
Lifelong Test- Time Adaptation. The lifelong task focuses on running TTA algorithms indefinitely to optimize the model in
hhaps://gitub..com/yyontt
17

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
dynamically changing environment. We define the 15 consecutive corruptions in the continual task 1 round. Meanwhile, use 10 rounds compose the lifelong task, which includes 150 consecutive corruptions. We construct the lifelong tasks on ImageNet-C and report the Top-1 classification accuracy. We use ViT-B/16 with the SGD with learning rate of 0.001, momentum of 0.9, batch size of 64, and no weight decay. We run experiments on 3 random seeds and report the mean and variance of the accuracy.
Semantic Segmentation. Beyond image classification, we demonstrate the effectiveness of PTTA in the semantic segmenta- tion task. We employ CarlaTTA as the benchmark and use the DeepLab- V2 (Chen et al., 2017) with ResNet- 101tt-0111111 as the backbone. We use the SGD with learning rate of 2.5 10-1, momentum of 0.9, and a weight decay of 5 x 10-1, We use the model's one-hot output as the pseudo-labels to calculate the KL-divergence loss. We set Num to 10 and a to 1.0. We adopt the standard mean- neean-ntersectioon-veerr standard deviation over 3 random seeds.
(mloU) as the evaluation metric, and report the average results and
To more closely simulate the performance of different methods under real-world attack scenarios, we focus on the indiscriminate and instant attack scenario, where the attacker injects a set of attacked samples into the I-th mini-batch after adapting to the previous t unattacked batches. Based on the experimental setup of DIA (Wu et al., 2023). we set the batch size to 100, the number of attack steps to 50, the attack optimization rate to 1/255, the initial perturbation to 0.5, and the perturbation constraint to 1.0. For CIFARIOO-C, we use ResNet-226 with EMA, and the malicious sample ratio is set to 20%. For ImageNet-C, we use a ResNet50 with a malicious sample ratio of 10%. In all experiments, we employ the Adam with a learning rate of 0.0005 and no weight decay. We employ MedBN (Park et al., 2024) and BN Adapt (Schneider et al., 2020) as the baselines. We replace the original BN layers with MedBN or BN Adapt, and combine these two approaches with various TTA methods.
Adversarial Defense.
C.3. Method Implementations
PTTA (Ours). The PTTA can be applied to other TTA methods. The sample selection criteria of existing methods are used to distinguish benign and malicious samples, and the benign ones are added to the memory bank, For sample-selection-base Tent variant, we employ the criterion of ETA. In general, the maximum length of the memory bank is Nmm = 1, 000 for ID respectively.
and OOD retrieval, and the values of are set to 1/3/3/3/1 for Ten/ETA//AAA//DeYON
Tent (Wang al., 2021). Tent minimizes the entropy of model predictions during testing. We use the same hyperparameters reported in the paper (Wang et al., 2019) unless otherwise specified.
COTTA (Wang et al., 2022). The hyperparameters of our implementations are consistent with the paper (Wang et al., 2022). The same random augmentation composition is applied, including color jitter, random affine, gaussian blur, random horizonal flip, and gaussian noise. We use 32 augmented versions for experiments and a restoration probability of p = 0.01.
SAR (Niu et al., 2023). We follow all hyperparameters that are set in SAR unless it does not provide. The reset threshold Eo is set to 0.2. For ResNet50 that has 4 layer groups, we freeze the 4-th layer. For ViT-B/16 that has 11 blocks groups, we freeze the 9-th, 10-th, and 11-th blocks.
ETA & EATA" (Niu et al., 2022a). We configure the entropy threshold En to be 0.4 x In C, where C is the number of classes. The threshold of cosine similarities e is set to 0.4/0.05 for LUFAR-I00/mageNe For the anti-forge regularizer, its weight 3 is assigned a value of 1 1/2000. In total, 2, 000 samples are used to compute the anti-forgetting regularizer.
DeYO (Lee al., 2024). We set the entropy threshold Eg = 0.4 In C and the entropy factor Teat 0.5 In C, where C represents the number of classes. In terms of the Pseudo-Label Probability Difference (PLPD) threshold TRLrD, we set TPLPD = 0.2 for ViT-B/16. As for other models, it is set to TPLPD 0.3. For ResNet50 which has 4 layer groups, we freeze the 4-th layer. For ViT-B/16 which has 11 blocks groups, we freeze the 9-th, 10-th, and 11-th blocks.
CPL (Zhang et al., 2024). CPL aims to fine-tune vasionne models with unlabeled data. Through experiments, we find that using generated pseudo-labels and multi-label loss is not suitable for TTA tasks. Therefore, the laeudo-label
'https: http://hhhhutcccom//gaeeeer///ea
Thttps: httpt////gghhaaac.ccccc//agaa/aa
htttae///gghaabb.coo///uu///////
18

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
selection mechanism of CPL employed select benign samples, and then entropy minimization used optimize the model. The selection ratio is set to 0.9. The top-K,, which represents the maximum number of selected samples per class, is originally set to 16 in CPL. For datasets with data volume less than 16 C (where C' is the number of classes), to ensure an effective filtering, the o is set to 4, 3, and 10 for ImageNet- V2, ImageNet-A, and ImageNet-R, respectively. The specified quantile a is set to 0.7.
TPT (Shu et al., 2022). TPT addresses the zero-shot generalization issue of vision-language models by proposing the Test-Time Prompt Tuning method. Due time constraints, we focus on a larger batch size for faster running speed. Specifically, we adopt a batch size of 64, which differs from the original setting of Instead of augmenting single sample 63 times form batch of 64, we directly input 64 samples into the model, thus achieving higher efficiency.
C.4. Efficiency of PTTA
We evaluate the running time of different TTA methods and their PTTA-applied versions under the following configuration Hardware: CPU: Intel Xcon@ Silver 4210 @ 2.20GHz
GPU: NVIDIA GeForce RTX 3090
RAM: 256GB
Software: PyTorch 1.9.0 CUDA 11.1
All experiments use a single GPU without Automatic Mixed Precision (AMP), except: 1) COTTA & CoTTA+PTTA employ AMP by default, 2) CoTTA & COTTA+PTTA for ViT-B/16 leverage dual GPUs via Distributed Data Parallel (DDP).
As shown in Table 8, the PTTA-applied versions incur approximately 10% 40% additional running time compared the base TTA algorithms, primarily due to the extra forward passes required per batch. Additionally, we quantify the storage overhead of the memory bank for OOD-retrieval purposes, implemented via first-in-firs-ouu queue with maximum length of 1, 000, as detailed in Table 9.
For a fair comparison, we evaluate the performance of base TTA algorithms and their PTTA-applied versions under comparable computational constraints, as shown Table 10. Specifically, for base TTA methods, we maintain a memory bank and randomly select benign samples from it to replace malicious samples during adaptation. Consistent with PTTA's implementation, we utilize first-in-firttoou queue with maximum length of 1, 0000 to store benign samples, with additional forward passes required for randomly replaced benign samples.
Table 8: The running time (seconds) per batch for different TTA algorithms and their PTTA-applied versions. The batch size is set to 64. A denotes the runtime increase ratio of PTTA-applied versions compared to the base TTA methods.
METHODS
RESNET50
VrT-B/16
METHODS
RESNET5O
VIT-B/16
TENT
0.157
0.264
CPL
0.157
0.310
PTTA
0.219
0.376
PTTA
0.225
0.400
A
39.5%
42.4%
A
43.3%
29.0%
ETA
0.158
0.270
SAR
0.225
0.540
+ PTTA A
0.225
0.384
+ PTTA A
0.328
0.770
42.4%
42.2%
45.8%
42.6%
EATA
0.171
0.291
COTTA
0.530
1.440
PTTA
0.236
0.410
PTTA A
0.673
1.597
A
38.0%
40.9%
27.0%
10.9%
DEYO
0.177
0.328
SOTTA
0.794
1.379
PTTA
0.254
0.420
+ PITA
0.872
1.597
A
43.5%
28.0%
A
9.80%
15.8%
Table 9: The storage overhead of the memory bank for firstin--irst-out queue with maximum length of 1, 000.
LOGIT- SALIENCY
PREDICTED PROBABILITIES
Raw IMAGES
INDICATOR
28.0 KB
28.0 KB
35.0 MB
hhttps://gi www../////.../t///oee
19

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Table 10: Comparison of base TTA algorithms with their PTTA-applied versions under comparable computational constraints.
SINGLE
CONTINUAL
METHODS
ETA
DEYO
CPL
AVG.
ETA
DEYO
CPL
AVG.
BASE ALGORRITMM
48.2
48.6
42.8
46.5
62.2
62.2
55.5
60.0
+ RANDOVLY REPLACE
48.6
48.8
44.0
47.1
62.2
57.8
58.1
59.4
+ PTTA
49.5
49.9
44.6
48.0
65.3
65.8
59.6
63.5
Table 11: Detailed experimental results (top-1 classification accuracy (%) for the 1-st round of the lifelong TTA task. We highlight the highest accuracy in bold and the second best as underline.
METHOD
GAUSS,
SHOT
IMPUL,
DEFOC
GLASS
MOTTON
ZOOM
SNOW
FROST
FoG
BRIT.
|CONTR.
ELASTIC
PIXEL
JPEG
AVG.
NOADAPT 31, 3..6..m0 31,6,0 oo 31, 31,6x0.00 co 31, 60 31, 6.000 31, 6o 31.6 11.60000
(3.66-000
31.6 0: 31.6 31, 31.6-0m03 (3.66000 31, 31.6.000 6.0.00
6.
31 11.6.0.u
TENT
44t.oo 1ad.11 12.1 2.2.0.09
0.1
0.2s0.00
0.2a0.00
0.3s5.01
0,123000
0.130.00
0.4g0.01| 0,1
0.1a0.00
0. 1.0000
0.200.00
L.
+ +PTIA
44.6a8.1 51,6aa.12 45..00 51, 11.4 53.5a 58, Januz 57.0
58 8an 63. 69 39.0ass 76, 2am 64 64.8anaz 58, 9anizs 69.6xn.z
68.0aum
60.0ka.ss
COTTA
23.6+a in 33.0+0 19 45.5 1n. 35. 7-n os 41.8-0
45.6 38. e5
36, 2-0 47. 3---- io 46 34 64. o7
[41 .44 at 40. 1-0.0 $2. 32.4.0.0 so
[2, 2...
PTTA
46.521.22 58.2a6.41 58.649.0 43. 52, 9aiss 52.
44.74
45.
Lane 51.7a033 53.7a 61.0unz 48 48.0a1.0 51.3an00 57.4 56.2
14.14
SOTTA
44..5 511. 63 55, 25.7 51. 51,0=0 0s 57. 3e0.z 55.6 10 59. 5.0.03 62. 0z 66 95000 76, 2:0 62.99...1 1..1.. 60, 65 68, 43..6.006 67. 67.3mm
59.55...3
PTTA
44. 851.00 52.4e0.32
56.2et.62 53, 4s0.ol 55.3et 59. 29.440.28 58, 780.38
61. 61.390.49 64.5.0 23 70.1 76.550.0 10 65 25.560.26 62, 120.5 70.4
68.4 4.
6g.3
SAR
45, 1 $3.3es.00 56.4 46.400.08 51, 85 54, Osn.11 57, 9.0.07
55, Osn. 12 (5. 6=0.03 62. 4:0 67.1=0.25 76, S:0
(63 30.00
858.1 68.6s0.03 67. 8.0.n2
60, 60.0ss.03
+PTIA
46. 4a0.02
33.705.00 57.5a3.00 53, 7a0o1 55, 720.14 59.628.02 58.320.00 61. 04.920022
70.1a 72004 63.6an.8 61 62.42 70.6.0.01 68. 830.3
61.
6ah.01
ETA
33.3. $7.6xc.0s 58.3.0.15 53, Bunos 56, Is 19.44...s
62. 11 63. La 68. 8ann a5 662 8...52 64, 1... 70. 10.00 67. 97
62. 24..0
PTTA
54.0+a.0 59.746.00 60.5 57.0-n 60. 63.5+0 63.6-n on 65.7
66 66.4-000 72.4-0005 77.6-001 67.5.0.1
67.5+0.3 13.1.0.0 70.740
65.3+a
EATA
k2.0 55.4 as $7.0ka.1 51 36 59, 840.15
38.9
62..7-0
64.3-n
70.7-033 31
666 340.0
64.5-a
70.4 31 68. 90..9...11
622.5.
PTTA
52. 3.0.3 57.4 52 58, 860.31 56, 4:0.28 59, Oen:36 36 62, 6.0. 10 62. (2..5...a 46 65.7=0 30 67. 0 26 73. 0.00s 77. 7-0 30 68 5.0.2 66. 100.2
72.5
70.3 94
64.7700.1
DEYO
14.44 58.6at.0
59, 4ab.on 89 Lato2 61.440.04
624.81
1.3.4 64. 39 70. 23 1s 64, Sa0.06 67.3amm22 71.9.0.0
69.5....
62.0sa.50
+ PTTA
55.0ka.0 59.8ac.00 60.55a..0 57. 0a 60.4 64.0kn.2 12 63. 7at
66. 12 67. 07.0an 73.0amoz 78, Danus 67. 8.0.33 68. 7an.0 73.7 71.2
65.8
CPL
42.4 47.9ac.14
52.5a0.11 46, Dutce 47. $3,4at.1 50.7 7a (54 9a
57, 13 60. 6un.4 75. Lanr [56. 1a.32 $7, lanzi 65, Ja.11 65.3
55, 55..s
PTTA
42.8 49.6 54.640. 94.9 52. 84n 57.64am 56.2-k
59.4.001
64 69.4.mm 76. 2-000 64.6m03 59.24am0 68.8 67.6
59.748.0
D. More Experimental Results
We conduct experiments on four types of TTA tasks, including lifelong, single, continual, and episodic tasks, and semantic segmentation as well adversarial defense. In the following, we present the detailed experimental results of the baselines and our proposed PTTA for each task.
D.1. Lifelong Test-Time Adaptation
We conduct experiments in a dynamically changing environment with 150 corruptions over 10 consecutive rounds, contin- uously updating the parameters of DNNs without resetting. We report the detailed experimental results of the 1-st, 5-th, and 10-th rounds in Tables 11, 12, and 13. Meanwhile, the average accuracy of each round is reported in Table 2. ETA applies prediction entropy and sample diversity as criteria for screening benign samples. This kind of sample selection method achieves excellent performance in short-term environmental changes. However, with the continuous change of the environments, the accuracy of the model will experience catastrophic degradation, iee., 62.2% in the 1-st round and 25. 1 1% in the 10-th round, The integration of our PTTA can effectively alleviate the performance degradation problem of ETA, with 65.3% in the 1-st round and 1% in the 10-th round. EATA mitigates the model's catastrophic forgetting as well by introducing the Fisher information of in in-distribution data for supervision, achieving 62.5% accuracy in the 1-st round and 61. .00 in the 10-th round. However, applying PTTA to EATA enhances the utilization of test data, improves the calibration of DNNs, and reduces over-confident predictions. We find that using the prior information of data distribution along with purification strategy can address the issue of performance decline in lifelong tasks, i.e., 64.7% accuracy in the 1-st round and 65.0% in the 10-th round. This conclusion is verified again when PTTA is applied to CPL.
D.2. Single & Continual Test-Time Adaptation
We conduct experiments on single and continual TTA tasks with ResNet and VIT architectures to verify the generality of our proposed PTTA for different DNNs. The experimental results are shown in Table 14 and Fig. 9. Compared with continual TTA, the single task places more emphasis on the adaptability of TTA algorithms in static distribution shifts. Since
20

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Table 12: Detailed experimental results (top-1 classification accuracy (%) for the 5-th round of the lifelong TTA task. We highlight the highest accuracy in bold and the second best as underline..
METHOD
GAUSS.
Shor
IMPUL,
DEroc.
GLASS
MOTION
ZOOM
SNOW
FROST
FoG
BRIT.
[CONTR.
ELASTIC
PIXEL
JPEG
AVG.
NOADAPT
31.6a
31.6a0.u 31, Butor 31, 31.6a
31.6a
31.6unou/
31. 3unup
31.6a0 31, 31.6u00 31, 31.6umm [3, 31.6wam
31.6u000 31 31.6m0m
31 31.6
31. 31..00.0
TENT
0.1-000 0.1-nos
0.1-n00
0.1.0
0.100
0.1.0.00
0.1.0.00
0.14000
0.14000
0.1.0.00
0.1.0.00
0.140 0.1.0.0
0.1-00
0.1-0.
0.1-000
PTTA
Lanop 0.
0.1on 0.1ango
0.langp
0.1aump
0.1am.0
0.1anoo
0.1a0.0
0.1ane1
0.1anoo
0.1amm
0.1anoo
D.
COTTA
27. 6..6...0 os
19 37.0 87.0 26.2-1 o 3-000
32.5+0.0
29.
ms
28, 03 34.3-1
8-0.4
47. 9-1 os (30. 69
30.2
38.74..1
37.94
33.5
PTTA
38, Se 42. 3a9.07 29, 95044 38, Sa031 38, Os0.30 33, 920.22 32. 100 38, 55 40. 70030 48. 7:0 36. 18.44 39.0e0:3 45.3
44, 720.13
39,
290.0
SOTTA
52. La.5e 14.1 55. 65.55 54, 7:0 12 56, 8en 13: 39.000. 57.0 10 62. (2..2.0.0 62.3= oz 67. 9=0.07 o5 [62 6...13 63. (3.770. to 20.200.00 67. 6.
[6 Se0.13
PTTA
54.268.67 56. 57.263.50
58. 58.5a1.31
020.32
64.3.0.15 64, 8.0 69. 69.9an00 76 2:0.06 a0.15 65.428.16 72.0s0.2 68.8.0.1
63.5
SAR
23.2a0.0 54.9ac.0 36..30.00 35.3a 56. 9asuus 59, 8an.u Sa.8
62.44mm.
62. 68 2ar3s 75.8amms (63.0am.s
63.5ammw 70.4.0.0 68, Ou..cn
7..740.0
PTTA
55.0+a.07 or 56.8 on 58. 18.244..0 20 57. o- 64 61.7-0.0 61. 1- 64.5 5. 64. o 70. 00 76, 31 66 0.0.00 65, 3- 72. 12.1.0.0 69.2
63.8+a
ETA
27.9 2 29.9 29.5 25. 4 O.5a: 1 Ouzs
30.9....7
32.2-mn 33.3.38. 44. 1-38: 221.2
33.8
39.9 36.9.m x
31.5
PTTA
56. 748.0s 58.766.0 04. 59,3eb.14 14 57, 860 15 59.9en 12 63, 33,33000 24 4e0.00
65.4 65.4-0.08
65.3-0 15 71.2 76, 4:0.05 66. 9.0 16 67. 750.00 72.6,0 10 69. 7
64.
940.0.0
EATA
51. 6a9.65 5..6 55, Aat.13 52, 950.s 420.5 280.11 57. 77..4
611.6.0.4
63
8a 75. 8an (65. 333.6 64. 255.5 70.2a0.45 68. 61
61. 6.0.44
+PTIA
55. Jan.s 57.6a0.1 58.940.19 57. Danes 59.4atcs
8ates4
66..3...
67.0an 25 72. Sanoo 77. 68 68.6an.7 66.6 72.7a0.31 70.5
65.1
DEYO
17.4a299 17.84306 17.7a105 4225.114 15. 8417 18.14312
4,627.51 15. 6226.9
18.333 21.2256. 25.0a43. 199.3
19.9234.2
22.7as 22, 02.7.
17. 7.9
+ +TTA
24e.ss 59.7ac.0s 10..0 58, Lanas 60. B5 64.5 64, Latosn
(66.7.m.. 7an.1 66, 2. 72, 72.2.m.3 2m. 67. .7.4...3 69.5 73.5 7..7.
65. 65.0
CPL
50.3 32 1e..5
75.000 51, 29 52, Os 56, 840.34 53, 85
60.
1s 60. os 62, 0o 74, 15
25..222.15
1 68.2 is 8an 58, 4...7
PTTA
53.420.0 64,66 56, 6a3.08 45.4a002 56, 8atv12 60. 59.2at.05 63. 32.500 43.4 69. 8aros 76, 2aton 64. 64.810.11 63. 71.0 68, 43..00
62t.00
Table 13: Detailed experimental results (top-1 classification accuracy (%) for the 10-th round of the lifelong TTA task. We highlight the highest accuracy in bold and the second best underline.
METHOD
GAUSS.
SHOT
IMPUL.
DEFOC
GLASS
MOTION
ZOOM
SNOW
FROST
FoG
BRIT.
|CONTR.
ELASTIC
PiXEL
JPEG
AvG.
NOADAPT 31. 625.00
31.620.00
31,620.00 31, Gunoo
62t00
31, 62t.o
31.
[31, 6anos
31. Guroe
31. 6unoo
[31 6anao 31,6o 31 31.6,0.00
31, 11.660.01/
311
11.620.00
TENT
Lano 0.
Lano
0.1.0
0.1.0
0.1.0.0
0.1.0.0
0.0.0.00
0.1.0
0.1.000
0.1a0.0
0.14000
0.1.0
0.1
PTTA
1:000 1:000
0. 0.000
100.00
5..00
1.000
5.000000
1.0.00
1.3.00
1.0.0.00
0.0.0.
0.0.0000
0.000000
0.100000
1:00000
5.0.0000
COTTA
25.741.00 ne 31.4
33.8+1.11
23. 8-
29. 64
29, 8+1 m
13
26.
26.34o
st 31.2-0
31. 31.94661
43. 2-00s
(28 441
27. 04.71
35.
34.941
30.6+a
PTTA
36, 959.25 40. 40.440..1
28, 960.36
36, 9c000s 36, Ss0.os
32, 9e0 16
31 31.0so 39 36.7=0 39.0sn
46. 1:0.0 35 8.0.1 37, 37.0sn.s 43.5
43 0.0000
37.
660.00
SOTTA
53. 4a.us 54.8a0.47 55.8a0.2
67.60 La.0 17119 60.02n.34
62.
7an.1
22.22z 68. Laouz 75. 235.4 (62 San.s 64.42n24 70.8 67. 1
61. 920.
PTTA
55. 1...9 56.740.6 S7.740.7 57.3 88.84 62. 14. 61..5 664 8a000 64. 64.8.033 70.0amm 75. 9anus 66 66.4mn.2 66. 72.4 69.0wa.1
64.0
SAR
47. 9a..u
53,0ac.1 35.990.4 $2, bu 54,3at01 58.0an27
54.7atce
54,
59.9am.4 62. 4aros 67 buros 76. 76.4am12 33.4...31
61.0a 69. 9...2 68. 2
60.4...0
PTTA
55. 9s
57.4x6.01 01 58.544.00 57.3-n 59. 0xaus 62.540mm
740mm 64.
9. 8-0 69 9. 75. 9.00s 66. 66.340.01 66.04amm 72.4 os 69. 69.2-m.0
644
00
ETA
Sais. 23.2.20.1
23.7a204 20. 0=17.4 23, 0.19 24.3a21 22, O2t00 24, 6:21.3 24, 7-21. 24. 6221 38, 0s32. 14 0s12.0
6.24.7
33.6229.1 30. 5025
2.5. 15.16
PPTAA
55, 9aaui
57.628.12 58.3a3.16
12 59.5at013 62, 7a0.05 62, 820. 12 64, 420.07
64.3 18 70. 20.2 75. 620.07
63. 23.77m 67.0a0.14
71.9
68. 8an.1
640.1
EATA
10.9 52.5at.12
54.7200.4 52, 3a0.2 54, 92nc2
57. 858.38
120.4 660
8:0.97
62, 8. 69, 2an11 25.66...7/6 664 S20.24
63.5
69. S...85
68. 33
61. O20.35
PTTA
1a.37 57.7a6.1 58.992.4 56, Sano2 59, Satceo
62. 9an.3 62.
7at.39
65.93
66
43 72.8 77. 6un.s 68.4
66.6. 72.6x0.3
70.5
65,
O20.41
DEYO
0. L-nno
0- 0. I-ans 0. 140.00 0. 0.0.0
0.000000
04.00
0.1 1..00
140.00
0.1-000 0.140.0 0...0.00
PTTA
56. 6e9.37 59,0 00..0...5 59.3 454.7 56.5a3.92 57.745.07
3...6...4 34
61 833.73
62.2 38 67.0p6
10.000 34 15244
63
61, 119.0 69, 29..22.22
66.425.88
61,
61..0
CPL
31.6xs.0 51..44.07 0z $3.2+9.11 52, 2-n 30 52. 35
57. 240.37
54, 3:
61.66
1.66--11
60.
5
61.7-a3 74. o4
(56
39 60, 8 68.8s0.10
65, 8.0.10
840.11
PTTA
15.0s 55. 33.50 57.2s3. 56, lan 57.4 60. 9an.o5
60.0aa
64,
220.09
65 6an 04 70. 70.0ano5 76.0s0.s 65. 1 64, 1anio 71,4
68. 780.00
63.2
the dataset of a single target domain provides limited test samples, is crucial to make full use of all samples. This can improve the optimization efficiency of TTA algorithms and avoid getting trapped in sub-optimal state. Applying PTTA to Tent, ETA, EATA, DeYO, and CPL leads an average accuracy increase of 1.5% for ResNet50 and 1.2% for ViT-B/16. As ResNet50 is significantly less robust to distribution shifts than ViT-B/16, common TTA algorithms such as Tent and YO experience severe performance degradation in continual tasks when using ResNet50. PTTA can address this issue by improving the calibration of DNNs and reducing over-confident predictions, enabling Tent and De to achieve accuracy increases of 37.0% and 26.5% respectively. We further validate the above conclusions using WRN-40 on CIFARI00-C. as shown in Table 15.
D.3. Episodic Test-Time Prompt Tuning
We conduct experiments on the episodic test-time prompt tuning task using CLIP models. Following TPT, we only update the learnable text prompt of text encoders for single- samples. The experimental results are referred to Table 4. TPT uses the default "a photo of a (CLASS)" or the text prompt learned from CoOp (Zhou et al., 2022) as the initial
21

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Table 14: Detailed experimental results (top-1 classification accuracy (%) on single TTA task. We highlight the highest accuracy in bold and the second best as underline.
NOISE
BLUR
WEATHER
DIGITAL
METHOD
GAUSS.
Shor
IMPUL,
DEroc
GLASS
MOTION
ZOOM
SNOW
FROST
FoG
BRIT.
|CONTR.
ELASTIC
PIXEL
JPEG
AVG.
RESNET5O
]
3. 3.Danno 3.7am0e 2.6an00 97.9 9.7am00
14.7am.0 22.5 16. 60. 13 24.0ano 59. 59.1amm] 5:450.00 16.5an00 20.9 32. 22.6.0.0 | 18.2as.00
TENT
29.7 7..... 31.5
31.1
28. 1- 26, 94
31.2 49. 17 47. Lo 40.9 9. 57. 10 67. 47.44mmm 26.4.0.1
54.84nm
58.4.0.1
52.4.0
42.88
PTTA
Ges.0. ou 311. Be5,31
32,9 19 29, 820.22 28, 8en 17 43, 2e0.26 26 5:0.07
48. 2:0 42. 8:0 17 58, 2s0os 67. 6:0.09 41 44.240
55.3+0.07
59.
52, 92.07
44, 940.0
COTTA PTTA
17.11 17.9
17.9a0.0 16, 1a 16.3 27. 8an.s 40. 40.40
33. 620.77 34. 14.12 49. Sanus 65. 10 17.
27.9 45.32m00
50.5.0.09 41 11.6wt.1
32.9
17.988.19 18.258.30 19. 16, 021.3 16,5 29. 791.2 42, Ial.04
36. 9s1.02 35.3a6.75 75 50.921.38 66,2st 36 20.1al.40 46, 180.5 32.1 42, 8..88
34.0a0.55
SOTTA
4a..2 39.99c.11 28. 5.0.44. 7a 25n2o 38.420.15 1s 47.620.0 [4.5 4...4 40.3am
16.1aros 66. 9amo [28. Jan22 53, 9an.2 57.5 51. 11.4
41.5a...0
PTTA
21 41.3-0 56.5am 66 66.. 37.1 54.0kn.3 57.4.0 $1.4.0 1a3.36 31,441.00 os 29.9x3.07 27. 27.4nn 26. 39, 9xm.72 44. 12 (46
42.9ka.0 16
SAR
31, 4e9.2s 31.6 32,4e0.44 29, 0s 28, 3.0.4 41.9to.11 49, 3an (47. 3..3.00.3 42. 0.0 57 4:0.02 67. 17 (36.066336961 (36.922.4 54, 54.5s0.08 58, 58.4.0.13 52. 32.300
(44,
0s..0...
PTTA
32.5 32.74: on 33.44) 37 3115-1 s4 30. 8-1.00 oo 44,2+1 16 50. 7-00d 48. 5-n 43.6-0 51 58. 1-0.44 67. 07.5-0.0 50 41.5 42 55, 25.2-mmm 59.3 41 52, 52.5 oz
45.5 es
ETA
06.0ee. 38.2es.3 37.5 4: 32, 8en.0 47. 27.76 52, Getooe
51 $1.920.1 45. 8: 59, 94004 67. 8:0.13 (45 44..19 57, 9...000 60. 800.t 55, 15.05 (48 48.2
PTTAA
37.5 39.6a8.11 39.0aa.1 35, 4atuos 34.5at.16 48. 98..5 7a.70
52. San.os 47.1 60. 9am.0 68 44.. 48. 48.2am.3 58. 61.5p0.1 55, 55.9a0m 49.5.0..0
EATA
35.5 37.6 18. 36.8 33 33 17.2.0.5 52. oz
51.6.0.11 4,5.5-o 24 59. 9 68. 1-0 11 [44 6...0.0.... 58.0
60.6 55. 1-
48.00...
PPTAA
37. 1a8.s 39.0sa.34 38.3a0.12 35, Duncs 7anz2 6an.6a to 53, Bascos
52.5amm
46.7am 20 60. 6anus 68 4ano 47.3wn.3
38, 61.2.0.11
55. 70.0
49. 44.00
DEYO
36.5 38.8
38.2 33, 33.9-0 33.5
48.6+0 7-. (22 62.6-.m0
46.3-oor oz 60. 3-0.0 68. 2-0 [44 441 58.5
61.5
55. 7-0 10
48.6+0..
PTTA
37, 9eauos 39.7at.63
39. 39.6a3.08 23.4 35, 750.3 49, 920.21 53, 8at.on
53. 27. 47.5 11 61 11.3an 11 68 Gen.ss 48 2,0.9 59, 2s0.04 62.1
56.5.0.0
49,9ss.10
CPL
29, 9ea. 32,0 00.00e..0 0s 30,8 16 27, 8-0 14 27. 7:0.13 40, 0s0.1 48, 3-0 14
(45, 1-
41.1-0 14 55, 9:00s 66 8:0 | [35 2.0.20 15013
56.8.0.00
6.0
|42. 42.8.0.. 04
PTTA
1 929.3o 82..1 33. 29. 520k25 29.42 42.520.20 50,020.1
(6. 8an.2 42.92000 57.22008 67. 67.120000 41 41.2a0.10 55.0aa117 58.3 51.5.0.1
44. 620.00
VIT-B/16
NOADAPT
66 8e 12.0es.co co 16,5...0 29, 2:0.00 23.6 00 33, 9.0.0 27. 3e0
15 0. 26. oz 47. 03 55, 4:0 (44. 330.00 31.0cn.00 45, 10.00 49, 1-0.00 (31660..0 6.600.00
TENT
44.4.a 42.74o.00 09 45.3 18 52.2--0 47. 7- 55,
50.5-
18.5.1 si 21.7-3 66 4-0mm 74, 1:
53.2
66.9A0 is 64.5
(51.34..11
PTTA
44.7 15 39.0as. 58 45.0a).15 52, Iaf 17 47.9 47.9 14 55.9a0.1
51, 8.. 15
18.7:2.7 31.6G5 02 67 440.2 75. 0s0.0 65 S50.00 54, 54.350.0 67, 37.450.07 64.3
52.
19:26
COTTA
23.6as.10 23.7a2.06 27.6 30. 620044 25, 9asous 38. $8.44 30, 92
(28 62n.24 41.5.2 51.7an.s 64.32n.17 (49. 6an.1
36.251.5 51, $1.4a0.55 53. 6:
38.5
PTTA
76.5 24.3a31. 46.443.84 54 40.5a1 39.3am05 49, 8an. 14 40. 74.9ss
27 27.3as3. 49. 59.1an 72. 58 38.88n.3 45.5 63.241.0 61.9An
48.4...8
SOTTA
44.4as.30 43.4at.3s 45.8at.on $2. butc 48. 9avzs Lan.s $2. Laterv
588.
9unzs 58. Saro: 67.0amos 75. 1a (64.
56. 7anzs
67.2x0.01 64.6wo.:
PTTA
44. 82 43.6+1 un. 46.0+ 34 53, I-0 34 740.5 53, 24 60.5-m 73 60.9-n
9-0 s4 69. 29.0aa33 75. 15.340 S1 65.9
58.4 67. 37.840.09
64.4.o m
5%
1
SAR
19.9 44.9a8.22 46. 95..1 53, Dzncos 49, 13 55, 9.0.0 51, Gen.
64, 24.440..9 59, 59.3a 59, 8an 68.2an112 75. 22002 65 65.40000 27.2 67.4a0.01 64.5a0.0
57. 2.022 52. 8:2 73 66. 32012 74. 8:0.s (64, 420.13
55.420.33
66.750.04
56.7
PPTIA
46.2a8.19 45.3a0.5 47.4a0.11 53, 3atvol 50, 62n.11 56.5a00 53.
58.0a0.04
ETA
33.3e0.01 13.4at.14 54.3a3.00 57, 620 57. 820.16 62, Lat.s 19.922.17
06..0.0.44 64. 85 72.5 77. 07.5 (67 8.n.13 66. 620.0 72. 22.2 69. 19
63. 7a0.05
+ PPTIA
54. 048.0
33.8ac.88 1a.1a 58. 22 35 63.
61. 12
67. La.1z 65. 73.3am15 77. 8anos
68 810.11 66.9 73.0ao.0
54.440..4
EATA
51. 61 $1..24.00
55.6-n 52.6+0.6
55.5 59, 8-0.s 57.5
63.0-- 61 61.6-0m6 71.0.a
76. 0-0 ds /67 00...:3 64. 50
70.3 67. 7.
61.7
PTTA
$2. 340.32 51.746.2 53.340.42 56,4.uz 56, 2atcs 61. 1anss 59.0atcss 65. 50 63. 4: 72.3ams 76. 2un2s 68 13..0 65.0aazs 71.2 68. 1ai.2
62.
DEYO
54.4 is 54.74s 18 55, 44.440.34
58. 0- 58, 450.05 63.1 1.0.0. 54, 6.n.94
66. 9: 65 8-
73.3-000 78. 2:0
(68 0.0.03
8.0.i
73.2
70.4
64.2 16
PTTA
55.0 029.0s 55.3a 56. 58, 85000 59.0a0.10 64. 62. 3atvo 15007
66 G2n 74. 0an00
78. S2009 69. 69.1 68. 120.10 73.9 70.
65.3
CPL
42.4as.v 41.0ac.1v 42.5xa.1 48, buncs 45, Jatus 49. 12.5 47, Gunca [55. ta 52, San 59, 9a 70. Lanso (59. an.1 $2, 7anz4
59.5.0.2 59. Oan.rr 52..4a..x
PTTA
41.9aa.1 40.6a8.11 43.0a3.13 50. 00.0 12 47.2 52.240.34 49.42 56.990 38.3a0 64.5anos 73, 620 63.1a0.06 14.120.0 63. a3.1 60. 60.620.14 54.628.00
parameters for test-time adaptation, which achieves a 1.2% accuracy improvement for CLIP-RN50 and a 1.1% improvement for CIIP-VIT/B-16. The introduction of our proposed PTTA improves the accuracy by 1.8% and 1.4% respectively. This validates the performance of our method on natural distribution shifts. Mcanwhile, we also carry out experiments on the single TTA task on ImageNet variants to further demonstrate the robustness of PTTA to natural distribution shifts, as shown in Table 3, achieving a maximum accuracy improvement of 2. 2.0%. It is worth noting that we compare the effects of using ID retrieval and OOD retrieval for the episodic TTA task. The experimental results show that OOD retrieval achieves comparable or higher performance than ID retrieval, with an average improvement of 0.3%. This benefits from the fact that OOD retrieval takes into account the prediction bias of DNNs caused by data distribution deviation.
D.4. Semantic Segmentation
Beyond image classification, we conduct experiments on semantic segmentation to verify the effectiveness of our proposed PTTA in practical applications, as shown in Table 5. We use the CarlaTTA dataset to conduct experiments in 5 continuously changing scenarios, including day2night, clear2fog, clear2rain, dynamic, and highway. This task simulates the environmental changes of autonomous vehicles driving in day and night, sunny, foggy, and rainy days, as well as on highways and in cities. MEMO based on episodic TTA and Tent based on continual TTA are used as baselines for comparison. We apply PTTA to these methods and achieve a maximum mloU improvement of 1.8%. This validates the applicability of our method to different application tasks and its expandability to different TTA methods.
22

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Table 15: Detailed experimental results (top-1 classification accuracy (%)) on single & continual TTA tasks using WRN-40.
We highlight the highest accuracy in bold and the second best as underline.
NOISE
BLUR
WEATHER
DIGITAL
METHOD
GAUSS.
Shor
IMPUL. DErOc
GLASS MOTION
ZOOM
SNOW
FROST
FoG
BRIT.
|CONTR. ELASTIC
PIXEL
JPEG
AVG.
SINGLE
NOADAPT
34.3am.0 40.0at 00.0ac.0 40.9aa.00 68. Danoz
66.5am. 67.7 558. 6anos 54, 48. bunoo 68. 4aa. 44.5am.0
59.7and
40.3
53.249.0
COTTA
55.0ka.t 55.44c.16. 51. 99... 66, 7. 52, 9atuss 66, Lan 66.4.
t. 60. Sarce 53. Sanos 69. 1an 62.9w..2
58,5
22.2.0.30 55. Ju 59.840.0
SAR
18.0+a.1 58.6.0.1 12 56.0 a 68, 9- 55.6xn is 67. 6. 68.3-0 62.9-n 63.3-6 47 58.5-0.7 70.8 70.8-0 64 93 60.4-0 64.9 57.2 21
62.4.a.0
TENT
58. 38.3 58.9at.17 56. O23:41 69, Os0.17 55, 929.06 67. 37.700. 19 68, 68.520.23
63. 2a0
63.7.0 58, ....5 71. 31.0sn.34/66 665 440.30 60, 720.16 65. 65.480.31
$7.4e0.3
62.6s8.01
PTTA
18.668.17 17 19.446.57 57 56.8e) 5o 69, 1:0 56, 1-0:00 68, Os0.33 33 68, 7 82 63 5.0.2 64.0:0 26 59.1=0 11 71. 1:0.0 66. 66.3.0.50 60, 9e0.34 66. 060.90 57 755 50
6 040.01
ETA
59. 2,0.11 60.2as.35 58, 440.14 69, 62 56, 921.31 33..0
644
3a0.0 64. 2.0.45 61.0an.54 71. 21.2.n.9 66.77..2 61.420.13 66.5.0.40 $8.4.0.23
63.7.
PTTA
59.648.15 60.4a6.3 59.0aa.t6 70. Dunc2 57. Sat5a 55..s 69.4
64, 6.m.12 64. 61.4au2: 71. bun.0 67.6.n:31 61.8an0s 67.0a0.4x 58, 38.99..2
64.1
EATA
54. 848 55.546.00 0s 52.2+0.00 on 67. 4-0 53, 4.0 26.050. 66.5 5...5
660.5.0.dt d6 61. I- 34.300
69.0+n 13 (62.8...22 8...1.1 58, 6en.3
62.3
55, 3
60. 60.0ee..
PTTA
58.746.9 s2 56.8.0.2 68. 5an.s 55, 94X227 67.4.0.: 67. 8.n 18
63. 33 9.. 58. 18.1 70.5 65.7
60.5
65.3v0.79
36
62.5
DEYO
59. 7:0. 60.7as.14
1...1 69, 6=n 57, Sa0.3 68, 6e0 12 69, 665
4:.4-.
64, San 62, 85n 71. 11.55n.22 34 668. 8.0.37
61.3a0.5
37.220.06. 59, 39.3
64, 64455s..0
+ +PTIA
1a.1s 60.9a8.26 61. 4a9.31 220.34 129.20 68, 620.25 69..22.24 63.5an33 65.3an 18 62, 7an22 71 8a0.2 69.83..33 62. Ian.3 67.5 39.7.0.31
64,920.00
CPL
55. 840.37
56,8 as 53.340.14 67.4 54.5av.5
66.444.2 66. Satoo 611.4. 61.6.. 55, 22 69. 6.
9. 58, 9un.o 62.940.71 56, 00
PTTA
56.528.22 57.3 54.,a 67.7a 54.9atc4 66, 8a5.77
66.8at.
61.7an.3 61. 56, Darcaz 8unc22 64 64.5am.33 59.3amu 63.5s0.41 56.420.4)
la..8
CONTINUAL.
NOADAPT
14.300.0 40,0at.co co 40.9 40.993..0 68, Oz0.00 12,020000 50.5 07.700.0 (58 Gen.0p 54, 8 48 Gen.s 68 420.00 [44, 550.00 59, 7:0.0 40, 3s0.00 57, 1.....00
(3, 2e0.00
COTTA
55.0a8.0 35.8at.4 52.5a3.10
19 74.7a
67.2a0.05
61.
61.2a0.34
6a.8aa 55.5anm0 70.1a0.0
64. 0a0.0.
60.3an.
64.3
37.2200.3
60. 8as.0
SAR
18.0a9.12 61.335.3
59, 99.00).20 68, 8an 12 57.7a0 14 67.5 69, 330
63 Gan.44 64.9a0 40 60. 7:0.22 70.6 67 0...33 62, 220.44
67.2s0.3 58, 220 25
63.78
TENT
58. 3am.1 61.2ac.79 58, 6a0.12 68, 89 57. 17.4 67.3an.25 2at. 18
63. Sun.14 64. 43 60. 60.3am22 70.1anz (67 0an.43 61.8an. 67. 7.....3 58, 28..0..2
63.66
PTTA
58.7 61.6 59.6.0.12 68, 9un 57.9 67.9k..1 69. 740.0
64.2.0017 65.3 61.2.0. 71.
68 38.8.0.3 63.0knms 68.0sa.1 59.0wo.1
644.36..0
ETA
29..1 62.2 x8. $9.2x0.22 67. 8anza 56. 9atces 66.440.2 3at
63.66.1.4
64.
10 68, (38.3an 1s (66
(6666666.... .4m.44 60. lancss
65. 30..0..2. 56, 4...4
6.3. 33.00e.1
PTTA
59. 62.6.6 4s 60.000000 0...0 31 68.5 58, 11 67.340 69, 74
64
65. 2--0 62.34- 69.
69. 1.0.11 62. 1 66.846.0 00.00. 58, 3..3..
64.
07
EATA
54. 86 55, 6a5.02 1a..s 67, 4:0.33 53, 33,420.14 66, 2:0.07 S3
600 (0.5aan.1
61. 54. 1: 69. 18 (6. 92..1
68.62 62.3 55, 35.3.0..1
60. 00.0ss.00
+ PPTIA
las.1s 19.9 57. 47.42 69.0aa.17 12 56, 8at.1o 67. 928.12 628.62
63. 6u
63. 9: 39, 39.3an22 21.32024 66. 8an.3 61. 1an.5 66.2.0.52 58.
63.220.21
DEYO
59.7 79 60.9a8.14 laa.5 65, Oztcoe $3, Gatoes 63. (3.0211.4 64, 920.74
(59 850.3 59. 6:1 as 56. 7a00 63. (3.00s1.1 (62 851.5 $3, 921.0 57.921.75 48.0a1.77
19.1asss
PTTA
6a.1e 61.5aa.2 59.4 66, 3ance 56, 56.0a 64.6 66, 843.4 62.3un 62. Sur4: 60.8 66 941.3 66 66.6.1.3 58.5 63.3x2.v 54.8a1.3
62.0aa.s
CPL
55. 840
$8.346 16. 56, 13 68, 2-n 14 31 67. 840 35 68.5-n 633 63..4-01s 19 64. 64.3-0 s 57.0.o 19 7- (66 040.0s 61. 1- 65 81.8 57.6 37.8
60.5
PTTA
56.5e8.30 59.7s8.3 57.6a3.35 68, 5:0.1 57.2 19 67.3 68, 3en 16 (2.77a. 63 620 58, 3a029 70.1 66.7a0 60.9en,
65, 65.8.0.1 56.8.n.s
62.7
D.5. Adversarial Defense
Table 16 shows the error rates of samples under the Indiscriminate and Instant Attack Scenario, while Table 17 presents the error rates of samples without any attack. MedBN (Park et al., 2024) effectively defends against attacks, for example, on CIFAR 100-C, the error rates for Tent, ETA, DeYO, and CPL decrease by 6.7%, 6.5%, 5.5%. and 7.7% under attack. However, MedBN hinders the model's 's adaptation process, leading to higher eπrог rates for samples without attacks compared to using BN Adapt (Schneider et al., 2020). Specifically, on 10FAR100-C, for Tent, the average error rate of samples without attacks is 39.4% for BN Adapt, while is 40.9% for MedBN. Incorporating PTTA can mitigate this issue. PTTA not only reduces the error rates of attacked samples but also those of samples without attacks. On ImageNet-C, PTTA also shows positive effects. For Tent, the error rate of attacked samples reduces from 69.4% to 65.4% with PTTA, and for non-ataccked samples, it decreases from 60.8% to 55.9%. This indicates that PTTA can enhance the model's 's robustness against adversarial attacks while maintaining good performance on normal samples, thereby validating its effectiveness in the adversarial defense scenario.
E. Additional Discussions
E.1. Effecc of Saliency Indicator
In Fig. 10, we visualize the saliency distance among test samples under 4 types of image corruptions, namely Gaussian noise (Gauss.), Defocus blur (Defoc.), Snow (Snow), and Contrast (Contr.). We sort the test samples in each type of image corruption in ascending order of the entropy predicted by DNNs and split them according to percentages. We calculate the average saliency distance between samples in each split and those in other splits, which is normalized to the range of 0 ~ 1. According to the sample selection criterion based on entropy, low-entrops samples can be regarded as benign samples for purifying malicious high-entropy samples. Therefore, we hope that all malicious samples can retrieve corresponding benign samples rather than malicious ones in the memory bank or within the mini-batch. A good saliency indicator should make the saliency distance between malicious and benign samples greater than that between malicious samples and less than that
23

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Table 16: Error Rate (%) of adversarial attacked samples under Indiscriminate and Instant Attack Scenario.
NoiSE
BLUR
WEATHER
DIGITAL
METROD
GAUSS.
SHOT
IMPUL..
DEFOC GLASS MOTION
ZOOM
SNOW
FROST
FoG
BRIT.
CONTR. ELASTIC PIXEL JPEG
Ava. J
CIFARI00-C
TENT
BN ADAPT
60.7 59. 59.3am
67.4
45. 10 59.8a0.1 48. 58..s 43.1mn0 511 11.9 53.2 55. 94nas 44.8arnd 50.7. 36.0. 46. 46..00..0 55, 55.9amm 53. 3.0 M
PTTA
18.1a 57.0au17
65.5am
44.5amos 58,330.43 48. Zanoz 42.0aa.34
20.0 51.0ka.10 55.3an.20 43.33m 51 51.5am83 55, Oancr 44, 6.0.37 55.5a 52, Oan I8
MEDBN
54.4.0.0 22
38. 78 54.5a 41.4
36.0wn.:
46. Jam.0 45. 74.31 48. 1.1.0 58.2.m55
42.4amn 49.6...7 39.5a0.00 50.1am.7 46. bu
PTTA
52.7a8.54 51.83
61.7a0.2:
37.9 52.420.39 41.1a 35.3
444.3an.2 44.320.39 47. 37.3at 44.33an.0 291.22 33.990.55 49.7a125 45.7.0.3
ETA
BN ADAPT
59.4ac.ax 58. ma 66.7at.0
(44.5... 58.7a0.z 17.99
42.5.m.34 (50.8an1 52.6 54, Bat.7I 44.1 1am
999.9
55.5un.42 45.6acon 55, 35.4arro 52, 4un.s
PTTA
57.428.22 36.8a 65.2a0.33
44.,4a0000
17.142247 47.350.3 41.669.16 49.7amms 50.550.34 54.3at.14 42.7am4:
51.33a554 54,4 44.220028 54,5am012 5:.5
MEDBN
53.9uc.42 52.3am24 60.8
37 52.3w0.3
40.8au.7
36.0an.t
45.3amm2 46.0aa.14 47. 4an.z2 38.2w1.7
41.9u 48. 6ue.s 38. 2a0o0 49.4amm 45, 9un
PTTA
52.728.9 51.5aros 6a.12
37.1a0 1a..4 40. 35.6,033
44.5am.s 44.4 46, 920.26 37.2a0 55
41.77a.06 48.3 37.420.45 19.1. 35.3.0
DEYO
ADAFT
61.84 60.0aa.50 68.3an..4
15.
61.8a0.44
50.3anos 43.440.24
13.1am1
54.2aa.31 56. 566.27 45.0wn
(51.4am31
57.3.0.27 46. Ban.ss 56.9uu. 54, 2.0.0
PTTA
58, 8ad.0e 57, 32 65. (53.3
14
58..300.1 48. Ia0.29 42.2st.5:
50.7ano.
$1,35b011 55, 35..3
13.55099
+5.020.57 23.2z 52, 3an 52. 0=0:57
MEDBN
15 54, 4-n.s 64.6xt.3
40. 15
36.7e0.56 45.0+0.:
38.3.0.3
47.5.n
48.
49, 6:0.37 39.5a0 36
44.6.00m
40, 740 9.00 48. 7:001
PTTA
54.0a6.35 53.1 1an.1 62.3s9.31 ss
38. 38.8. 53.820..4 41.750.00 37.1 44.92a.o 45.723.64 47. 880.27 37 37..8
42 42.42 201.0110 38, 820.45 50.2an.27 46. 6at 10
CPL
RN ADAPT
61 4 60. 6-00 1o 68.7.0 is (46.4.. 4..4 ss. 60.8 11 51. $1.0aooss is 43.7 (53. 740 53.940.x 6..5-o 45.6.a h: (54.3-0 56, 7-0 46, 9-0 57. 67.9.0.s 54, 9
PTTA
60. 0s0.40 59, Sa00a 67.0
47. Oao.30 30 59.22.0.3 51, (40.31 44.2a0.55
55.000ss 53. Pe.0e..o 61.9 46.3000
56, 2:0.09 46, 86.77 57.7a004 54, 6e0 12 44.5-0mm
La0.2
MEDEN
54.0 14.00..57 53.9 9..--mm 24 62.0 in
52.660.17 17 42. 940.5 36.4
1-.1-o
45. 15.440.5 5 8-0 19 38.7-0
48, 8-0 s 39. 34..00 5 51.7-0.7 47. 27.2-0m
PTTA
53. 78847 53, 450.5 61.2a0.5)
39. 90,990.3 52.4 42. 92nos 37.430.3
45.4 44.920.11 52, 750.6
38.92073
44.750.4 48.5.007 39.5.0.5 51.12071 12.12
IMAGENET-C
TENT
ADAPT
84.4ac.0 80.4
40.4m
81.8 53 88.4a0.4 69.221.00
31.0aa.3s 660.8 76. Laoas 55.2a0.11 41.7am (95.0a 53.5.aas 50.5aa.37 55.1an 69. 4
PTTA
78.8-1
76. 44
74.5 10
83.6.0 so 84.0
35..0.0.0
59.340,37
58, an 71 [..4 06 54, 440.0 411.9a 17 (75.6-5 70 53, 0-n.ns 50. 1-0000 54.4-0 na 65, 4-0
MEDBN
65.7.
84.0amar
83.6x0.13
85 86.441.4 72.5.1.50 58.0a0.
61.6w0s0 70.740.0 5anda 35. 88.8.a.cs
53.9.03s 47.0am0s 55.0ama 67. 8
PTTA
85.4+0:00 85.8+000 82.0ka no
81.00a no 8442200.0 69.0=e..
57.3 00 60. 6-0.0 65.340.00 48, 860.t0 36.240m 76.2-000 53, 7-0.0 48.0<000 56.4-0m 66 0-0.m
ETA
BN ADAPT
76.5 12. 35. 678.6 78.5 65 burcs4 58.6
(57.2amus 64.4a00s 53. 9anos 40.5amm 72.22ams5
51.8. 49.0aa.1s 53.5amx 63, 2a
PTTA
140.15 71.344 S4 73.940 10
77.440. 19 77.4 14.22
57.5
56.5.a
63.34 53, 24 40.2-am
69.5-a
51, 51.4-033 48.54005 $2.7-m 3i 62. 1-o
MEDBN
82.0uo.e2
80.5au 80.7am.:
80.9an
81.4a0.: 69.4am31
56.9ta.3
59.9an3s
62.8 47. 9an.1 34.9an11
76.33
52.2.000 47.0aa 54.9amss 5..
PTTA
83.3 82, 82.62000 82.8a0.00
81.6
82.3e0.00 68.77 56.720.0
58. 63000
62.9.0.0 48..0 35.5
75.3amu0 53.7.0.00 47.740000 56.0a00 120.00
DEYO
BN ADAPT
74.3a 70.7auss 73.4a0.22
77..
17.88axs
77.220.06 64.6an20 59.1 $9.1an 56.66n0s 64. 64.3
14.1a 41. 222
(.1.6 Maauss 49. 49.3a0.2 $3. $3.221 62, 6an.s
PTTA
826.12 70. 10.1 43
76.
16.9a0.06 76.990.3 63. 850.2
58.5
$6.3an.s 63.820.12 53.
41.0aa33
10..1...9 5:0.1 48.94041
92.95 12.5
MEDBN
83.021.27
2an2s 81.7a0.4
#2.2amx3
82, Mac.3 7an.a
$7.3a0.24
60.4am29 63.7a0.26 48, 2at.46 35 25.22024
78. Danca
52. 3u0.22
55.0ar.9 65. 3a 47.665.42
PTTA
13.00
82. 629.00
11.400000 82.2s0.00 950.00 57.420.8
39.8an03
64.3220.0 49.300.00 35.7a003 76.920.00 53.5-0000 47.776000 15000 65, 620.00
CPL
BN ADAPT
78.6 73. 8a1.1 76.920.s
[81.0aa.15
81.2a0..9 68. 9an.o
61.2an.0 (60.8an20 67. 67.6e007 57. Ast.1 42, 20.00o (79.6 VV.66a.00
44.4 51.9ua27 56.3aus 66. 1ator
PTTA
75.9:8.0
75.4a0.5
34.3
78.779030
06n.4
59.3 59, 0e0.1 65, 669.04 55, 9a0, 42.22:0
70.990.14
830007 50, 8a0.6 $5.00 64, 2:000
MEDBN
70.72
79. 9anci
28.830
50.8a0.5
81.440.32
$7.2
61.42mus
63.6e0.24 49. 85n.31 36.3w023
78.0a1.5 52, 6un.4 48. 740.r9 55, 25.12 64, 92.92
PTTA
82.5st 80. 85000
80.8 80.9 81.800.00 70.7s0.00 58.3 61. 8:0.0 64.469.00 50, 720.00 37 (7.88000) 76.95000 54, 0:0.00 49, Ge0.00 56.550000 65, 8:0.00
between benign samples. This can prevent malicious samples from retrieving malicious samples and benign samples from retrieving malicious samples for Mixup. A typical example is shown in Fig. 10 (d). We compare the logit-, feature- and pixel-saliency mentioned in Sec. 3.2, as shown in Fig. 10. In general, the logit-saliency indicator has characteristics that better meet the above requirements compared to the other two and is applicable to different DNNs, including ResNet50 and ViT-B/16. However, it is not very stable in representing different distribution shifts. The feature-saliency indicator is slightly weaker than the logit one but can still meet the above requirements. The pixel-saliency indicator fails to accurately represent the above requirements. Moreover, the effect of pixel-saliency distance on ResNet50 cannot meet the basic requirement of retrieving benign samples for malicious samples, while ViT-B/16 can meet this basic requirement. Overall, the logit-saliency indicator has better effect in encoding benign and malicious samples, incurs lower computational overhead, and is more suitable for purifying malicious samples.
E.2. ID, OOD, and IB Retrieval
In-Distribution (ID) retrieval uses benign samples sampled from the source training distribution candidate samples. (OOD) retrieval uses samples from the test data distribution sampled during the test-time adaptation process as candidate samples, which are updated in real-time. In-Batch (IB) Retrieval, without building an additional memory bank, uses test samples in the mini-batch at time step f as benign candidate samples. For TTA tasks that continuously update the parameters of DNNs, including single, continual, and lifelong tasks, OOD Retrieval utilizes test samples in the recent historical period as candidates and maintains consistency with the model updates. However, for episodic tasks, where the
24

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Table 17: Error Rate (%) of samples without being attacked.
NOiSE
BLUR
WEATHER
DIGITAL
METROD
GAUSS.
SHOT
IMPUL..
DEFOC
GLASS MOTION
ZOOM
SNOW
FROST
FoG
BRIT.
CONTR.
ELASTIC
PIXEL
JPEG
AvG. J
CIFARI00-C
TENT
BN ADAPT
46.840.36
45.9 1s
54.9w0.1
31 s1 46.5a0.0 34, 74nms 29.5 13. s4
38.5 40. 240.s 31. 31.9wa
12 42. Less 33.33 a4. 39. 4.0.1
PTTA
45.2a6.19 44.4am22
33.8a0.3:
30.1an
45.5a0.24 33. 69n.19 28.6...2
37.5an24
37,0xa.0
8an.22
30.9wss:
31.4an.5 41,0aa02
61.61 43.2am57 38. 2ao to
MEDBN
49.740.35 47.94u.7
57.5
12.0ar 48, 7a0.3 35..snms
30.5wm.n
39.
4...
33.0wrm
13.7.
13.5...15 33.8ac.07 45.9.m02 40. 9.0.0
PTTA
47.523.45 46.5at.0 56.7a0.33
31.0aasss 46.9at.3 34.3an07 29.420.35
39.3anm0 38.12 40.2at.01 31.5u033 33.0an30 42, 8.nat 32, 22..4 44.5am5: 39, 12
ETA
BN ADAPT
6. 44. s4
54.1a0.
331.5..: 45.3.0.
14.0am 29.5am.:3
38..2 37, 37.8 39, Danuss 31.6am0s
331.8anto
41.5am.75 32.6a0.3 43.3amts 38. 7an.3
PTTA
44.628.12 44.7am.13 53.6a0.41
29.8ap:3 43.990.01
33.330:22
28.5
37.0gnae
36.1s344 37.7a0.1 30.4a039
33.33n0 40,421.30 31.920.14 42.3am04 37.72037
MEDBN
28.2.
47, 4a
$6.1
32.2...s 47.0wa.21 34.9
30.2
39, 99.99m 39.7 40. Sanoo 32.5uu.0
33.5.n.9 42. Bun.e 33.440.s 45.4amms 40. 2.0.
PTTA
46.628.15 45.9am1s 56.0a0.3
30.4a0.40
45.6a0.11 33, 33.920 29.2.0.14
38.9 37.9.0.2 39.3at.00 31.5a0.15
32.3an. 41.921.02 31.720.18 44.2.0.41 39.020.0
DEYO
ADAFT
92..0 47.4 56.3a0.57
34.1 49.3a0.07 37. 97.99 31.9
(40.
60.6.113
40.0aa.03
33.3an0s
333.6n.44 45, Zatur 34, 6an.3 46.4am. 41. 4an.os
PTTA
32.328 45 Oanos 54.2a).23
45.0a0.29 34, Dan.19 29.7an.1
37
37. 1a3.22 39.
020.6e
31.1
La.
1:045
32.5e0.00 43.3am14 38. 7
MEDBN
51. 1 49. 3:0.33
60.6+1.5
35 51.0=0.1
39.33
33,4
42.4:0
41.8 43,
34.7.0
35.2.0
46, 9-0.0 35. 840.00 47.9.033 43, 2:0
PTTA
48.3 47.5an.13 57.2s0.33 92
31.330.13
47.5a0.24
8an.55
30.8.0.5
39.1an.24
1a0.23 40. 330.00
32.0 32.0go:3
32.4400.0 33.92023 33.5 45.0an.7 40, 2a0 to
CPL
RN ADAPT
45.74 45.5 ox 55. 1:
3 45.3+0
2- 30.0 [40.0.0 38.55 433 6-0 32.2. ss (34.6-0 42. I- 33, 83.5 45, 45.2-0 57 39. 9-0.
PTTA
45.7:6.55
45 8a0os 54.5
31,55 44.6
75..22 29. 39.9
1203 38.629.30 44, Detc4i 32, 32.5.0.
34.7
41, Ge0.02 33, 33.22 45, 45.3a0 39, 8
MEDEN
51.2-1.9 51.0+1 35 49.243
35.645 49.5+147 39.8-1.9 33.3
43.442
42.443.7 48. 2:4.9 35 8.. 50
39.8+6 5a 45. 9-1 06 36.343 18 49.3+c os 44, D-
PTTA
17.1 47.2
55.8
32.0ao35
45.8 0s0.2 30.2s0.4
(40. 65. 39.5.0.25 44. 750.6 32. 2..9
120k 22.32 33.628 46.2a0.4 40, 6en 19
IMAGENET-C
TENT
ADAPT
77.4 74.6.m 74.5xo.x [80.1
81.7atas 58, 44.4 49.9am.e 62.6 67.330.21
41.5 33.0amm (89, 19 13.8-..10 8anas 46.8.a 60. 8-0
PTTA
71.0-1:91 73.2-1
66.340 1a
75.
76.2-1.4 54.0+o
47.8 36 49. 8-ont 61.741 51. 40, 340.1 32.440 os
62.66x633
9- 40.0-000 45.4-000 55, 9-0
MEDBN
84.9...9
83.8amm
83
85.0aa.2 85.8 71.4.1.38
56.6
61.1
70.240.04
17.14 34.3amm
58.9 52. 22.4.0.3 46. 240227 53.8
67.0...1
PTTA
84. 3...6-000
85.7-000 81,5 no
80.3 no
83,3.0.00 67.3+0.0 55.7
59.7-00
64.240.0 47. 340.00 35.0.am 76.2-000 51.7-000 46,8-n 440.00 50.8-000 64, 9-0
ETA
BN ADAPT
66.6.
64.2auz7
(68.5an 68.8a0.24 53. 9u1177 47.22m.
[48.5
54.4a0.3 39.7an.s 32.1amm (57.8an2
4..8a 39.2as.te 44.8amm 2an.
PTTA
65.4
63. 1-
63.8 24
66. 67.55a.. 22 46, ka.m
47. 7..6.mm
53, 140.34 39. 00..0...0.. 31.54a
55.00-a0
41.4-0m 38, 640.0 44.1-000 51.7-om
MEDBN
81.0uo.m
80.6amm
80.4a0ax
80.7am. 81.3a0. 688 6unoo 55.6,n.0
59.77
62.55a.te
4a.1a 34.1an11
76.7a 50.7.0.s lanss 53.7amzz 63, 9.
PTTA
82.3 82.2amu
82.2a0.00 81.550000 81.7a0.00 67.22000 55, 55.0a0.0
58. 62.3e0.00 46, 220.00 34.5 76.52m00 51.8.000 46.320.0 54.5200 22.2.0.00
DEYO
BN ADAFT
65. 020.09
63.1
63.4a0.:e
(67 a3uus 67.5a0.21 $2.3ama 46.9 [47. 52112 53. 13..69 19.11 31.8a
(56. Janas 11.1a 38, Sa0.os 44].0anos $1. $an.os
PTTA
64.5 62.3a0.4
63.2a0.11
66./41..44
66 8s0.54 43.4s 46.3
1..0..4
13.93
38. 31.
620.35
84.861
38.360 43, 8an2s 51.3 60.7.
8:0.12
MEDBN
82.423.32
82.5am 81.5aa.22
81.6aru 82.6u0.4 688 Kanoi
56.0w0.2
60. Dan.s
63.220.22 47. 2a0.39 34, 22mr4
46. 4as.o S3.8a 6a0.0s
PTTA
82.925.0 82.7 81.8
80.9a000 81.720.00 68.550.0
55.620.0
59. $9.420.03 63.220.00 47. 270.00 34.5a000
77.420.00 51.3:0000 46.0s5.00 54.4 64.5.0.0
CPL
BN ADAPT
220.2 1a.1
67.6
72.4
57.22004
49.1an.1e [51, 6.6os $6.8.001 41, 9an.ss 32.7amus
(66.5
43.3a 41, 41.24004 46.8aun 55, 620.6
PTTA
66. 858.49 7..75 65.2a9.0
68.9a9.:
Ta.. 54.
44.4 47.4 49. Sano: 54.959)25 40, 7:0.0 32.7a0m0 56.99622
42,9:021 40, Ta0.00 46, 26.00 53, 33000
MEDBN
80, 12.. 79. 6un.s
80.4
11.1 68. 9asos 16.1.
61.1anus 63.4 48, 9an.ss 35.5
78.721.56
11.1 87.84 54.3am 64.42 to
PTTA
8 258.00 80.2sno0 80.0sa.00
80. 81.2200.00 69. 0e0.00 56.4
61.22000 63.5ss..0 48. 920.00 36.5 76.880.00 8:.000 48.500000 $5.0.000 64, 7:0.00
model's parameters are reset after making predictions on the mini-batch at time step f, ID Retrieval better represents the data distribution related to DNNs' parameters, because DNNs generally have higher confidence in ID samples. The experimental results in Table 4 and Table 19 show that for episodic tasks, ID Retrieval performs better, while for the other three TTA tasks, OOD Retrieval has better performance. Meanwhile, due the advantage of not requiring a memory bank, IB Retrieval reduces the storage and retrieval computational overhead.
E.3. Necessity of Benign Sample Retrieval
Different from vanilla Mixup (Zhang et al., 2018), we employ a saliency indicator to retrieve benign samples that exert completely opposite influences on the objective function relative to individual malicious samples, forming the matched pairs for Mixup. Considering the use of Mixup technique (Zhang et al., 2018) to purify malicious samples, we compare with the vanilla Mixup (Zhang et al., 2018) to demonstrate the necessity of benign sample retrieval, which stores historical test samples in a memory bank (maximum length 1, 000) and randomly selects samples from it to mixup with the samples in current mini- batch at time step t. Results in Table 18 show that while vanilla Mixup can improve the performance of base TTA algorithms in some cases, our proposed PTTA demonstrates superior performance and robustness.
E.4. Unnecessary of Sample Selection Criteria
We claim that benign samples selected based on the sample selection criteria should be used as candidates for purifying malicious samples. The reason is that benign samples cause less disruption to DNNs during test-time adaptation compared
25

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Table 18: The comparison of PTTA with the vanilla Mixup method. Also the comparison of different purification scopes in PTTA, which purify all test samples by default.
SINGLE
CONTINUAL
METHODS
ETA
DEYO
CPL
AvG.
ETA
DEYO
CPL
AVG.
BASE ALGORITHM
48.2
48.6
42.8
46.5
62.2
62.2
55.5
60.0
+ VANILLA MIXUP (ZHANG ET AL.. 2018)
48.6
49.1
38.2
45.3
64.7
64.5
55.9
61.7
(w/ 1K MEMORY BANK)
+ PURIFY ONLY BENIGN SAMPLES
48.9
49.6
44.4
47.6
65.2
65.3
59.5
63.3
(W/ OOD RETRIEVAL)
+ PURIFY ONLY MALICIOUS SAMPLES
49.1
49.6
44.3
47.7
64.7
65.3
59.0
63.0
(W/ OOD RETRIEVAL)
+ PTTA (W/ OOD RETRIEVAL)
49.5
49.9
44.6
48.0
65.3
65.8
59.6
63.5
to malicious samples, thus stabilizing the model's optimization and self-training processes. Existing sample selection criteria are highly dependent on the choice of thresholds. The threshold determines which samples are benign and which are malicious. However, different DNNs often require different thresholds for sample selection when facing different distribution shifts. Therefore, the sensitivity to the threshold determines the practicality and portability of existing TTA methods. To avoid the impact of sample selection thresholds on PTTA, we conduct experiments using all test samples candidates for purifying malicious samples. That is, instead of using sample selection to distinguish between benign and malicious samples, all test samples are stored in the memory bank or within the mini-batch. Ideally, as stated in Appendix F. an excellent saliency indicator can retrieve benign samples that effectively purify malicious samples from a candidate pool mixed with benign and malicious samples, while ensuring that benign samples are not disrupted by malicious samples. Consequently, the experimental results in Fig. 5 verify the performance of Tent+PTTA with or without sample selection. With the logit-saliency indicator, PTTA effectively purify malicious samples without the assistance of sample selection. This further validates the effectiveness of the saliency indicator we proposed in Sec. 3.2.
49.5
$0.0
(00 Conenoo
65
Accuracy
40
A Accuracy
44.5
Accuracy
-9.00
-4
Accuracy
Accuracy LY
DeYO Bagle PTTA
DeYo (Gotirl
(Megle) (singc
43.5-
PITA (TTpai
49.0
35
43.0
3
4
Value of a
Value of a
Value of a
Value of
Figure 11: Ablation study of in Eq. 8. We compare the performance of PTTA when applied to Tent, ETA. DeYO, and
CPI. for Single & Continual TTA tasks.
E.5. Ablation Study of a in Eq. 8
The hyperparameter a in Eq. 8 is used to balance the loss function of the original TTA method Lita and the purification loss we proposed Cpur- In the main experiments, we set a = 3.0 by default for sample-selection- based TTA methods and methods, such as ETA and DeYO, assign larger weights to
= 1.0 for selection-free methods. Since sample-selection-ee high-confident samples with low entropy, introducing can balance the effects of these two losses. In Fig. 11, we compare the impacts of different values of on the application of PTTA to Tent, ETA, DeYO, and CPL. The experimental results confirm that our default settings are appropriate.
E.6. Different Size of Memory Bank
In the pipeline we proposed, a memory bank designed to store selected benign samples. Subsequently, PTTA retrieves benign samples from the memory bank that match the malicious test samples for Mixup in the purification process. The memory bank is defined as a fust-in-first-ou (FIFO) queue structure with a limited length. The size of the memory bank directly affects the scope of benign sample retrieval, thus influencing the purification effect of PTTA. Meanwhile, it also determines the amount of storage space needed to store benign samples, their predicted pseudo-labels, and the calculated saliency indicator. However, we also propose an alternative without using the memory bank, where the retrieval is directly conducted within the test samples of the mini- arriving at the t-th time step. This In Batch retrieval method eliminates
26

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Table 19: We demonstrate the generality of PTTA across various sample selection criteria. Meanwhile, we compare the performance of PTTA with different saliency indicators, types of retrieval, and sample purification scopes. We count and compare the number of forward (#FWds) and backward (#Bwds) propagations for different TTA methods.
SINGLE DEYO
CONTINUAL DEYO
METHODS
TENT
ETA
CPL.
AVG.
TENT
ETA
CPL.
AvG.
NFWDS
#BWDS
COMPARISON WITH EXISTING PURIBICATION (PURIF.) METHODS.
BASE ALGORITHM
02
48. 2.
48.6us.11
42.8
45.6wus
14.2ates
62.2aum
62.2ans
55.5
48.5x2.71
Nostch Noutch Nostch X
Noatch
PURIE W/ FGSM
12.0a017
04.92004
47.0st.0
12.1
44.2,0.03
2.7
61.7amm
62.2:0.9
56.428.5
45.7aa.5
Nistch X
PURIE, w/ SOAP (5 TTER)
30.8
42.4 07
47.7
36.6+340
40.7sa.3
10,7
57.0c7.0
64, I-0m
49. 193.33
45.2423.
Nostch X G
ABLATION STUDY OF DIFFRRENT SALIENCY INDICATORS (IND.).
PAXEL-SALL IND.
44
49.0
50.0.c.09
43.0-a8s
46.5.005
28
66.0-am 65.6-am
59.3+a.0
54.9
Noutch
Nustch X
FEATURE-SALL IND.
49. 10000
49. 95t.08
43, 43.06
46.6.0.1
14.3e11.6
65. 4:0.0
65 8:0m
58, 9e0.06
11.
Nostch X
Nestoh X
LOLIIT-SALL IND.
49.5.mm 04
49.94a.1
44.6-0.8
47.2.005
60.1.0.s
65.3-005
65.8-am
59.6.8.1
62.740.11
Nacton X.2
Noateh
ABLATION STUDY OF DIFFERENT TYPES OF RETRIEVAL (RET.).
ID RET. (Nim= 1&)
42.6w
25
48. 38.332
13.1a
30.0a17. 45.5wm24
64.4
63. 63.3
58, 2a..x.
54.0a1.3
Nostch x2
Noatos Noatch Noatos
OOD RET. (Nima 1k)
05
49.92s.10 49.5.0.00
44.622.5
47.2am0
60.1s000
65.3am0s
65.8.m0
59.628.01
62.7x0.4
Nostch X22
IB RUT. (Nos = 0)
03
20.3.0000 08
50.0s8.04
43.6x0.06
46.8.005
31.84127
65.5.003
65.8.a00
59. Le..6
55.5
Noston X
ABLATION
PURIFICATION SCOPEs.
PURIF. ALL SAMPS. PURIE. MALL SAMPS.
49.5-mm 05
49.9 10
44.6-0.s
47.2-005
00.1
65.3-000
65.8-0.1
59.6.a 01
62.740.0
Noztch 2
Nistch
45.0
49. Get.06
44.3.0.06
47.0
11.4
64.75004
65, 3a0.0s
59, 020.0
40.0x2.0
Nostch X
Nuntoh
the need construct a memory bank, reducing storage overhead. But it also restricts the scope of benign sample retrieval and degrades the performance of PTTA. In Fig. 7, we compare the effects of constructing memory banks of different sizes on the application of PTTA to Tent, ETA, De and CPL. Here, a maximum length of the memory bank (MB) equal to O indicates that we use In- Bn-ch retrieval for PTTA. The experimental results show that a memory bank of an appropriate size, such as the default maximum queue length of 1, 000, can achieve higher performance. While an extremely large-length memory bank will lead to performance degradation. Due to the impact of the dynamically changing environment in TTA tasks, storing historical samples adjacent to the current time step t is the best choice.
E.7. Comparison of Retrieving Top-K Benign Samples
In Sec. 3.3, we retrieve the top-K samples with the largest saliency distance from either the memory bank or within the mini-batch for Mixuping with malicious samples. We uniformly set K" = 1, and the À used for calculating the linear interpolation in Mixup is set as À 1/(K + 1). Intuitively, we think that the larger the value of K, the less the components of malicious samples retained in the mixed samples, and the less damage to PTTA. However, we compare the experimental results using K 1, and 3, as shown in Table 7. The ablation experiment results show that the value of K does not affect the performance of PTTA. On the contrary, larger value of K will dilute the information of malicious samples in the purified samples. When the value of K approaches infinity, it can be approximately considered that the malicious samples have been completely filtered out, so that DNNs will not learn the potentially useful information of malicious samples. Therefore, we default to set K to maximize the retention of the original information in the purified samples.
E.8. Single-Sample Test- Tism-TimeAo Adaptation
To verify the performance of our proposed PTTA under the single-sample TTA setting, where the batch size equals 1, we conduct experiments on single and continual TTA tasks. Single-sample TTA requires that TTA algorithms learn information about the target data distribution from a single test sample and ensure that DNNs do not suffer from severe interference during online adaptation, leading over-fitting and catastrophic forgetting. This is more challenging task compared to multi-sample TTA and is closer to real- world applications. The experimental results selection-free Tent exhibits a significant performance degradation in continual tasks, while the application of PTTA can alleviate this problem. Meanwhile, the application of PTTA in sample-selection-basee TTA methods can also enhance the performance of the original algorithms.
shown in Table 20. The
E.9. Purification via FGSM, SOAP, and DDA
As stated in Sec. 3.2, motivated by Adversarial Purification (Tang & Zhang, 2024), we propose PTTA. A common approach in adversarial purification is based on FGSM (Goodfellow et al., 2014). Specifically, the partial derivative of the objective function with respect to pixels is used to determine a fixed step size noise superimposed to the original image. SOAP (Shi et al., 2021) is another adversarial purification method that iteratively superimposes purification noise onto the original
27

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Table 20: Detailed experimental results (top-1 classification accuracy (%) on single and continual TTA tasks with the batch size equals 1. We highlight the highest accuracy in bold and the second best as underline.
NoISE
BLUR
WEATHER
DIGITAL
METROD
GAUSS.
SHOT
IMPUI..
DEPOC
GLAS
MOTION
7O0M
SNOW
FROST
Fon
Barn.
CONTR.
ELASTIC
PIXRI.
JPBG
AVG.
SINGLE
NOADAPT
16. 8atoo 12. 12.0am0
16.5
29.2am00
23, Gannoo 33.90am0
27.3arco 15. 15.8o 26, Gan 47. 37.50000
45.4
44.3
31 11.0an00
49. 1oa.
31 1.600.0
SAR
45 2e0.00 43.0eo
46.2se
13
50,6 34 57.5e000 53,2 15 (58.8
55, 823 68, 68.8 74,8m0 s:
65.743.1
58.3
68.9ea.1
57.8-0.4 66.4
TENT
14.5am. 43. Oanus
45.5aa.1e
52.5urz
48.1am.s
50.7 55.5ua.1
1...1
17.5
66.6aus
75.0wo. 19
64.842.7
53.6un.44
66.9aa.1
64.7an.1
51.0.n.24
PTTA
44. 740.33 13.0wa:s
46.2xb/1
36
19 56.0w6.3
31.9
11.44...5
26.3
67.7am
75.2
65.6x308
$4.2.4.3
67.5x3.1
64.4
51.9.0.5
ETA
50. 1.9.3 49. 19 21.2s0.2 56, 76.70
55.7am.6 61.1s 57.8
63. 1s
12.1
77. 0z
67.522.04
64. 6.6.2
71.8500s
69.
62. 1:0.54
PTTA
50. 9: 49.744 51.948/1
57,340 in 56, 1-0 18 61.6-8m7 05 (65.0+a.4 64.6
72,8 in 77.6 11
68.3+331
65.144.7
69
220
62.7-0.4
RATA
5.r.5 30. sn 35.4 35.4.3.1
53
39.4.1.7
45 41. 41 (36. I-5.0 41.6x307
61.4+1
is
14
46. 16
21
10.22x1.0 as
46.8-3.1
PTTA
35 5.2.3 31.662 36.5x2.01
44.7
40.4at.6 47.1. 42.9a1.6 (40.1 14.6
62.4a1.1
67.5Al 76
62. 22..33
47.3at.8
60.7a1.53
59 5.1.50
48.2.1.9
DEYO
55. 1e9.00 55. 2: 56.4
58. 08
59, 3a
64, 3eam 8exx.n4
688 Ocn.ss 66, 5a0
73.7am.s
78, 4804
68.3
68. 850.0
74 Dea.s
71. 11.00a.ss
63.9-0.5
PTTA
55. 922.0 56.3a122
56.92ncs
19.5an
60.1a 64.92a.s 63.3an34 (68,9anus 67.2an 10
74.3am0
78. 15
69.
69.42am
74.3
71 71.25
66,0.0.5
CPI.
14
30. 6wams
32.7-
42.5
06.4 43.9 a. 40.2.033 (44 0...1 42. 1s
54.7
21
42. 12.9
54 24.2....s
6x
8...5
PTTA
35 4 33.26023 36, 1s8.0
45.5a0 t0
39,750.12 12 46 8eans 43.8 iV 29:2260:2 50.5eo 19 29.2 69.9ts 22
58. Beb.21
46.889.0
57.763.0
56 859.13
48 65..9
CONTINUAL
NOADAPT
16, 8:0.00
12.0+am
m
23.6
m3
to
(15.8 26, 6m
47.5
55.4
44, 14.3-3
31 04a.0s
49. 10..10..0
31, 11.6-0.0
SAR
45.2a0.03 3a.12
57.9s802
53.220us
56.6a00
59.7aa.u
57.2
61.4
64.0a
67.4at
74.2a0
66.0aa:2
60.2s0.2
67.5a199
69.3a
61.0.a.9
TENT
44.5 14..4
25.2
17.4.28.0
14.7a25.33
0,5an.56
0.2s0.07
0.3
1.
0.140.00
0.448.2
0.1amm
0,1an0
0. oo
10.2.
PTTA
44.7a8.33 51.42014
55.0aa.14
51.77000
53.4an:1
58.5um.9
56.9
(57.7a830
47.663483
27.2236.33
22.1242.4
21.775744
19.3as3
23.3240.1
22.7.2011
41.2246.17
ETA
50.144.33 55.44435
56. 36.6+0446
54.5
16.8-2.1
54.5
(58.
$9.340
65.5wmm
73,7 14
57.34144
38.646.3
65.743
62.244
58.7
PTTA
49.7s9.12 58.4
59.7s8.5
1angs
59,5a000 63 63.2e9. 63.0am.0
(65.3a
66.2
72.5a005
77.530
67,
66.449.2
73.1a00s
70.669.0
64, 64.6
EATA
35. 322.2 35.5
la2.33
44.7a135
12.1atao 47. 47.9a143
43.6a1.5
(39.6.4.11
46.0as
62.4a148
67.3a1 no
61. 7a1.3
47.2a1.94
60.9a1.57
60 8at.ts
49.0.2.0
PTTA
14.9 38.443
40.4+3/46
45.341
42.7+1 95 48 67 45.141
(44.6-141
50.5+s
64.34166
68.741
63.04344
48.241
61.44
60.1-1.03
50.2-3.0
DEYO
55. 1an.2 $9. 02n.1
59.428.2
15.2
58.2an15
63,32 25 35.00s.s
(63.4an.1 64. 120
70.3a021
76, 26.929.1 10
62, 7a200
67. 92..9..2
71.923.12
69 62s.0
61.3:0000
PTTA
0s 60. 3s0.0s 60.7s88)
57.9
61.15006 65 Des.a 64.6x033 667 6:8.6 67.22000 73.2
78 Oso
68.249.9
69
850.1
74.269.0
66.3-0.0
CPL
34. 1e 39. 9. 46.5+9/07
43.0s0.s to
43.7.0.12 12 51 Se0.0 47 37.66m11 12 (50 50.9ce.1 is 56. 250 61.3
56, 3e3.2
53 43.15
63 6e
64 8e0.0
52.5
PTTA
35 15.4 42.64000 49.943/0 47.6
49.040 58,248m1 52.3 (58.7480 6..
67.2-mm
0:
62.243:18
55.646.11
66.1
66.244.x
56.246.8
image over multiple cycles (5 iterations by default). In Table 19, we compare with the performance of using Adversarial Purification to purify test samples for TTA, including FGSM and SOAP. The results show that the method based on Adversarial Purification is not suitable for existing TTA methods and may even disrupt the original algorithm. Since adding noise directly to pixels would damage the semantic information in the original image, causing the model to fit the noise components and ignore the objects themselves, resulting in biases in DNNs. Therefore, the sample purification method based on natural image Mixup achieves better accuracy and stability. Also, thanks to the data augmentation of MixUp. it can improve the calibration of DNNs and reduce over-confident predictions. We also compare with DDA (Gao et al., 2023). a diffusion-based image purification method. Due to time constraints, we only conduct experiments on Gaussian noise and Defocus blur corruptions, each requiring 4 x 4090 GPUs and approximately 45 hours. The results in Table 21 demonstrate that diffusion-based DDA is not consistently effective. More importantly, DDA incurs significantly higher runtime and computational overhead compared to existing TTA algorithms, limiting its applicability.
Table 21: The comparison of PTTA with purification via FGSM, SOAP, and Diffusion-based DDA.
RESNEI50 DEYO
VIT-B/16 DEYO
METHONS
ETA
CPL
ETA
CPL
GAUSS
DEFES
GALSS
DEPCS
GAUSS
DEFCS
GAUSS
DEFCS
GAUSS
DEFCS
GAUSS
DEPES
BASE ALGORITHM
36.0
33.4
36.5
33.9
29.9
27.8
53.3
57.6
54.4
58.2
42.4
48.6
+ PURIFY W/ FGSM
36.1
31.5
36.6
32.0
30.2
28.1
53.3
52.8
54.3
54.0
42.7
46.8
PURIFY SOAP
35.5
30.0
36.4
31.6
26.9
9.16
53.8
54.7
55.0
45.1
35.8
PURIFY W/ DDA
44.3
14.9
44.3
14.9
44.3
14.9
60.0
30.2
60.0
30.2
60.0
30.2
+ PTTA
37.4
35.4
37.8
35.1
32.1
29.8
54.0
58.2
55.0
58.8
42.8
50.0
Comparison of different purification scopes. In Table 18 and 19, we compare purifying only malicious test samples, purifying only benign samples, and purifying all test samples. The results of comparative experiments show that purifying all test samples achieves higher accuracy. Through learning from more augmented samples using Mixup. DNNs can achieve better calibration. This helps reduce the over-confident predictions for out-of-distribution (OOD) samples and mitigates the risk of disrupting the model's optimization. From another perspective, in TTA tasks, OOD samples are malicious relative to in diddriibuuiion samples, due to the poor generalization of DNNs. Therefore, using samples that are benign to DNNs to purify all OOD samples can effectively alleviate the problem of degradation in the model's 's prediction performance.
28

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
Single TTA
Continual TTA
Single TTA
Continual TTA
45
Accuracy(9%
Badlie
Mis Neke
Accuracy
40
Accuracy
44-
Accuracy(%5)
44-
30-
bodio
43-
44
20
42
43
10
#2
Logit
Feature
Pixel
Logit
Feature
Pixel
Logit
Feature
Pixel
Logit
Feature
Pixel
(a) Tent
(b) CPL
50
Accuracy'' (%))
Accuracy
49
Accuracy
Accuracy
20
45
48-
Mox thile
15
47
Biraee
40
MichHecer
49
46
10
Logit
Feature
Pixel
Logit
Feature
Pixel
Logit
Feature
Pixel
Logit
Feature
Pixel
ETA
(d) DeYO
Figure 12: Comparison of maximum saliency distance, maximum direct cosine distance, and maximum direct cosine
similarity at the logit, feature, and pixel levels
E.10. Saliency Distance vs. Direct Cosine Distance
In Sec. 3.2, we propose using saliency distance the metric for retrieving benign samples. Matching malicious sample with the sample having the maximum saliency distance has maximum opposite impact on the objective function. Another metric for retrieving benign samples is to directly calculate the cosine values between the logits or features output by the model, or image pixels for two samples 2; and z.. i.e., Cosine (xx)), fa (zz)). Then, either the maximum cosine distance or the maximum cosine similarity is taken as the metric for retrieving benign samples. In Fig. 12, we compare the schemes of using maximum saliency distance, maximum cosine distance, and maximum cosine similarity as the metrics for retrieving benign samples. The results show that the benign samples retrieved by the maximum cosine distance cannot fully match the malicious samples, and thus cannot improve the performance of the original TTA algorithms. Furthermore, the retrieval based on the maximum cosine similarity will obtain samples that share similarities with malicious samples in terms of the objective function, and thus cannot achieve the purpose of purifying malicious samples. Therefore, the retrieval scheme based on our saliency distance has universality for logits, features, and pixels, as well as generality for different DNNs.
E.11. Process of Continual Test- Time Adaptation
Fig. 13 visualizes the intermediate processes of two Test-Time Adaptation (TTA) algorithms, Tent (top) and DeYO (bottom), along with their PTTA-applied versions. Distribution shifts caused by abrupt corruption transitions induce oscillations in Deep Neural Network (DNN) optimization, undermining the effectiveness of TTA algorithms. Visualizations empirically validate that PTTA enhances DNN robustness against dynamically changing distributions and improves generalization on out-of-distribution samples by introducing Mixup.
F. Limitations and Future Work
The saliency indicator we proposed is beneficial for encoding benign and malicious samples. And the saliency distance built upon it is an excellent metric for retrieving benign samples that have an opposite contribution to the objective function compared to malicious samples for the purpose of purification. However, the saliency indicator is not good enough. Developing an indicator with robust representation against arbitrary distribution shifts will be our future work.
We primarily validate the effectiveness of PTTA vision models in the context of Test-Time Model Adaptation. Since the malicious sample hazards persist in a wide variety of machine learning and deep learning tasks, it is interesting future research direction to extend purification strategies and PTTA to various tasks and applications.
29

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
60
Ne Acggiin
80
No Adptaake
Tox PITA
60
Accuracy
60-
Accuracy
40
26
Gana.
Impul
Delic
Gless
Musion
Zow
Soott pros
yet pei Contt Elaute puad
Gsazz
Shot
losal.
Deooe
Glees
Medion
Zeam
Serne
Fostt
Cost.
WE0
Adaptation Order (+)
Adaptation Order (+)
(a) Tent with ResNe150
(b) Tent with ViT-B/16
80
Alapaaic
Accuracy
Accuracy(P5)
60
20
Guae
Shot Inpul
Delos
dbes
Modion, Lout
Snowt prent pol thit Comn Elostle piad IPEC Adaptation Order (+)
Giass
Shod
Iosaal
Dofoe
Metion
Teamn
Sooe For not Come
Elauss Picd D50
Adaptation Order (+)
(c) ETA with ResNet50
(d) ETA with ViT-B/16
Algglieee TATA
Accuracy (%)
ECC 2TTA
Accuracy(%o)
60
FATA PTTA
80
20
Guand
Shot iapul
Deloc
Glas
Modion
Snott peest pos Bot A Adaptation Order ( (+)
Zoom
Cour
Elmiie Piad
IPE
Shed Goast
lospll
Dofoe
Glass
Modion
Zeom
soct Forst For gnit Comt
Elants Fad DEG
Adaptation Order (-)
(e) EATA with ResNer50
(f) EATA with ViT-B/16
80
(a)
Accuracy (%)
Accuracy(%)
60
80
20
20
Ganee
Shat Impul.
Detoc
Gld
Mudion
Sool pron
yet geit Cont..
filettie pind w80
Grass
Shot Impel
Donee
Glass
Modion
Zeam
Spck
past
yor ant Come
Eluatle pord 105
Adaptation Order (+)
Adaptation Order (+)
(g) DeYO ResNet50
(h) DeYO with VIT-B/16
Adgtaac
Accuracy (%)
Accuracy(9
60
Guane
Shol Inspil
Detce Gue
Mution Loom Spait Frost you uri Come Elartic Food
Ganss
Stad Impel.
Devk
Medion
Teom
Snow posst you yau. Cunte Elants Pind
Adaptation Order (+)
Adaptation Order ( (+)
(i) CPL with ResNet50
(j) CPL with ViT-B/16
Figure Detailed experimental results (top-1 classification accuracy (%)) on continual TTA task.
30

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
ResNet50
ViT-B/16
3.00
(a) Logit Segit-Saleneyy (b) Feature- Feature-Saliency (c) Pixel-Salien Fixel-Salencyy
(d) Logit-Saliency (e) Feature Saature-Salieeyy (f) Pixel Fxel-Saliengy
Gaussian noise (Gauss.)
(g) Logit-Saliency (h) Fcature-Salicncy (i) Pixel-Saliency
( Logit-Saliency (k) Feature-Saliency (l) Pixel Fixel-Saliency
Defocus blur (Defoc.)
1026
320
(m) Logit-Salienge (n) Feature Saauune- (o) Pixel-Salee Salle-Sl
(p) Logit-Saliency (q) Feature Salture-Sllieeyy (I) Pixel-Saliena
Snow (Snow)
(s) Logit-Saliency (t) Seature-Saliency
(u) Pixel-Saliency
(v) Logit-Saliency (w) Feature-Saliency (x) Pixel-Saliency
Contrast (Contr.)
Figure 10: Saliency Distance (normalized to the range of 0 ~ 1) among test samples sorted in ascending order of predicted entropy and split according to percentages. We visualize the logit-, feature-, and pixel-saliency distances of ResNet50 and ViT-B/16 for types of image corruptions: Gauss., Defoc., Snow, and Contr..
31

---

PTTA: Purifying Malicious Samples for Test-Time Model Adaptation
80
Top-1 Accuracy (%)
NoAdape
60
mm yommynn
40
20
10-
Gauss Shot Impul Defts
Gils
Min
Zm
Snw
Fist
Fg
Brt
Cit
Els
Px
Jpg
Adaptation Order (-)
Top-1 Top-1 Accuracy (%)
60
50
40-
NoAdapt
20
DeYO
10
+PTTA
Gauss
Shot
Impul
Defes
Gls
Mm
Zm
Snw
Frst
Brt
Cut
Els
Px
Jpg
Adaptation Order (-)
Figure 13: Visualization of intermediate Continual Test-Time Adaptation processes of Tent (upper) and DeYO (lower), along with their PPTA-applied versions. The figure intuitively demonstrates the impact of abrupt distribution shifts on DNN optimization trajectories.
32


Reviews:
[{"review_id": "Ti9TghtGCC", "reviewer": "Reviewer_G5c6", "summary": "This paper focuses on leveraging malicious samples during Test-Time Adaptation (TTA) to improve data utilization. The authors propose PTTA, a plug-and-play method that retrieves benign samples with maximal divergence from malicious samples and employs a Mixup strategy to purify malicious samples for TTA. PTTA demonstrates compatibility with existing TTA methods, and extensive experiments validate its effectiveness across multiple datasets, backbones, and tasks.\n\n\n## update after rebuttal\nMy concerns have been addressed. I think this paper can be accepted now. This paper provides a valuable exploration of malicious sample utilization in TTA, with extensive experiments validating the effectiveness of the proposed method. In my opinion, this work is well-executed and deserves acceptance. To further strengthen the contribution, I would suggest extending the discussion that could provide deeper insights for readers. \n\nFirst, it would be further improved by including a brief discussion with more TTA approaches. The paper could consider some of the recent TTA methods, such as FOA [1], ROID [2], MGTTA [3] and EATA-C[4]. Unlike this work, these methods do not perform purification on malicious samples but instead design general optimization strategies using all samples, achieving significant performance improvements as well. Second, some studies explore enhancing TTA performance from the perspective of improving TTA process. For example, improving entropy loss [5-6], refining batch normalization[7], and optimizing the inference process [9]. Third, the paper primarily discusses the application of TTA in image classification and semantic segmentation. It would be worthwhile to explore whether the proposed method can be extended to other domains and tasks, such as image super-resolution [9], video classification [10-11], and visual question answering [12].\n\n[1] Test-Time Model Adaptation with Only Forward Passes, ICML 2024.\n[2] Universal test-time adaptation through weight ensembling, diversity weighting, and prior correction, WACV 2024.\n[3] Learning to Generate Gradients for Test-Time Adaptation via Test-Time Training Layers, AAAI 2025.\n[5] TEA: Test-time Energy Adaptation, CVPR 2024.\n[6] Decoupled Prototype Learning for Reliable Test-Time Adaptation, arXiv 2025.\n[7] Unraveling batch normalization for realistic test-time adaptation, AAAI2024.\n[8] Boost test-time performance with closed-loop inference, arXiv 2022.\n[9] Efficient test-time adaptation for super-resolution with second-order degradation and reconstruction, NuerIPS 2023.\n[10] Video Test-Time Adaptation for Action Recognition, CVPR 2023.\n[11] Exploring Motion Cues for Video Test-Time Adaptation, ACM MM 2023.\n[12] Test-time model adaptation for visual question answering with debiased self-supervisions, TMM 2023.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort put into reviewing our paper and providing valuable feedback. We would like to address your questions below and provide a link to figures and tables.\n\n[link] https://anonymous.4open.science/r/PTTA/tab_fig.pdf\n\n\n\n>  **R1Q1**: Lacks detailed computational complexity analysis.\n\nWe provide comparative analyses of **running times** between different TTA methods and their PTTA-applied versions in Table A of [link], along with **quantified storage overhead** for PTTA's memory bank in Table B of [link]. Additionally, Table E of [link] compares state-of-the-art TTA methods with their PTTA-applied versions **under comparable computational constraints**. The results support PTTA's superiority.\n\n\n\n> **R1Q2**: Motivation of the proposed method is incomplete.\n\nAdversarial purification methods typically leverage the first-order partial derivatives of the objective loss function with respect to the image $x$, i.e., **the saliency information**: $\\gamma = \\xi \\cdot \\text{sign}(\\nabla_x \\mathcal{L}(f_\\theta(x), y(x))$, as an unit vector in the image space for sample purification. \n\nSaliency information quantifies which individual pixels require the most modification to minimize the objective loss function. Consequently, **samples contributing similarly to the objective loss function exhibit aligned directions of their saliency information unit vectors**, resulting in small Cosine distances that serve as the *saliency indicator*. \n\nWhile Entropy Minimization (EM) is a widely adopted objective loss function in TTA methods, we use EM-derived saliency indicator for noise encoding. Notably, we validate in Table C of [link] that the saliency indicator based on the teacher-student consistency loss function (used in CoTTA and CTDA [1]) also achieves strong efficacy, demonstrating the **flexibility in choosing objective loss functions**.\n\n[1] Wang, et al. Continual test-time domain adaptation via dynamic sample selection. In WACV, 2024.\n\n\n\n> **R1Q3**: On a brief theoretical or empirical justification for using Mixup.\n\nWe provide a brief theoretical and empirical justification for using Mixup in PTTA. Please refer to **R2Q2** for details.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes", "theoretical_claims": "Yes. I have checked the proof of Logit-saliency indicator.", "experimental_designs_or_analyses": "Yes. The experimental design rigorously evaluates common TTA baselines and tasks, with comprehensive validation on diverse benchmarks.", "relation_to_broader_scientific_literature": "This paper is closely related to the field of test-time adaptation, sample selection, and the use of malicious samples.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "Strengths\n\n1.\tThe paper is well-structured and clearly written.\n\n2.\tPTTA is a simple yet practical plug-and-play approach, seamlessly integrating with existing TTA frameworks.\n\n3.\tThorough ablation studies and evaluations across diverse scenarios (datasets, backbones, tasks) strongly support the method’s efficacy.    \n\nWeaknesses\n\n1.\tLacks detailed computational complexity analysis. While PTTA improves performance, Table 1 indicates increased time (additional forward passes) and memory costs (memory bank). A critical analysis is missing: How do SOTA methods perform under comparable computational constraints? (For example, maintain a memory bank and randomly select benign samples for adaptation when encountering malicious samples.)\n\n2.\tMotivation of the proposed method is incomplete. The connection between adversarial purification and the proposed saliency indicator (Sec. 3.2) requires stronger justification. The rationale for linking gradient directions of entropy minimization to noise encoding remains unclear.", "other_comments_or_suggestions": "1.\tMixup Rationale: A brief theoretical or empirical justification for using Mixup in Sec. 3.3  would strengthen the methodology.\n2.\tThe main text seems to lack a reference and introduction to Figure 1.", "questions_for_authors": "See the comments above", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "komDV8uNed", "reviewer": "Reviewer_kYsv", "summary": "Existing TTA algorithms often focus on selecting benign samples for self-training, which leads to wasted test data. To address this, the authors propose PTTA, which uses a saliency indicator to identify benign samples with opposing effects on the objective function and combines them with malicious samples via Mixup. This strategy effectively leverages the information in malicious samples, improving online test accuracy. Extensive experiments across four TTA tasks, as well as classification, segmentation, and adversarial defense, validate the method’s effectiveness.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort put into reviewing our paper and providing valuable feedback. We would like to address your questions below and provide a link to figures and tables.\n\n[link] https://anonymous.4open.science/r/PTTA/tab_fig.pdf\n\n\n\n> **R2Q1**: What's the results of purifying only begin samples?\n\nWe present an ablation study in Table F of [link] comparing PTTA and direct removal of malicious (harmful) samples with purification of benign samples via Mixup. The results indicate that purifying only benign samples yields performance improvements of base TTA methods. Furthermore, additionally leveraging information from malicious samples via PTTA further enhances the performance.\n\n\n\n> **R2Q2**: Why is Mixup for PTTA effective?\n\nIn general, Mixup helps enhance model robustness against adversarial attacks and improves generalization on out-of-distribution data [1]. It has also been demonstrated to refine the calibration of DNNs and mitigate overfitting [2]. \n\nNext, we perform a Taylor expansion of the purification loss (Eq. 7) at mixup ratio $\\lambda$ equals 0:\n\n$$\n\\mathcal{L}\\_{pur}(x^-, x^+) = - (\\lambda y^- + (1-\\lambda)y^+)^T \\log f(\\lambda x^- + (1-\\lambda) x^+)\n= -(\\lambda y^- + (1-\\lambda)y^+)^T\\log f(x^+) + \\lambda (x^- - x^+) \\nabla\\_{x^+} \\mathcal{L}\\_{ce}(f(x^+)) + \\mathcal{O}(\\lambda^2),\n$$\n\nwhere $x^-$ is a malicious sample, $x^+$ is a benign sample, $y$ is an output vector, $f$ denotes the model, $\\mathcal{L}\\_{ce}$ is the Cross-Entropy loss. \n\nMinimizing $\\mathcal{L}\\_{pur}(x^-, x^+)$ reduces the loss of $-(\\lambda y^- - (\\lambda-1)y^+)^T\\log f(x^+)$ while mitigating the interference of perturbations on $x^+$ to the predictions of $f(x^+)$, thereby enhancing model robustness. Large differences between $x^-$ and $x^+$ could enhance the influence of the first-order term in $\\mathcal{L}\\_{pur}(x^-, x^+)$, which verify the necessity and effectiveness of our proposed benign sample retrieval. \n\nFurthermore, we provide intermediate continual test-time adaptation process in Figure A of [link], empirically validating the effectiveness of using Mixup for PTTA.\n\n[1] Zhang, et al. How does mixup help with robustness and generalization? In ICLR, 2021.\n\n[2] Thulasidasan, et al. On mixup training: Improved calibration and predictive uncertainty for deep neural networks. In NeurIPS, 2019.\n\n\n\n> **R2Q3**: On the claim of $K$ in the ablation study.\n\nDue to the lack of clear trends in Fig. 9, please refer to Table 14 for detailed results, which demonstrate that as $K$ approaches infinity, it will dilute the proportion of original malicious data information in purified samples. Because we set $λ=1/(K+1)$, as $K$ approaches infinity, $\\lambda$ approaches 0, causing $\\mathcal{L}\\_{pur}(x^-, x^+)$ to degenerate into $\\mathcal{L}\\_{ce}(f(x^+))$, thereby reducing PTTA to base TTA methods that remove all malicious data information.\n\n\n\n> **R2Q4**: On comparison with works [3] and [4].\n\nWork [3] mixes test samples with randomly selected training samples to perform an auxiliary task, which is designed for auxiliary test-time training and diverges significantly from the TTA framework. \n\nWork [4] employs negative learning to improve the utilization of malicious data. However, purification strategy-based PTTA outperform work [4] in TTA tasks, as validated in Table C of [link].\n\n[3] Zhang, et al. Mixup for Test-Time Training. arXiv:2210.01640.\n\n[4] Wang, et al. Continual test-time domain adaptation via dynamic sample selection. In WACV, 2024.", "claims_and_evidence": "The claims made in the submission seem to be supported by clear and convincing evidence. However, in the ablation study section, the author states that “as K approaches infinity, it will dilute the proportion of original data information in purified samples.” This conclusion, however, cannot be directly drawn from Figure 9. Furthermore, what would be the result if the harmful samples were directly removed and only the corresponding benign samples were used in the Mixup? Rather than the fact that using more samples can improve TTA performance, I am more interested in understanding why the samples generated by Mixup on the original samples are effective.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are appropriate for the problem at hand.", "theoretical_claims": "I did not check the correctness of proofs for theoretical claims.", "experimental_designs_or_analyses": "The experiments in the paper are thorough, covering four types of TTA tasks as well as classification, segmentation, and adversarial defense, with well-executed analysis of the results.", "relation_to_broader_scientific_literature": "Mixup is a commonly used data augmentation technique, and the author applies it to malicious samples in TTA to achieve higher sample utilization.", "essential_references_not_discussed": "Work [1] applies mixup to test-time training to prevent performance degradation in the main task and mitigate the mismatch problem. Work [2] improves sample utilization by performing negative learning through complementary labels. I believe these works share similarities with the ideas presented in the paper, and the authors should discuss it further to better highlight the key contributions of the work.\n\n[1] Mixup for Test-Time Training. arXiv:2210.01640\n\n[2] Continual Test-time Domain Adaptation via Dynamic Sample Selection. WACV 2024", "other_strengths_and_weaknesses": "NA", "other_comments_or_suggestions": "NA", "questions_for_authors": "See above.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tTKQM5ECb9", "reviewer": "Reviewer_iMq6", "summary": "The paper introduces a method called Purifying Malicious Samples for Test-Time Model Adaptation (PTTA), a plug-and-play solution. Instead of filtering out, the authors identify that malicious samples in test data, though reflecting the data distribution, can undermine the stability of TTA algorithms. To address this, PTTA aims to transform malicious test samples into benign ones. It uses a saliency indicator to encode the impacts of benign and malicious samples on TTA, and retrieves benign samples with opposite contributions to the objective function compared to malicious samples. And then, the Mixup technique is employed for sample purification. Extensive experiments under various scenarios demonstrate that PTTA improves the performance of existing TTA methods.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort put into reviewing our paper and providing valuable feedback. We would like to address your questions below and provide a link to figures and tables.\n\n[link] https://anonymous.4open.science/r/PTTA/tab_fig.pdf\n\n\n\n> **R3Q1**: The paper lacks quantification of the memory and computational overhead comparison.\n\nWe provide comparative analyses of **running times** between different TTA methods and their PTTA-applied versions, along with **quantified storage overhead** for PTTA's memory bank. Please refer to **R1Q1** for details.\n\n\n\n> **R3Q2**: It would be better to verify whether PTTA is effective when combined with teacher-student structure methods.\n\nWe conduct experiments applying PTTA to two teacher-student structure methods, i.e., **CoTTA and CTDA [1]**, with results presented in Table C of [link]. These demonstrate that PTTA effectively enhances the performance of teacher-student structure TTA methods.\n\n[1] Wang, et al. Continual test-time domain adaptation via dynamic sample selection. In WACV, 2024.\n\n\n\n> **R3Q3**: On comparison between PTTA and diffusion-based malicious-to-benign sample transformation methods.\n\nWe provide comparative experiments between PTTA and Diffusion-based DDA [2] when applied to base TTA methods (ETA, DeYO, and CPL) in Table D of [link]. The results show that Diffusion-based DDA **significantly degrades the performance of base TTA methods in some cases**, while incurring **prohibitive computational overhead** (requiring $4\\times 4090$ GPUs and approximately 45 hours to complete an experiment for a single corruption type), rendering Diffusion-based DDA impractical for TTA tasks.\n\n[2] Gao, et al. Back to the source: Diffusion-driven adaptation to test-time corruption. In CVPR, 2023.", "claims_and_evidence": "Instead of filtering out, the authors identify that malicious samples in test data, though reflecting the data distribution, can undermine the stability of TTA algorithms. To address this, PTTA aims to transform malicious test samples into benign ones. Extensive experiments under various scenarios demonstrate PTTA improves the performance of existing TTA methods, validating the claim.", "methods_and_evaluation_criteria": "PTTA aims to transform malicious test samples into benign ones. It uses a saliency indicator to encode the impacts of benign and malicious samples on TTA, and retrieves benign samples with opposite contributions to the objective function compared to malicious samples. And then, the Mixup technique is employed for sample purification. Evaluations are also reasonable with experiments show that PTTA improves the performance of existing TTA methods under various scenarios.", "theoretical_claims": "The proof of logit-level saliency indicator is insightful, transforming a gradient-based method to a forward only solution.", "experimental_designs_or_analyses": "Upon inspection of the experimental designs, I find them to be reasonable. The utilization of a diverse range of benchmarks offers comprehensive experimental support, effectively validating the proposed approach's efficacy.", "relation_to_broader_scientific_literature": "Most of previous TTA methods focused on filtering out malicious samples in the test data. The authors, however, pointed out that instead of simply discarding them, these malicious samples can potentially undermine the stability of TTA algorithms. Their approach is to transform these malicious samples into benign ones. Through comparisons with existing techniques such as the FGSM, the advantages of the proposed PTTA method are demonstrated.", "essential_references_not_discussed": "Comparing PTTA to diffusion-based malicious-to-benign sample transformation methods (Gao et al., 2024) would strengthen the evaluation.\n\n[Gao’24] Gao, Jin, et al. \"Back to the source: Diffusion-driven adaptation to test-time corruption.\" CVPR. 2023.", "other_strengths_and_weaknesses": "## Strengths:\n1. The paper addresses a significant issue in TTA by focusing on malicious samples. PTTA's approach of transforming rather than discarding them, along with its non-sensitive threshold nature, makes it suitable for TTA scenarios, demonstrating innovative thinking.\n2. The detailed experiments comprehensively validate PTTA across various settings like continual, lifelong, and adversarial, also with VLM backbones. Its excellent performance on multiple benchmarks and plug-and-play feature strongly prove its effectiveness, which is commendable. \n3. The paper's explanation of the logit-level saliency indicator is profound. Converting a gradient-based method into a forward-only solution showcases technical sophistication. \n4. The paper is written in a smooth and easily understandable manner.  \n\n## Weaknesses:\n1. The paper lacks quantification of the memory and computational overhead comparison between the original methods and when PTTA is incorporated. For a plug-and-play method, especially in the TTA field, such quantification is important. It would help better understand its practicality and broad applicability in different resource-constrained environments.\n2. Although PTTA has been well-validated in several scenarios, it would be better to verify whether it is effective when combined with teacher-student structure methods like COTTA. Also, comparing PTTA to diffusion-based malicious-to-benign sample transformation methods (Gao et al., 2024) would strengthen the evaluation.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "N/A", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "2Zw5qWgDrU", "reviewer": "Reviewer_yigZ", "summary": "The paper presents PTTA, a plug-and-play method for purifying malicious (unhelpful) samples for test-time adaptation.\nPTTA selects benign samples by comparing the samplewise gradients.\nInstead of simply filtering out malicious samples, PTTA transforms them into benign samples via Mixup with benign samples.\nPTTA results in high accuracy improvements over various scenarios.\n\n## update after rebuttal\n\nThe rebuttal addressed some of my concerns, including insufficient experiment results and computational complexity, so I raised the score from 1 to 2. However, I still worry (1) the problem (malicious sample hazard) is straightforward and (2) the solution lacks novelty.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the time and effort put into reviewing our paper and providing valuable feedback. We would like to address your questions below and provide a link to figures and tables.\n\n[link] https://anonymous.4open.science/r/PTTA/tab_fig.pdf\n\n\n\n> **R4Q1**: On the claim on entropy-accuracy relationship.\n\nWe clarify that the entropy-accuracy relationship discussed in Sec. 3.1 is employed to demonstrate the **malicious sample hazards** in TTA methods. Our primary objective is to emphasize that directly utilizing malicious samples for TTA would undermine the stability of DNNs. Additionally, in contrast to EATA, we further demonstrate empirically that test data distributions with higher average entropy exhibit lower overall accuracy (Fig. 3 (down)), which is validated across both ResNet and ViT.\n\n\n\n> **R4Q2**: The key claims about PTTA are generally supported by the experimental evidence.\n\nWe also provide a theoretical justification for PTTA. Please refer to **R2Q2** for details.\n\n\n\n> **R4Q3**: It is unclear which version of PTTA is used in the main experiments.\n\nWe clarify that the default version of PTTA employed in the main experiments (**logit-saliency indicator & OOD retrieval**) is detailed in the **implementation part of Sec. 4.1** (lines 304–329). We hope it can address your concerns.\n\n\n\n> **R4Q4**: On the evaluation of computational overhead.\n\nWe provide comparative analyses of **running times** between different TTA methods and their PTTA-applied versions, along with **quantified storage overhead** for PTTA's memory bank. Please refer to **R1Q1** for details.\n\n\n\n> **R4Q5**: Issues on experimental designs and analyses.\n\nWe conduct experiments applying PTTA to CoTTA and SAR in Table C of [link], the results support our claims. We clarify that ETA, DeYO, and CPL were **selected as representative sample-selection-based TTA methods**. Due to EATA's foundational version ETA is **entirely based on sample selection**, so ETA is more suitable to be a baseline. Also, EATA use a Fisher regularizer measured in source domain, which is not source-free.\n\n\n\n> **R4Q6**: On some missing experimental results.\n\nWe provide results for Tent on lifelong TTA task in Table C of [link], and MedBN+PTTA for adversarial defense on ImageNet-C in Table G of [link]. MedBN relies on a large batch size, which underperforms BN Adapt for adversarial defense. Therefore, BN Adapt+PTTA achieves the best performance on ImageNet-C for adversarial defense.\n\nSoTTA [1] was not selected as a baseline since its sample selection criterion aligns with CPL (Zhang et al., 2024), as both aim to maintain uniform class sampling. Additionally, CPL is a new state-of-the-art method in this field. However, we also provide a comparison with SoTTA [1] in Table C of [link]. We hope these studies can address your concerns.\n\n[1] Gong, et al. Sotta: Robust test-time adaptation on noisy data streams. In NeurIPS, 2023.\n\n\n\n> **R4Q7**: On the novelty of this paper.\n\nWe emphasize that the main contributions of this paper include: 1) analysis of **malicious sample hazards** in TTA tasks, 2) a **saliency indicator** to effectively encode benign and malicious data, and 3) a **plug-and-play PTTA framework** for malicious sample purification. \n\nWhile PTTA employs Mixup technique, we clarify that **vanilla Mixup alone proves ineffective for TTA methods**. PTTA's effectiveness fundamentally stems from our proposed purification strategy and framework. \n\nNotably, PTTA uses OOD retrieval by default **without requiring any source data**.", "claims_and_evidence": "1. The claim on the entropy-accuracy relationship is straightforward, but the literature (e.g., EATA) has partially discussed it.\n1. The paper's key claims about PTTA's effectiveness are generally supported by the experimental evidence.", "methods_and_evaluation_criteria": "1. Please consider revising the manuscript regarding the methods. The current manuscript lists various approaches for saliency indicators, benign sample retrieval, and purification methods. It is unclear which version PTTA is using in the main experiments. Please improve the claims to support the best-working approach.\n\n1. The evaluation setting of datasets aligns with standard benchmarks in the TTA field.\n\n1. The experiment does not include the evaluation of computational overhead (e.g., memory, latency) with and without PTTA.", "theoretical_claims": "There are no novel theoretical claims that need to be checked.", "experimental_designs_or_analyses": "The paper experimented with full TTA (single) and continuous TTA (continuous) settings with various datasets, including ImageNet-C. Baselines include TENT, EATA, DeYO, and CPL.\n\nIssues:\n\n1. It is not specified why PTTA is not applied to CoTTA and SAR in Tables 2 and 3.\n\n1. TENT is missing in Table 2.\n\n1. It is unclear why ETA is used as the baseline in Table 1 instead of EATA.", "relation_to_broader_scientific_literature": "PTTA can be applied in broader scenarios where test sample quality is low or includes noisy samples.", "essential_references_not_discussed": "1. The paper does not discuss SoTTA [a] as a baseline, which robustly adapts to noisy data streams.\n\n\n[a] Gong, Taesik, et al. \"SoTTA: Robust Test-Time Adaptation on Noisy Data Streams.\" NeurIPS 2023.", "other_strengths_and_weaknesses": "Minor Weakness\n1. The technical novelty of sample purification is just a simple mixup technique, limiting the novelty.\n1. Using source training data for in-distribution retrieval might limit the applicability.", "other_comments_or_suggestions": "1. Please thoroughly check the citations. A few citations lack published years (e.g., Lee et al., Chen et al.).", "questions_for_authors": "1. As stated in the abstract, sample purification is the main novelty of the paper, but purification is just a simple Mixup, limiting its novelty.\n\n1. It is unclear which method will be the final method for PTTA. The current manuscript lists the potential methods and shows the ablation study. The writing should be improved to better convey the current method's importance.\n\n1. The experiment is inconsistent across the evaluations. Please answer the concerns in the experimental setting. \n\n1. Please report the computational overhead compared to the baselines (with and without PTTA).", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces PTTA, a plug-and-play method for test-time adaptation (TTA) that purifies malicious test samples via saliency-guided Mixup, avoiding their discard and improving data utilization. Reviewers acknowledge its practical novelty and empirical effectiveness: PTTA consistently enhances baseline TTA methods (e.g., ETA, DeYO) across classification, segmentation, and adversarial defense tasks. The logit-saliency indicator and Mixup strategy are praised for simplicity and compatibility with existing frameworks.

However, concerns include: computational overhead (≈40% increase in runtime due to memory bank and forward passes), limited theoretical depth (e.g., reliance on Mixup’s heuristic benefits), missing references with discussion and missed comparisons with diffusion-based methods like DDA. Despite these caveats, PTTA’s robust experimental validation across diverse scenarios (4 TTA tasks, multiple backbones) and clear practical value justify its contribution. The authors are recommended to revise the paper carefully according to the reviewers' comments.
Decision: Accept (poster)