On the Adversarial Vulnerability of Label-Free Test-Time Adaptation
Authors: Shahriar Rifat
Jonathan Ashdown
Michael J. De Lucia
Ananthram Swami
Francesco Restuccia
Venue: ICLR 2025 Poster (2025)
Keywords: Test Time Adaptation
Adversarial Attack
Primary area: alignment, fairness, safety, privacy, and societal considerations
TL;DR: We present an attack algorithm to assess adversarial vulnerabilties of test time adaptation (TTA). Our attack does not assume access to labelled samples unlike exisiting methods.

Abstract: Despite the success of Test-time adaptation (TTA), recent work has shown that adding relatively small adversarial perturbations to a limited number of samples leads to significant performance degradation. Therefore, it is crucial to rigorously evaluate existing TTA algorithms against relevant threats and implement appropriate security countermeasures. Importantly, existing threat models assume test-time samples will be labeled, which is impractical in real-world scenarios. To address this gap, we propose a new attack algorithm that does not rely on
access to labeled test samples, thus providing a concrete way to assess the security vulnerabilities of TTA algorithms. Our attack design is grounded in theoretical foundations and can generate strong attacks against different state of the art TTA methods. In addition, we show that existing defense mechanisms are almost ineffective, which emphasizes the need for further research on TTA security. Through extensive experiments on CIFAR10-C, CIFAR100-C, and ImageNet-C, we demonstrate that our proposed approach closely matches the performance of state-of-the-art attack benchmarks, even without access to labeled samples. In certain cases, our approach generates stronger attacks, e.g., more than 4% higher error rate on CIFAR10-C.

Published as a conference paper at ICLR 2025
ON THE ADVERSARIAL VULNERABILITY OF LABEL-FREE TEST-TIME ADAPTATION
Shahriar Rifat', Jonathan Ashdown", Michael De Lucia', Ananthram Swami and Francesco Restuccia
Northeastern University, United States DEVCOM Army Research Laboratory, United States Air Force Research Laboratory, United States
ABSTRACT
Despite the success of Test-Time Adaptation (TTA), recent work has shown that adding relatively small adversarial perturbations to a limited number of samples in test data leads to significant performance degradation. Therefore, it is cru- cial to rigorously evaluate existing TTA algorithms against relevant threats and implement appropriate security countermeasures. Importantly, existing threat models assume test-time samples will be labeled, which is impractical in real- world scenarios. To address this gap, we propose new attack algorithm that does not rely on access to labeled test samples, thus providing concrete way assess the security vulnerabilities of TTA algorithms. Our attack design grounded in theoretical foundations and can generate strong attacks against dif- ferent state of the art TTA methods. addition, we show that existing defense mechanisms are almost ineffective, highlighting the need for further research on TTA security. Through extensive experiments on CIFARIO-C, CIFAR I00-C, and ImageNet-C, we demonstrate that our proposed approach closely matches the per- formance of state-of-the-as attack benchmarks, even without access to labeled samples. In certain cases, approach generates stronger attacks, e.g., more than 4% higher error rate on CIFAR 10-C. Source code for the experiments is available at ht https: //gi thub. com Rest a -Groorrooaaa srroppttaaaa addav, adv.git.
INTRODUCTION
Although Deep Neural Networks (DNNs) are expected generalize beyond the training data dis- tribution, it is nearly impossible to account for all the variations of the input distribution during training. As such, Test-Time Adaptation (TTA) is proposed to adapt a trained DNN using supervi- sion from information extracted from unlabeled test data (Wang al., 2020; Schneider et al., 2020; Goyal et al., 2022). Although designing loss functions from unlabeled data is challenging, TTA has been shown to outperform conventional DNN inference in various computer vision (CV) tasks, particularly in classification (Wang et al., 2020), object detection (Ruan & Tang, 2024), semantic segmentation (Wang et al., 2023), and multiple object tracking (Segu et al., 2023).
Despite the effectiveness of TTA, its robustness to adversarial samples is still unclear. In this threat model, imperceptible perturbations are tactically generated by adversaries degrade the perfor- mance of TTA. Notice that this threat is fundamentally different from traditional evasion attacks (Goodfellow et al., 2014; Madry et al., 2018) or poisoning attacks Shafahi al. (2018); Carlini & Terzis (2021), as adversaries do not need to modify specific samples or access the DNN training routine to induce erroneous predictions. Therefore, ial threats to TTA. In particular, it is desirable to achieve appropriate utility-security trade-offs and implement defensive measures before deploying TTA algorithms real-world high-stakes contexts.
imperative to rigorously study adversar-
The first work to investigate TTA security vulnerabilities was DIA (Wu et al., 2023), where it was shown that due to the transductive nature of learning in TTA, crafting an attack on some samples can potentially cause drastic performance degradation in non-malicious samples. The key issue is that DIA assumes access to the true labels of all test samples, thus crafting an adversarial perturbation that overestimates the difficulty of TTA attacks. Even if malicious insider had access the TTA system, as assumed (Wu et al., 2023), only the prediction of the adapted DNN would be available to the adversary, and not the true labels. Under such assumption, the attack proposed in (Wu

---

Published as a conference paper at ICLR 2025
(Towards Adversary)
(From Adversary)
Impractical X
O
Unattacked
DIAtwith labels)
Loss
DIA(wlo labels)
Function
True Labels (Y)
30-
Predicted Labels
Data Stream (X)
Adversary
Error (%)
Attack Generation
20
Adversary System
Attacked DNN
Test Time
Adapter
Malicious
Hl mx?
Victim's System
H.700
Insider
THBN TENT EATA SAR SoTTA
(a) Adversarial attack scenario on TTA
(b) Error (%) of two variants of the DIA attack different TTA algorithms for CIFAR-10C
Figure 1: la shows that even with malicious insider involved, adversary trying attack TTA system would not have access to true labels. Fig. lb shows the efficacy of adversarial attacks on TTA for different cases of Distribution Invading Attack (DIA)(Wu et al., 2023) for CIFARIO-C. The attack efficacy also vanishes if access to true labels is not assumed.
al., 2023) becomes weak, which gives false sense of robustness of existing TTA algorithms. In stark contrast, in this work we show that even in the absence of true labels, an adversary can craft strong malicious samples that can degrade the performance of all the existing TTA algorithms. For the first time, our work provides practical testbench to evaluate the adversarial robustness of TTA algorithms against malicious samples.
Our main contributions can be summarized as follows.
We evaluate an impractical assumption in the current design of adversarial attacks on TTA specifically, the access to test data labels that renders it inapplicable in practical settings. Our study reveals that existing TTA methods demonstrate considerable robustness when this assumption is relaxed. This finding potentially opens new avenues for testing the security vulnerabilities of TTA practical scenarios.
We derive formulation of the robust risk for TTA, which is more nuanced and helps us highlight the inherent security threats in existing TTA algorithms. From our formulation we design novel attack algorithm termed Feature Collapse Attack (FCA) that can generate strong attacks even in absence of labels. Through extensive numerical experiments on three benchmark datasets and five TTA methods, we demonstrate that in practical label-free settings, existing TTA algorithms almost equally or even more vulnerable in some cases against our proposed attack compared to an attack that assumes labelled data access.
We show that existing defense mechanisms for robust TTA deployment are either ineffective or not directly applicable against our settings. This makes our proposed attack strong threat requiring countermeasures to ensure robust and reliable deployment of TTA in real-world applications.
RELATED WORK
2.1 ROBUSTNESS OF TEST-TIME ADAPTATION
Significant effort has been made to improve the robustness of TTA in a variety of challenging sce- narios, i.c., continual distribution shift (Wang al.. 2022), continuously changing smooth transition between domains (Press et al., 2024). when the distribution of labels changes in online test batches (Gong et al., 2022; Zhou al., 2023), when these shifts happen concurrently (Yuan al., 2023) and when some outlier samples are mixed in the data stream (Gong et al., 2024). However, the threat of the model adapting to potential adversarial samples mixed in the test data stream - as well as related defense mechanisms predictive performance of certain samples under TTA can degrade without direct adversarial manip- ulation, simply by perturbing other parts of the online data batch in both white-box (Wu et al., 2023) and black-box settings (Cong et al., 2024). However, (Wu et al., 2023) assumes access to labeled samples to craft attacks, which is impractical in the TTA framework. On the other hand, the attack
still underexplored. Two recent studies have shown that the
2

---

Published as a conference paper at ICLR 2025
proposed by (Cong al., 2024) relies mixing portion of adversarial samples into consecutive test batches to achieve adversarial action. If the adversary can manipulate test batches intermittently or the model is reset frequently to prevent performance collapse, such threats become non-existent, potentially failing to reveal security concerns regarding TTA. As these threats gain attention, several defense mechanisms have been proposed to mitigate their impact, including entropy-based sample filtering combined with sharpness-aware minimization (Gong et al., 2024), median normalization (Park al., 2024) and adversarial training (Wu al., 2023). However, we show in Section 7.1 that these defense strategies are either ineffective or violate key assumptions of the original TTA frame- work. Ultimately, this highlights the need for more rigorous evaluation of TTA algorithms, and calls for the design and evaluation of robust TTA methods that can withstand adversarial action.
2.2 ADVERSARIAL MACHINE LEARNING
The literature on adversarial machine learning can be broadly divided into poisoning attacks on train- ing data and evasion attacks test data. Evasion attacks (Biggio et al., 2013; Carlini & Wagner, 2017) attempts to have the DNN misclassify inputs that have undergone minor perturbations. In white-box setting, where an adversary can query the model and obtain gradients of backward pass, this direction can be efficiently found through gradient- based optimizer by maximizing the loss with respect to the input while constraining the perturbation. Data poisoning leverages the training process of DNNs to make it erroneously classify a benign sample by injecting some poisoned sam- ples in the training set (Koh & Liang. 2017). There has been growing interest adversarial attacks within low-label regimes and unsupervised settings. For example, (Kim et al., 2020) introduced instance-wisg wise attacks without labels by maximizing contrastive loss for representation encoders, an approach further improved by (Fan et al., 2021) who incorporated high-frequency views. (Cemgil et al., 2020) developed adversarial attacks for variational autoencoders by maximizing the Wasser- stein distance to the representations of clean samples. In the low-label regime, (Yang et al., 2023a) proposed generating adversarial attacks through adaptive weight regularization and knowledge dis- tillation improve the robustness of semi-supervised learning. However, none of these adversarial methods are directly applicable to our setting as they are not designed to deal with unlabeled test data in TTA scenarios.
3
PRELIMINARIES
3.1 TEST-TIME ADAPTATION
Without loss of generality, we cast TTA in the context of multi-class classification problem. Let X' C Rd be the input space and the set of C' classes or output labels be denoted as )' = =1 (1....C) C).
Given a training data set from the source domain D = {x) (ee.v.). v } } we learn fo X + RC
parameterized by the DNN parameter 0 through iterative minimization of some loss function L(((*;;); y*) (e.g., cross-entropy loss), where X = ( C X and Y == [ C. The trained DNN is then deployed to perform inference on test dataset D' = { (
where the labels y := m are unknown. In conventional inference, it is assumed that D= and D' are sampled from the same distribution and the final predictions are calculated through Fe = arg max. [ff( c y. However, in real-world deployments, training and test data distribu- tions may differ, which ultimately results in poor performance of fa on X := [-]] (Hendrycks & Dietterich, 2019). To address this issue, TTA dynamically adapts So to the test data X without supervision from Y thus obtaining the adapted DNN før Unlike Unsupervised Domain Adapta- tion (UDA) where the entirety of the test samples assumed be available, in TTA fø' obtained Nu
with the test data available in batch mode X == { by solving
e' (Xx) = -arg min LTTA (xh:8): 8,C0'
(1)
where o' (Xx) = 0', Jel Upp. Here, 0', indicates all adaptable parameters, OF are the fixed param- eters and f/2 are the normalization statistics {u. of} across different layers and Crra(-) indicates the unsupervised TTA loss. Some existing works (Schneider al., 2020; Gong al., 2022) on TTA provide empirical evidence of performance improvement by only re-estimating the statistics of the Batch Normalization (BN) layers from test data. The absence of supervision is typically covered by
3

---

Published as a conference paper at ICLR 2025
two unsupervised forms of losses, i.e., entropy minimization (Wang al., 2021; Niu et al., 2022; Goyal et al., 2022) and invariance regularization (Wang et al., 2022; Nguyen et al., 2023).
3.2 NEURAL COLLAPSE (NC)
The NC phenomenon was first observed by (Papyan et al., 2020) in DNNs optimized with SGD, where some distinctive characteristies become increasingly apparent, particularly during the termi- nal phase of training. We formalize NC to ease the understanding of our attack algorithm. Let the i-th sample within class be denoted by Ti,e and the last layer feature of k-layer DNN as h = f0 (xi,e) The class mean and global sample mean are denoted by Pe 1 E, h and n
JAG = nC Eie hi.c. respectively, where c = I....,C C' and n the number of data points per class. We leverage the following three definitions directly from (Papyan et al., 2020).
Variability Collapse: For every class c, the within-class variation collapses to zero:
bi Me Vi E [n), , E [C]
(2)
Equinorm: Class feature mean vectors converge to equal distances from the global mean vector:
d(pe: ((He)K)) d(j pc) 0 A c. EC
Simplification to Nearest Neighbour Classifier (NNC): The prediction of the DNN becomes equivalent to that of the NNC formed by non-centered class means:
arg max (w,, ht) + by ss arg min d (e)
d'
3.3 UNDERSTANDING THE SECURITY VULNERABILITIE OF TTA
We start by describing the robust population risk (0), which a commonly used term adversar- ial robustness (Zhang et al., 2019). Specifically, for batch of test samples X with true labels Y and predictive model fo(.). (-), the final predictions are obtained using F-(-) as:
R(0) E (x max
{ (o + YB (xbB,(Xb.e)
(5)
where Bp(2,c) = f x: - ر|| I {c and 1(-) is the indicator function. Following the for- mulation of (Yang et al., 2023b), the robust risk for a batch can be written as the sum of the natural risk Rnat (0) and the boundary risk Rady (0):
R(0) = Rmat (0) + Rody (0)
(6)
which the natural and boundary risk terms are defined as
Riai (0) = E ( 1 Pa (x(X + #
Rody (0) = Exx;, fXE E (x Fo(X + Fo(EE), Fo(X)) -Y
The natural risk represents the inherent failure of the DNN to fit certain data points, thus resulting in erroneous predictions. Conversely, boundary risk refers the risk associated with regions in the manifold that are very close to the original data points but lead to incorrect predictions. In a tradi- tional evasion attack, adding bounded perturbation to samples potentially causes misclassification of those specific samples if it has high boundary risk. However, during the inference with TTA, the DNN is updated with the current data batch - 0', which can increase both the boundary risk Raay and the natural risk Rnat(0 (0') for the unperturbed samples if a small portion of the test samples are maliciously perturbed.
THREAT MODEL
We provide a description of the threat model considered in this paper. In our attack settings, the adversary has the ability manipulate small portion of the test samples within the entire batch
4

---

This decomposition provides a more insightful look into the design of TTA attacks. Firstly, an adver- sary needs to increase the natural risk (first term) which needs access to true labels. In Section 5.1, we provide details on how this can be achieved from the assumptions of NC. Secondly, to maximize the boundary risk (second term), we need to add perturbations so as to increase the probability that the prediction changes after adding perturbation. Moreover, the second term needs to be multiplied with the probability assigned to the true class from the current DNN. These straightforward attack design directives are not readily available from Equation 6 and 7. Published as a conference paper at ICLR 2025
Xj,, termed as compromised samples Xzom- The attack is executed by adding imperceptible noise to those samples. Thus, the attacker targets the victim who gets a source model fe and tries to update it to fo using TTA as in Equation 1. The attacker's objective is to indiscriminately degrade the predictions of the benign portion of the hatch, X Blcom which can be defined as
[[ (XD) 0 ((X com + + U U Broma rBxcom)
max IL Se
(8)
where L [+] denotes loss function that directly relates the prediction enror. Similar to (Wu al., 2023), we consider the worst-case scenario where the attacker has knowledge of the TTA algorithm and has white-box access the latest updated model parameters provided by some malicious insider. However, unlike (Cong et al., 2024), we do not assume access to source data or true labels Y Breom as in (Wu et al., 2023). Notice that this is not design choice but an inherent constraint of the TTA settings. Additionally, we realistically do not assume that the attacker has any access the training routine of the deployed DNN.
5 ATTACK DESIGN
The attacker's objective defined in Equation 8 essentially becomes the maximization of the robust population risk R(00) ) for the benign test samples. In (Wu et al., 2023), this is directly estimated by maximization of the cross entropy loss, which cannot be directly calculated without Y B.comm A viable alternative is to directly use the DNN prediction y Bicom or its soft prediction scores. How- ever, as demonstrated earlier and the experiments, this leads to an ineffective attack. Therefore, we provide an expression for the robust population risk that can be leveraged to successfully attack TTA and assess its robustness when true labels are not available. The following theorem provides an alternative expression for the risk in Equation 6. Details are provided in Appendix Al.
Theorem For a batch of test samples, the robust population risk can be expressed as
R())= = xz) 1 (x(( + YA} +
3rE B E Bx Fo(x + Fo(X') P rrr (YF==F(Xb))IX
5.1 ESTIMATION OF ROBUST RISK
Using our decomposition of the robust risk, we design several loss functions that reliably estimate the loss term IL..) in the attack objective without requiring labels. Below, we detail the loss functions motivated by Equation 9, which provides high-quality estimates of the objective function.
Nearest Centroid Loss: The natural risk (first term) is estimated using loss function that we refer to as the nearest centroid loss Lnee:
Lace
(
(10)
xEE Hycom
where d(-) denotes some distance measure (e.g. cosine distance) and Pe denotes the class centroid vectors of the last layer features calculated from the activation values of the predicted classes Ò in a batch. This is based on the third assumption of NC described in Section 3.2 suggesting that an increased distance from the nearest centroid correlates with a higher likelihood of wrong prediction.
Feature Collapse Loss: As we are depending on the predicted labels for calculating the class cen- troids and the number of samples is small, these estimated class centroids are not guaranteed to be
5

---

Published as a conference paper at ICLR 2025
reliable and might not provide proper guidance for the design of perturbation increase the natural risk term. Hence, we incorporate the following term in our loss function:
Loot
Σ
(fø (x.) ma)
(11)
34 X Blccm
The predicted global mean Ne is calculated by averaging out the activations in a batch. From the Equinorm assumption of NC, this moves the TTA update to direction that compromises the distance of class means from the global sample mean. This acts as a regularizer that helps guide the attack design that is disrupted by the unreliable estimate of class centroids.
The second term in our robust risk formulation estimated leveraging the Variability Collapse assumption of NC. As the last layer activations converge class mean vectors, the likelihood of overlapping regions with different class centroids diminishes. The following loss scatters the features away from forming tight clusters by maximizing the difference in the activation values before and after the TTA update:
Feature Scattering Loss:
Σ d (x) ( (x.)
Lec
(12)
TEX Bl.om
Here, for is the DNN state after update on test data with malicious examples. According our decomposition of robust risk in Equation 9, this loss term is multiplied by the probability of pre- dicted class p (c == Fo(2:)) Some deviation from the original value occurs when the predicted class is incorreet. An attacker would potentially benefit through more accurate estimation of this term. Despite this non-ideal estimation, we can still craft highly-effective attacks against TTA as demonstrated in the experimental evaluation. The final loss function is:
L = Lince Leoo + Lase p = Fo(c:))
(c==)o(x.))
(13)
For the distance measure d(.), we use cosine similarity as avoids explicit normalization. (x;)
5.2 FEATURE COLLAPSE ATTACK (FCA): ALGORITHM DESIGN
The earlier loss functions are inspired from the assumptions of NC. As such, we name our attack algorithm as FCA. The process of adversarial perturbation generation in FCA is summarized in Al- gorithm 1. The algorithm is relatively casy implement uses basic iterative projected gradient descent. At each iteration step, the perturbation vector is added to the compromised samples (line 6). Next, the DNN is updated using the TTA algorithm being considered (line 7). Then, using Equation 10-13, our proposed loss value is calculated, In line 10, gradients of the loss are calculated. Finally, the perturbation is updated using adversarial learning rate and clipped (-e, e) to maintain the Lo norm constraint. The perturbation ó is also constrained such that the pixel values always remain within the [0, 1] range.
EXPERIMENTAL SETUP
Dataset and Architecture We leverage three primary benchmark datasets typically used for TTA performance evaluation, i.e., CIFARIO-C, CIFAR 100-C, and IetgeNet-C. We directly obtain the CIFARIO-C and CIFAR100- test dataset from Robustbench (Croce et al., 2020). For ImageNet-C, use the provided data by (Hendrycks & Dietterich, 2019). These datasets are modified versions of the original test data by 15 different synthetically generated corruption of 5 severity levels. For our experiments, we use ResNet-32 for CIFARI0-C and CIFAR100-C datasets and ResNet-50 for ImageNet-C*. For the two corrupted CIFAR datasets, all 15 corruptions for 5 severity levels are used. For Imagenet-C, results are reported only for corruption of severity level
Baseline TTA Methods. We evaluate TTA methods as victim algorithms whose robustness is tested against FCA. In line with the previous studies (Wu et al., 2023; Park et al., 2024), we select TTA methods that focus on updating batch statistics the affine parameters of BN layers. Test- time Normalization (TeBN) (Nado et al., 2020) updates BN statistics for each test batch, while
Model defination and trained weights are directly obtained from httpa://github..co chenyaofo/ hoaffo/pyoorchc ci far-model. We have used the model definition and weights from torchwissiom(resnet50-222)

---

Published as a conference paper at ICLR 2025
Algorithm 1: FCA Algorithm
1: Input: Test batch X = X com U X
B com ; Adversarial learning rate o; Perturbation
constraint Model paramaters before adaptation 0; Iteration steps for attack
2: Initialize: Adversarial perturbation 8 |mm xcxxxx -> (c, h, w)
represent the channel,
height, and width of an input sample
3: Output: Adversarial perturbation vector 8
4: Make separate copy of 0
5: for step = 1,
n do
6: X (X com +5 U X c/com
7: Update 9 + [Eq. 1]
8: Calculate A (X Blcom) and f8. pk-1 (x Byoom)
9:
Calculate ((X F3ymos [Eq. 13] c(rmm)mmm))
10:
Calculate grad V 6oom |xbreom
Calculate 8 + -lip (16 + a sign (igg(ggad))) (),,e)
12:
Calculate 5 +- clip (1Xc com + deom). (0. 1) X com
end for
14:
Return: X 6
Test-time Entropy Minimization (TENT) (Wang et al., 2021) adjusts the affine parameters in BN layers through entropy minimization. Efficient Anti-forgetting Test-time Adaptation (EATA) (Niu et al., 2022) enhances sample-efficient entropy minimization and incorporates a Fisher regularizer to prevent knowledge loss from the pre-trained model. Sharpness-aware and Reliable Optimization (SAR) (Niu et al., 2023) utilizes BN layers and sharpness-awar minimization to mitigate the adverse effects of large gradients, and Screening-our Test-time Adaptation (SoTTA) (Gong et al., 2024) employs sample filtering and sharpness- aware minimization.
Evaluation Setup To benchmark the performance of FCA, we consider each test batch as a trial. For given test batch, randomly select some data samples compromised ones and add pertur- bations generated by one of the baseline attack methods as well as FCA. We assume the scenario where the DNN is updated by the considered TTA algorithm up to the 1 trial and the adversary attacks the update of trial I, (.(()) with access to model parameters fg We evaluate the effec-
tiveness of FCA by its average error rate increase on the benign (unperturbed) samples compared to normal adaptation without attack across all trials. Unless otherwise specified, we use test batch size of 200 for each trial where 20% samples are selected as compromised ones, adversarial learning rate a 2/255, perturbation constraint 8/255 and iteration steps for attack to be 100.
7 EXPERIMENTAL RESULTS
We compare our proposed FCA against five state-of-the-art TTA methods across three benchmark datasets. For attack baselines, include three benchmarks: the Distributionally Informed Attack (DIA) (Wu et al., 2023), its pseudo-label variant (IA(P and TePA (Cong et al., 2024), originally proposed as a test-time poisoning attack. In our experiments, we adapt TePA's loss formulation under our threat model and within the context of each TTA method. Pseudo labels can be obtained using the current model's prediction probabilities (soft PL) or the argmax of these predictions (hard PL) for benchmarking, Since both variants result in similarly low attack efficacy, we use hard pseudo labels throughout and refer to them as PL. for brevity. Notice that the original DIA setup assumes access to the true labels of the data, which are unavailable at test time. From Table 1, we observe that in CIFAR-IOC, the proposed FCA consistently increases the error rate by more than 4% for TENT and TeBN and by similar margin in other TTA algorithms. However, without access to the true labels, DIA is almost ineffective. This is because the predicted labels are inherently noisy due to the distribution change in the data. For CIFAR- 100C and ImageNet-C,, our proposed attack mechanism also performs very close to the DIA benchmark. Across these three datasets, we observe that with the increasing number of unique classes, attack is still very effective but weakens slightly compared
7

---

Published as a conference paper at ICLR 2025
Table I: Performance (% error) comparison different TTA methods.
TTA Method
Dataset
Attack Method
TeBN
TENT
EATA
SAR
SoTTA
w/o Attack
19.98
19.72
23.05
19.70
20.27
DIA
34.11
33.62
36.80
33.50
34.39
CIFARIO-C
DIA (PL)
21.04
20.67
24.23
21.03
20.98
TePA
21.11
20.81
24.27
21.01
21.04
FCA
38.87
37.95
40.15
37.33
38.37
w/o Attack
51.44
49.39
52.56
49.39
50.01
DIA
64.21
62.64
63.76
62.57
63.45
CIFAR100-C
DIA (PL)
52.21
52.67
53.69
51.03
52.07
TePA
52.29
52.68
53.81
51.09
52.12
FCA
64.03
62.21
63.97
62.12
63.23
w/o Attack
43.78
42.33
43.38
43.37
42.82
DIA
51.54
50.65
49.54
52.06
51.23
ImageNet-C
DIA (PL)
46.29
45.11
44.88
44.77
44.79
FCA
51.06
50.63
49.06
50.22
50.33
to DIA. We believe this is due to the effect of the NC assumptions getting slightly relaxed as the number of classes increases.
Performance Evaluation grey box settings For the results reported in the main section, we evaluated the effectiveness FCA under a worst-case scenario where the adversary has access to the latest DNN weights during model updates, Although this setting reveals the worst-case scenario, may always practical for real-world deployment. To assess the performance of FCA under more restrictive grey-box settings, we relaxed the initial assumptions. We assume threat model where the adversary only has access to the source DNN weights and architecture used for TTA, but not the latest updated parameters. The evaluation results, presented in Table 2, show that our proposed attack mechanism remains highly effective these settings, with only a 2% reduction in attack effectiveness across different TTA methods.
Table 2: Performance (% error) comparison in grey box settings
TTA Method
Dataset
Attack Method
TeBN
TENT
EATA
SAR
SoTTA
w/o Attack
19.98
19.72
23.05
19.70
20.27
DIA
33.22
31.77
33.53
31.03
32.09
CIFARIO-C
DIA (PL)
20.91
20.63
24.23
21.05
20.87
TePA
21.11
20.81
24.27
21.01
21.04
FCA
36.03
35.98
37.41
34.83
36.29
w/o Attack
51.44
49.39
52.56
49.39
50.01
DIA
61.03
60.21
60.51
60.11
62.34
CIFAR100-C
DIA (PL)
52.13
52.58
53.66
50.81
52.05
TePA
52.29
52.68
53.81
51.09
52.12
FCA
60.22
60.04
60.58
69.91
62.07
w/o Attack
43.78
42.33
43.38
43.37
42,82
DIA
49.36
48.88
47.15
50.11
49.38
IetgeNet-C
DIA (PL)
45.23
44.87
43.15
43.39
42,92
FCA
49.03
48.55
47.13
50.07
49.21
Effect of Sample Size In Table 4, the performance the FCA and DIA attacks is reported for different numbers of maliciously perturbed samples the CIFAR-10C dataset. Interestingly. FCA exhibits a higher error rate across all TTA benchmarks compared to DIA, indicating that it does not have any extra sensitivity number of malicious samples compared the DIA. To evaluate the
8

---

Published as a conference paper at ICLR 2025
efficacy of the attack with different batch sizes, Table 3 reports the performance of FCA three different batch sizes for the CIFAR-1O0C dataset. The CIFAR-1000 dataset was chosen due to its larger number of classes compared to CIFAR-IOC, which could reveal any potential sensitivity of our designed attack as class centroid estimation becomes unreliable with smaller batch sizes and observe that the performance of FCA does not decrease significantly compared to DIA for low sample sizes.
higher number of unique classes. However,
Effeet of Different Loss Components FCA To analyze the efficacy of different FCA loss terms, we evaluate the performance of FCA on CIFAR-IOC in the common attack parameter settings. From Table 5, can be observed that feature scattering loss the most influential in the efficacy of attack. However, each of the loss terms has a notable effect on the effectiveness of our porposed attack.
Table Performance (% error) across different batch sizes on CIFAR-100C
Batch
TTA Method (DIA/FC)
Size
TeBN
TENT
EATA
SAR
SoTTA
64
70.93
68.98
70.86 69.03
71.04 / 69.22
69.03 / 68.55
69.21 67.84
128
64.93 64.24
63.96
62.30
65.01
64.88
63.12/ 62.74
63.44 / 62.29
200
64.21 64.03
62.64 62.21
63.76 63.97
62.57 / 62.21 63.45 / 63.23
Table 4: Performance (% error) with different numbers of malicious samples on CIFAR- I0C
Malicious
TTA Method (DIA/FC)
Samples
TeBN
TENT
EATA
SAR
SoTTA
10 (5%)
23.33 / 25.11
23.02/ 24.81
24.11 / 26.77
23.03 / 24.21
23.02 24.221
20 (10%)
26.33 33.222
25.77 / 29.98
27.64 / 32.24
25,44/
29.67
36.75
30.11
40 (20%)
34.11/38.77
33.62
37.95
36.80/ 40.15
33.50/
37.33
34.39 /3.337
80 (40%)
45.33 /49.81
43.77 48.21
47.03/
43.98 47.75
43.21 /47.33
7.1 LIMITATION OF EXISTING DEFENSE MECHANISM
Sharpness Aware Learning Sharpness-aware learning (Niu al., 2023) aims enhance the stability of model parameters by steering them towards a flat minimum the loss surface. This method grounded in the idea that flat minimum preferable for model robust- ness, particularly when dealing with noisy or large gradients. However, shown Fig. 2a,
Table 5: Performance (% error) on different FCA variants CIFAR-10C
FC Variant
TeBN
TENT
EATA
SAR
SoTTA
w/o attack
19.98
19.72
23.05
19.7
20.27
Lne
21.21
20.93
22.01
21.03
20.87
Loot
23.22
23.01
24.55
23.97
22.66
Dee
34.33
33.11
36.24
35.18
34.13
Loot + Lne
23.24
23.03
24.61
23.95
22.67
Lse + Loot
35.91
35.11
37.89
36.22
34.74
Lee + Lue
37.91
37.03
38.94
36.54
37.26
Final
38.87
37.95
40.15
37.33
38.37
Unurasked PC Atlak
Altack (MollIN)
0.5
Malicious
Density (Log Scale)
Benign
Density
0.4
Benign
10
Malicious
30-
0.3
10a
Brrort
20
0.2
0.1
0.0
8
12
0.0
0.5
1.0
2.0
Gradient Norm
Entropy
TEBN TENT EATA SAR SOTTA
Histogram of Gradient Norm
Histogram of Entropy
(c) MedBN Defense
Figure 2: Limitation of different defense algorithms. The plots generated with CIFAR10-C with the default attack parameters setting.

---

Published as a conference paper at ICLR 2025
adversarial samples generated by our attack mechanism exhibit gradients similar to those of normal samples. Additionally, there is a high concentration of malicious samples in the region where gradient norms small. This suggests that sharpness-aware learning may not be an effective strategy for mitigating the impact of adversarial data. From, Table I, it is also evident that methods that incorporate sharpness aware learning (SoTTA) are not more robust than other TTA scheme that does not involve such a mechanism.
Sample Filtering Scheme As defense mechanism, (Gong et al., 2024) proposed filtering out high entropy samples. However, the histogram in Figure 2b reveals that the entropy values of adversarial samples and normal samples are similarly distributed. In fact, most adversarial samples crafted by our method are concentrated in the low entropy region. Consequently, TTA methods that use simple entropy-based filtering scheme, as suggested by (Niu al., 2022; Gong et al., 2024), do not demonstrate significant robustness in our experiments.
Robust Statistics Estimation (Wu et al., 2023) proposed using robust BN statistics by treating source statistics prior and updating them with test data statistics using momentum term. Since source data statistics remain unaffected by adversarial attacks, it improves robustness. However, overemphasizing source statistics can hinder the extraction of information from test samples, affect- ing TTA performance. Therefore, selecting the appropriate momentum value, which balances this trade-off, is crucial. However, the optimal momentum value varies across different TTA methods, as shown in Table 6, making the defense mechanism highly dependent on this hyper parameter choice. A recent defense mechanism proposed by (Park et al., 2024) demonstrates that Median Batch Normalization (MedBN), being robust to outliers, a viable al- ternative to BN. However, in- corporating median normaliza- tion or its variants only compli- cates the generation of adversar- ial examples. A critical flaw in this defense mechanism is that it only superficially increases the difficulty of generating adversarial samples without providing sub- stantial robustness to the TTA method. An adversary can easily bypass this defense by crafting adversarial perturbations using a traditional BN layer instead of the MedBN layer. Figure 2c illus- trates that even when the victim TTA adapters are equipped with MedBN, it is still largely ineffective preventing performance degradation the event of attack.
Table 6: Performance (% error) for different momentum values for robust statistics estimation on CIFARIO-C
Momentum
TeBN
TENT
EATA
SAR
SoTTA
38.87
37.95
40.15
37.33
38.37
0.4
37.70
35.26
35.33
35.22
34.67
0.6
36.21
34.48
35.36
35.14
33.67
0.8
34.22
33.56
35.33
35.03
33.71
8 CONCLUSIONS
Our study highlights significant vulnerabilities in current TTA methods when faced with adversarial attacks, especially under practical conditions where test sample labels are not accessible. We chal- lenge the prevailing assumptions in existing threat models and demonstrate that many TTA methods are more robust than previously thought when these assumptions are relaxed. However, our newly proposed attack algorithm, which does not rely labeled test samples, reveals that TTA methods still possess inherent security risks. Our extensive experiments on benchmark datasets confirm that approach generate strong attacks, sometimes surpassing ofate-of-the-art benchmarks which assume access to labels. We find that existing defense mechanisms are largely ineffective against these types of attacks, underscoring the need for further research and development in this area.
ACKNOWLEDGMENT OF SUPPORT
This work has been funded in part by the National Science Foundation under grants and CNS-2312875, by the Air Force Office of Scientific Research under contract number FA9550- 23-1-0261, by the Office of Naval Research under award number -00014-23-1-2221.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Smnić, Pavel Laskov, Gior- gio Giacinto, and Fabio Roli, Evasion attacks against machine learning at test time. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD, pp- 387-402. Springer, 2013.
Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. arXiv preprint arXiv:2106. 00666 2021.
Nicholas Carlini and David Wagner. Towards evaluating the robustness of neural networks. In 2017 IEEE Symposium Security and Privacy (SP), pp. 39-57. IEEE, 2017.
Taylan Cemgil, Sumedh Ghaisas, Krishnamurthy Dj Dvijotham, and Pushmeet Kohli. Adversarially robust representations with smooth encoders. In International Conference Learning Represen- tations, 2020.
Tianshuo Cong. Xinlei He, Yun Shen, and Yang Zhang. Test-time poisoning attacks against test- time adaptation models. In 2024 IEEE Symposium on Security and Privacy (SP). pp. 1306-1324. IEEE, 2024.
Francesco Croce, Maksym Andriushchenko, Vikash Schwag, Edoardo Debenedetti, Nicolas Flam- marion, Mung Chiang, Prateek Mittal, and Matthias Hein. Robusthench: standardized adver- sarial robustness benchmark. arXiv preprint arXiv:2010.09670, 2020.
Lijie Fan, Sijia Liu, Pin- Yu Chen, Gaoyuan Zhang, and Chuang Gan. When does contrastive learning preserve adversarial robustness from pretraining to finetuning? Advances in Neural Information Processing Systems, 34:21480-21492, 2021.
Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. Note: Ro- bust continual test-time adaptation against temporal correlation. Advances in Neural Information Processing Systems, 35:27253-27266 2022.
Taesik Gong, Yewon Kim, Tacckyung Lee, Sorn Chottananurak, and Sung-Ju Lec. Sotta: Robust test-time adaptation noisy data streams. Advances in Neural Information Processing Systems, 36, 2024.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Sachin Goyal, Mingjie Sun, Aditi Raghunathan, and J Zico Kolter. Test time adaptation via conju- gate pseudo-labels. Advances in Neural Information Processing Systems, 35:6204 2022.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common cor- ruptions and perturbations. arXiv preprint arXiv: 1903. /226/, 2019.
Dan Hendrycks, Norman Mu, Ekin D Cubuk, Barret Zoph. Justin Gilmer, and Balaji Lakshmi- narayanan. Augmix: A simple data processing method to improve robustness and uncertainty. arXiv preprint /912.0271 2019.
Minseon Kim, Jihoon Tack, and Sung Ju Hwang. Adversarial self-supervised contrastive learning. Advances in Neural Information Processing Systems, 33:2983-299,, 2020.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International Conference on Machine Learning, pp. 1894.1894. PMLR, 2017.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018
Zachary Nado, Shreyas Padhy, D Sculley, Alexander D' D'mour, Balaji Lakshminarayanan, and Jasper Snoek. Evaluating prediction time batch normalization for robustness under covariate shift. arXiv preprint arXiv:2006. 10963, 2020.

---

Published as a conference paper at ICLR 2025
A Tuan Nguyen, Thanh Nguyen-Tang Ser-Nam Lim, and Philip HS Torr. Tipi: with transformation invariance. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 24162 2417212117,, 2023.
Test time adaptation
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In International Conference on Machine Learning, pp. 16888-16905. PMLR, 2022.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In The Eleventh International Conference on Learning Representations, 2023.
Vardan Papyan, XY Han, and David L Donoho. Prevalence of neural collapse during the terminal phase of deep learning training. Proceedings of the National Academy of Sciences, 117(40): 24652-24663 2020.
Hyejin Park, Jeongyeon Hwang, Sunung Mun, Sangdon Park, and Jungseul Ok. Medbn: Robust test-time adaptation against malicious test samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5997-6007, 2024.
Ori Press. Steffen Schneider, Matthias Kümmerer, and Matthias Bethge. Rdumb: A simple approach that questions our progress in continual test-time adaptation. Advances in Neural Information Processing Systems, 36, 2024.
Xiaoqian Ruan and Wei Tang. Fully test-time adaptation for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1038-1047, 2024.
Steffen Schneider, Evgenia Rusak, Luisa Eck, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. Improving robustness against common corruptions by covariate shift adaptation. Ad- vances Neural Information Processing Systems, 33:11539-11551, 2020.
Mattia Segu, Bernt Schiele, and Fisher Yu. Darth: holistic test-time adaptation for multiple object tracking. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp- 9717-97271 2023.
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. Advances in Neural Information Processing Systems, 31, 2018
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. arXiv preprint 1.X.v:1005.12152 2018.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. arXiv preprint arXiv:2006. J07226 2020.
Dequan Wang, Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. In International Conference on Learning Repre- sentations, 2021.
Qin Wang, Olga Fink, Luc Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7201 2022.
Wei Wang, Zhun Zhong, Weijie Wang, Xi Chen, Charles Ling, Boyu Wang, and Nicu Sebe. Dy- namically -eiddd adaptation: backward-fres approach for test-time domain adaptive semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24090-24099, 2023.
Dongxian Wu, Shu-Tao Xia, and Yisen Wang. Adversarial weight perturbation helps robust gener- alization. Advances in Neural Information Processing Systems, 33:2958-299 2020.
Tong Wu, Feiran Jia, Xiangyu Qi, Jiachen T Wang, Vikash Sehwag, Saced Mahloujifar, and Prateck Mittal. Uncovering adversarial risks of test-time adaptation. In International Conference on Machine Learning, pp. 37456-37495. PMLR, 2023.
12

---

Published as a conference paper at ICLR 2025
Dongyoon Yang, Insung Kong. and Yongdai Kim. Enhancing adversarial robustness in low-label regime via adaptively weighted regularization and knowledge distillation. In Proceedings of the IEEE/CVF International Conference Computer Viston, pp. 4552 45624456,11 2023a.
Dongyoon Yang, Insung Kong, and Yongdai Kim. Improving adversarial robustness by putting more regularizations on less robust samples. In International Conference on Machine Learning, pp. 39331-39340 PMLR, 2023b.
Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 15922-15932. 2023.
Hongyang Zhang, Yaodong Yu, Jiantao Jiao, Eric Xing, Laurent EI Ghaoui, and Michael Jordan. Theoretically principled trade-off between robustness and accuracy. In International Conference on Machine Learning, pp. 7472-7482. PMLR, 2019.
Zhi Zhou, Lan-Zhe Guo, Lin-Har Jia, Dingchu Zhang, and Yu-Feng Li. Ods: Test-time adaptation in the presence of open-world data shift. In International Conference on Machine Learning, pp- 42574 42588. PMLR, 2023.
13

---

Published as a conference paper at ICLR 2025
A1 DETAILS OF THEOREM 1
Here, we provide a detailed proof of the decomposition of the robust risk given in Eq. 9.
Theorem 1. For DNN for parameterized by the robust risk R(0)) for batch of samples (X Y3) can be written as:
R(0) = ((gg. 1 Fo(X3) # v +
(AI)
p f E (x E) (x( + 1o((t)) pl ((r = (o(X 1x/)
Proof. From Eq. 6. R(0)= = (0) + Rody (0). where, Rm) = x2) (1F(( + YA}
and Ridy (0) = [x), y,y1 (3Xe E B,((X Fo(X)) + Fx(8)), F yB}.
Since,
Rody (0) EX) (3X B,(X + Fo(x)) =Y
= -E(xg,yg) 1 EX E B, ( 1 (Fo(X Y3}
Exx [1 (386 (x [x(b)) + =o(( BoaIx)) 1 -
P. Law of Iterated Expectation]
[ (382 (x (x(X)) + (o( P PY = Fo(Xb) x3)]
-Ex;
-Exg [ {3ax% e B c) (x(()) + Fo(X
}}
Exx [p (y;Y; = (x())) xx)]
= P (3X² E (x) (o(X)) # (oo(8)) = 1Xa)
Thus the equality holds.
A2 EFFECT OF DNN ARCHITECTURE ON FCA
To examine whether the source DNN architecture significantly impacts TTA vulnerabilities, we eval- uated the performance of FCA the MobileNet family of DNNs, specifically using MobileNet-vv as the source DNN. We assessed the performance of five baseline TTA methods across three bench- mark datasets and report the results in Table 7. Our findings show that TTA methods exhibit similar vulnerabilities on CIFAR-10C 10C and CIFAR- 100C datasets as those observed with ResNet variants. However, for the ImageNet-C C benchmark, MobileNet-v2 proved be even more vulnerable, with performance degradation under FCA being ~ 10% greater compared to the ResNet-50 results.
A3 PERFORMANCE EVALUATION FOR ADVERSARIALLY TRAINED MODELS
A potential defense against the vulnerabilities highlighted by FCA is to proactively use an adversari- ally trained source DNN. To evaluate this, we utilized the adversarially trained WideResNet-22 with an a budget (ex = 8/255) from Robustbench Croce et al. (2020) by Wu et al. (2020), and assessed its performance on CIFARIO 10-C and CIFAR- -IFAR-100C benchmark datasets. The results are reported in Table 8. Adversarially trained DNNs are highly effective against FCA when the same perturbation is used for both crafting adversarial examples and training the source DNN. However, with a dif- ferent perturbation budget, such as an la norm constraint of (E 8/255), FCA can still degrade performance by approximately 4%. Furthermore, for the CIFAR- 100C dataset, adversarially trained source DNNs result in more than 10% increase in eπror rate during adaptation with benign data. This is unexpected, as TTA is generally intended to handle online data batches without adversar- ial perturbation, raising concerns about the robustness-utility trade-off in deploying adversarially
14

---

Published as a conference paper at ICLR 2025
Table 7: (% Error) comparison on MobileNet architectures.
TTA Method
Dataset
Attack Method
TeBN
TENT
EATA
SAR
SoTTA
w/o Attack
21.04
21.55
23.94
20.93
19.91
DIA
35.54
35.11
35.56
34.44
34.48
CIFARIO-C
DIA (PL)
22.07
22.64
24.87
21.55
21.03
TePA
22.44
22.61
24.81
21.63
21.15
FCA
39.33
38.34
40.01
38.07
38.09
w/o Attack
45.55
44.81
45.83
44.63
43.84
DIA
57.13
55.45
57.03
56.14
55.93
CIFAR100-C
DIA (PL)
46.67
46.55
47.03
46.41
45.59
TePA
46.74
46.51
46.98
46.55
45.71
FCA
56.88
55.19
57.21
56.04
55,45
w/o Attack
54.2
52.97
53.78
52.83
51.29
DIA
71.56
70.37
70.45
70.87
68.55
ImageNet-C
DIA (PL)
57.5
56.44
56.29
55.31
54.92
FCA
71.44
70.01
70.31
70.15
70.22
Table 8: (% Error) comparison adversarially trained models.
TTA Method
Dataset
Evaluation Setup
TeBN
TENT
EATA
SAR
SoTTA
Chattacked(Standaaa Unattacked(Advv trained)
17.14
16.98
19.21
16.88
16.42
19.21
16.22
18.44
17.91
15.40
CIFARIO-C
FCA (e 8/255)
21.44
18.01
20.25
19.83
17.17
FCA (f2 = 0.5)
23.45
20.14
22.03
21.55
19.03
Unattacked(Standaal Unattacked(Adtt trained)
31.27
30.91
31.87
30.9
29.3
42.04
41.59
42.14
41.51
41.04
CIFARI00-C
FCA (6.0 88/255)
43.01
42.57
20.25
42.79
41.85
FCA (E2 0.5)
46.44
45.22
44.55
45.01
44.76
trained DNNs. Additionally, adversarial training known to reduce accuracy clean data Zhang et al. (2019); Tsipras et al. (2018). Thus, further scrutiny is required to develop computationally lightweight test-time defenses that are effective against FCA without impairing TTA performance on clean or benign samples from different domains.
Table (% Error) comparison on robust models
TTA Method
Dataset
Evaluation Setup
TeBN
TENT
EATA
SAR
SoTTA
Unattacked(AugMixx FCA (6.0 = 8//25
15.37
14.81
16.47
14.53
14.11
CIFARIO-C
24.33
23.21
24.98
23.05
22.87
Unattacked(AugMixx
29.34
28.77
30.21
28.55
27.87
CIFAR100-C
FCA
36.41
35.22
37.02
34.75
34.28
A4 PERFORMANCE EVALUATION FOR ROBUST MODELS
To further understand how the robustness of the source DNN influences FCA, we analyzed the per- formance of FCA against source DNNs known for their robustness distribution shifts. Specifically, we utilized the WideResNet-22 28 model trained with AugMix Hendrycks et al. (2019) from Robust- bench Croce al. (2020), and the evaluation results are presented in Table 9. While AugMix-traines
15

---

Published as a conference paper at ICLR 2025
models are effective in enhancing robustness against various distribution shifts, they remain highly vulnerable to FCA when deployed for TTA.
16


Reviews:
[{"review_id": "N5uAMXoCHp", "reviewer": "Reviewer_fWBZ", "summary": "This paper proposes a novel label-free adversarial attack against Test-Time Adaptation (TTA) algorithms, called Feature Collapse Attack (FCA). The authors argue that existing TTA attack methods unrealistically assume access to test data labels, making them impractical for real-world scenarios. They address this gap by formulating a new attack that leverages the principles of Neural Collapse (NC) to degrade TTA performance without needing ground truth labels.", "questions": "1. **Assumption of White-Box Access:** The attack assumes white-box access to the model parameters, which might not always be realistic. Analyzing the attack's performance under more restricted settings, such as black-box or gray-box access, would provide a more complete picture of its capabilities.\n2. **Attack Algorithm:** The attack part of the FCA algorithm is almost entirely borrowed from the PGD in the adversarial sample attack, which will affect the contribution of this paper. Furthermore, does this suggest that the method could be well enhanced by using more powerful attacks?\n3. **The Robustness of Victim Model:** The effectiveness of the proposed attack may be limited when targeting inherently robust models **[R]** or those with defense mechanisms (adversarial training), since it builds upon adversarial attack principles. How significantly does the attack's performance degrade when faced with robust or defended victim models?\n\n**[R]** MEMO: Test Time Robustness via Adaptation and Augmentation.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The paper addresses a significant limitation of existing TTA attack research by introducing a label-free approach. This is crucial for practical security evaluations of TTA, as real-world adversaries are unlikely to have access to test labels.\n2. The attack design is well-motivated by the theory of Neural Collapse, providing a clear rationale for the chosen loss functions.\n3. FCA achieves competitive performance against state-of-the-art attacks that utilize labels, and even surpasses them in some cases. This demonstrates its effectiveness and potential impact.\n4. The authors evaluate their attack across multiple datasets and TTA methods, providing a thorough assessment of its generalizability.", "weaknesses": "See Questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "x8VFX0wIM7", "reviewer": "Reviewer_JDoU", "summary": "The paper concentrates on the adversarial vulnerability in test-time adaptation (TTA). To explore more practical adversarial attacks, the paper designs a novel attack algorithm, named Feature Collapse Attack (FCA), which can achieve comparable performance without access to labeled test samples. This method consists of three loss designs, which are Nearest Centroid Loss, Feature Collapse Loss and Feature Scattering Loss. Experimental results demonstrate the outstanding performance of the proposed method.", "questions": "The paper considers the worst-case scenario where the attacker has knowledge of the TTA algorithm and has white-box access to the latest updated model parameters provided by some malicious insider. Under this assumption, is it reasonable to hypothesize that the attacker has no access to the true labels?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "+ **New Scenario**: The paper explores a novel scenario, where the attacker has no access to the true labels.\n+ **Simple Implementation**: The proposed method is relatively easy to implement and follow.", "weaknesses": "+ The number of baseline attack methods is limited. More comparison with multiple attacks is expected.\n+ In line 134, there is a repeated word \"the\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rkKfnHENXv", "reviewer": "Reviewer_k7PY", "summary": "The authors study the robustness of test-time adaptation (TTA) methods and propose a novel label-free attack. They argue that the assumption of the attacker having true labels on the test set is too strong for the real-life scenarios and without it the existing DIA attack is much less effective. However, the FCA attack proposed in the paper, that uses the idea of neural collapse, remains effective in the label-free scenario which emphasizes the need to further study the TTA robustness.", "questions": "1) Why is the perturbation initialized with 0.5 and not e. g. as random noise? (Algorithm 1, step 2) Wouldn’t the optimization benefit from this?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Originality**: label-free attacks on test-time adaptation seem to be a novel direction. The paper demonstrates the weaknesses of existing defence mechanisms (Section 7.1)\n\n**Quality**: the results are supported with numerous experiments on different datasets (CIFAR10-Cm CIFAR100-C, ImageNet-C) and have theoretical justification (Theorem 1 with proof in Appendix). Ablation study with respect to different loss components is performed (Table 4). \n\n**Clarity**: the paper is in general well structured.\n\n**Significance**: robustness of test-time adaptation is important in real life applications.", "weaknesses": "**Quality:**\n\n1)\tThe experiments are performed only with the ResNet architecture variants (line 316). Empirical results would benefit from including some other popular architecture e. g. one based on a vision transformer. At least in the Appendix. \n\n**Clarity:**\n\n1)\tFigure 1a: the process in the Figure 1a is hard to grasp for an outside reader. The diagram flow is confusing. Consider making it clearer where the flow starts and ends.  The overview figure has to be more intuitive to help the reader better understand the paper idea and not to confuse them. Consider simplifying it or breaking into several Figures. The top dotted line is missing an arrow on its end.\n\n2)\tPlease elaborate on how the pseudo labels are defined or add a citation (line 364)\n\n**Typos and minor comments:** \n\n1)\tLine 090 - “as” not needed: This makes our proposed attack **as** a prevalent threat that requires countermeasure\n\n2)\tLine 134 - additional “the”: and **the** the set\n\n3)\tLine 172 - Why is “Nearest Neighbour Classifier” abbreviated as NCC and not NNC?\n\n4)\tLine 265 - the sentence is too complex and poorly constructed, it is hard to understand its point. The more the last layer activations collapse to class mean vector, the likelihood of having a region close to it where the actual prediction deviates if the sample is projected into that region decreases\n\n5)\tLine 314 - delete one “use”: we use the use the provided data.\n\n6)\tIt would be good to include “w/o Attack” value (Table 1) in the Table 4 to better the relative improvement of each variant. The L_nc + L_col would also be good to include for completeness. Perhaps there is some synergy between these two terms that make their combination stronger than the sum of each term.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "TTIVn5V5oK", "reviewer": "Reviewer_AcHd", "summary": "To address the distribution shift of deep neural networks (DNNS), test-time adaptation (TTA) updates the model parameters using test samples. However, recent work reveals that TTA algorithms are vulnerable to adversarial attacks. This paper proposes a novel attack named Feature Collapse Attack (FCA). To be specific, the adversarial generated through FCA can be mixed with benign samples, thereby degrading the performance of the target model. A main feature of FCA is that it does not need the label of the test samples.", "questions": "Thanks for submitting the paper to ICLR 2025!\n\nThis paper aims to address a timely and important problem, i.e., evaluating the adversarial robustness of TTA algorithms. As the authors claimed, launching adversarial attacks against TTA algorithms is not a novel question, but at an early stage. Meanwhile, mounting adversarial attacks against TTA algorithms is not trivial, as the capability of the attacker is limited. The authors demonstrate the effectiveness of their proposed FCA attack. However, I have the following questions.\n\n- My biggest concern is the threat model of this paper.\n\n     - One of the main claims of this article is the practicability of the attacks. The paper pointed out that previous attacks like DIA need the label of the test sample. The reviewer agrees that it is very valuable to design a label-free attack. However, in Section 4, the authors claimed that the attacker has white-box access to the latest updated model parameters provided by some malicious insider. In my opinion, this assumption is impractical. First, who is the malicious insider? Can the authors provide a real-world example? \n     - In the threat model of (Cong et al., 2024), they proposed a black-box attack named TePA. TePA does not need access to the parameters of the updated model, which is more practical.\n     - The authors also claimed that \"existing threat models assume test-time samples will be labeled\", and TePA (Cong et al., 2024) relies on access to the entire source dataset to train a substitute model. Actually, TePA also is a label-free attack, and it does not need access to the entire source dataset. It only needs a surrogate dataset that has a similar distribution of the source dataset. I suggest that the authors reorganize the introduction of related work.\n\n- In experiments like Table 1, this paper does not compare the attack success rate with TePA (Cong et al., 2024). \n- This paper considers five target TTA methods, which all need the test samples to be fed in a batch-by-batch manner. However, some TTA methods like TTT [R1] can accept only one sample to adjust the model. I am curious if FCA can attack such kinds of TTA methods.\n- The proposed attack seems not to be stealthy. In other words, FCA needs a high poisoning rate (Table 3).\n- How many batches does FCA need to effectively decrease the target model's performance?\n- In (Cong et al., 2024). they also proposed several defenses, which are not discussed in this paper.\n\nReference\n\n[R1] Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. ICML 2020.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- This paper is well-written and easy-to-follow\n- TTA algorithms are the key developing trend of AI, and the evaluation of the adversarial robustness of TTA algorithms is timely and necessary.", "weaknesses": "- The threat model of this paper is impractical.\n- The experiments are insufficient.\n- Important baseline attack is missing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)