On the Adversarial Risk of Test Time Adaptation: An Investigation into Realistic Test-Time Data Poisoning
Authors: Yongyi Su
Yushu Li
Nanqing Liu
Kui Jia
Xulei Yang
Chuan-Sheng Foo
Xun Xu
Venue: ICLR 2025 Poster (2025)
Keywords: test time adaptation
continual learning
data poisoning
Primary area: transfer learning, meta learning, and lifelong learning

Abstract: Test-time adaptation (TTA) updates the model weights during the inference stage using testing data to enhance generalization. However, this practice exposes TTA to adversarial risks. Existing studies have shown that when TTA is updated with crafted adversarial test samples, also known as test-time poisoned data, the performance on benign samples can deteriorate. Nonetheless, the perceived adversarial risk may be overstated if the poisoned data is generated under overly strong assumptions. In this work, we first review realistic assumptions for test-time data poisoning, including white-box versus grey-box attacks, access to benign data, attack order, and more. We then propose an effective and realistic attack method that better produces poisoned samples without access to benign samples, and derive an effective in-distribution attack objective. We also design two TTA-aware attack objectives. Our benchmarks of existing attack methods reveal that the TTA methods are more robust than previously believed. In addition, we analyze effective defense strategies to help develop adversarially robust TTA methods. The source code is available at https://github.com/Gorilla-Lab-SCUT/RTTDP.

Published as a conference paper at ICLR 2025
ON THE ADVERSARIAL RISK OF TEST TIME ADAPTA- TION: AN INVESTIGATION INTO REALISTIC TEST- TIME DATA POISONING
Yongyi Su Yushu Lit.4r, Nanqing Liu Kui Jia', Xulei Yang", Chuan-Shenp Foo", Xun Xu
South China University of Technology
Southwest Jiaotong University
"The Chinese University of Hong Kong, Shenzhen
IIntiituee for Infocomm Research (T²R), A*STAR (eesuyongyi, eeyushuli @mmmm...... scut..ssstt.... wcu.sscu..c.ooo.... cn lansingl 630163 (3.163. com, poo_chuan_shoe
edu. en
xu aatar..traaaaaa..m.....a edu...rredu.gg..
ABSTRACT
Test-time adaptation (TTA) updates the model weights during the inference stage using testing data to enhance generalization. However, this practice exposes TTA to adversarial risks. Existing studies have shown that when TTA is updated with crafted adversarial test samples, also known as test-time poisoned data, the perfor- mance on benign samples deteriorate. Nonetheless, the perceived adversarial risk may be overstated if the poisoned data is generated under overly strong as- sumptions. In this work, first review realistic assumptions for test-time data poisoning, including white-box versus grey-box attacks, access to benign data, attack order, and more. We then propose an effective and realistic attack method that better produces poisoned samples without access to benign samples, and derive an effective in-distribution attack objective. We also design two TTA-awar attack objectives. Our benchmarks of existing attack methods reveal that the TTA methods are more robust than previously believed. addition, we analyze effective defense strategies to help develop adversarially robust TTA methods. The source code is available at /ottts::/giithi www./ooooiioiieeee Lab- SCUT/RITDP
INTRODUCTION
Test-time adaptation (TTA) emerges effective measure to counter distribution shift inference
stage
(Wang et al. 2020; Su et al. 2022 Zhong et al. 2022 Chen et al. 2023 Chi et al. [024 Wu
al. 2024). Successful TTA methods leverage the testing data samples for self-training (Wang
et al.
2020 Su et al. 2024b). distribution alignment (Liu et al. 2021 Su et al. 2022) prompt
tuning (Gao al. 2022: Liu et al. 2025). Despite the continuing efforts into developing computation
efficient and high caliber TTA approaches, the robustness of TTA methods has not picked up until recently, leading to studies examining the robustness of TTA methods under constant distribution shift (Song et al. 2023), correlated testing data stream (Su et al. [2024a] Niu et al.) 2023), open-world testing data (Li et al.| (023), adversarial robustness (Wu et al. [0023] Cong et al. 2023), , etc. Among these risks, adversarial vulnerability warrants particular attention due to its potential for evading human inspection and the significant consequences of admitting malicious samples during TTA.
Existing research frames the ad adversarial risk of Test- Time Adaptation (TTA) the crafting of poisoned testing data, resulting in models updated with such data performing poorly on clean testing samples (Wu et al., 2023) Cong al. 2023). Consequently, this task is also referred to Test-Time Data Poisoning (TTDP). The pioneering work DIA (Wu et al. (023) introduced a poisoning approach by crafting malicious data with access all benign samples within minibatch, leveraging real- time model weights for explicit gradient computing, i.e., a white-box attack. Another concurrent
'Equal contribution. 'Correspondence to cxu_xunee Sunniiraann @2r...staar...du...... This work was done during Yongyi Su, Yushu Li and Nanqing Liu's visit I'R.

---

Published as a conference paper at ICLR 2025
study (Cong al.) 2023) implements poisoning by preemptively injecting all poisoned data to attack the model even before TTA starts. While these explorations conclude that TTA methods susceptible to poisoned data, evaluations based on unrealistic assumptions may exaggerate the adversarial risk for several reasons. i) Access to real-time model weights (white-box attack) is often considered overly optimistic, especially given that models are constantly updated during adaptation. Therefore, grey-box or even black-box attack is preferred for TTDP. ii) The adversary is typically assumed only to be aware of the query samples submitted by themselves. Thus, benign samples submitted by other users should not be utilized for crafting poisoned data, for instance, through bi-level optimization (Wu et al. 2023). iii) Crafting poisoned data requires querying the model with testing samples (Wu et al., (023). Repeatedly querying the model from a single user could easily trigger alerts defensive systems. Therefore, any query sample, whether adversarial or benign, should be counted towards the attack budget. iv) Following the above concern, the adversary should not monopolize the entire testing bandwidth. This constraint translates to a scenario where poisoned data only partially occupies the testing stream, and the adversary is not allowed to inject all poisoned data at once even before TTA starts (Cong et al. (023).
To the best of our knowledge, existing attempts at test-time poisoning have not fully addressed the above realistic concerns. In this work, we aim to propose a threat model that advances towards more Realistic Test-Time Data Poisoning (RTTDP). Firstly, we formulate the threat model under a grey-box attack scenario, where initial model weights are visible. We distill a simple surrogate model from the online model using only the adversary's 's queries, enabling efficient gradient-base based synthesis of poisoned data. Empirical analysis demonstrates that the distilled surrogate provides sufficient information for crafting effective poisoned data. Moreover, to constrain the attack budget, we reformulate the bi-level optimization objective proposed in prior work (Wu et al., 2023) by replacing benign samples with poisoned data only. Through reasoning on generalization error, we illustrate that the attack loss defined on poisoned data can be generalized to benign samples if the distributions between poisoned and benign samples are identical. This insight motivates us to introduce a feature distribution consistency regularization for in-distribution attacks, eliminating the need for additional benign samples to construct the outer objective. Finally, we devise two alternative attack objectives tailored to the unique features of Test-Time Adaptation (TTA) methods. We first propose a high-entropy oriented attack to generate poisoned samples biased towards high entropy. This approach proves effective in compromising TTA methods based on entropy minimization (Wang) et al. 2020). However, high-entropy attacks may become less effective with the introduction of simple detense techniques, such as confidence thresholding. Therefore, we explore a low-entropy based attack objective aimed at attacking towards non- non-ground-o ground-truth class. The combined threat model is applied diverse range of TTA methods, resulting more effective outcomes compared to existing threat models. An overview of the overall framework is presented in Figure 1
In addition to crafting effective threat models, we delve into exploring practices conducive to enhanc- ing Test-Time Adaptation (TTA)'s adversarial robustness. Contrary to the reliance on adversarially trained models (Madry et al.) 2018) and robust batch normalization estimation (Wu et al.) (023), we draw inspiration from empirical observations of robust TTA methods. Our validation reveals that confidence thresholding. data augmentation, exponential moving averaging (EMA), and random parameter restoration represent potential directions for improving the adversarial robustness of TTA methods.
We summarize the contributions of this work follows.
We argue the unrealistic assumptions, e.g. white-box attack, access to benign subset and offline attack order, made in existing attempts at TTDP may overestimate the adversarial risk of TTA methods. To address this, we first propose key criteria for defining realistic test-time data poisoning scenarios.
Under our proposed realistic test-time protocol, we analyze the generalization error and introduce an in-distribution attack strategy with feature consistency regularization. This strategy eliminates the need for additional benign samples in evaluating the outer objective. In addition, we tailor attack objectives specifically for TTA methods, resulting in more effective poisoning.
We perform extensive evaluations state-of-the-ar TTA methods, demonstrating the efficacy of our proposed in in-distribution attack strategy. Furthermore, we identify certain practices that are conducive to improving realistic adversarial robustness.
2

---

Published as a conference paper at ICLR 2025
Adversary View
Benign User View
Bab
BB
Ba n B,
((
(Eq.
P P(P))
Lack
((6.)) Me2
((%))
Surrogate Model
BI
y2
3/4
35
Distill the surrogate model fees via Lave (E4. I)
Calculate Emer Rate on Benign Users' Samples
TTA pipeline
Figure I: Illustration of the proposed Realistic Test-Time Data Poisoning (RTTDP) pipeline. Bas indicates the adversary benign subset, and B, indicates the adversary poisoned subset where the samples are poisoned from the clean samples in B.t.. B, indicates the benign users' subset where the samples are used to validate the adversarial risk TTA pipeline poisoned data by attacking regularized objective without accessing henign samples from other users. Model attacked when carrying out TTA on testing data stream mixed with benign and poisoned data.
these samples cannot be access by the adversary. Adversary generates
2 RELATED WORK
;; Niu al. (2022); Su et al., (2022): Li et al.) (2023): [ et al. (2024); Wu et al. (2024) and Wang et al. (2025) have shown significant success in bridging the domain gap by using stream- based testing samples to dynamically update models real time. The success of TTA is mainly attributed the self self-supe learning testing data. While it has proved sensitive to confirmation bias (Arazo et al. 2020). many solutions were proposed to minimize the influences of the wrong pscudo-labels, including minimizing sample entropy (Wang
Dest-Time-A Adaptation (TTA): Wang al. (2020)
ct al. 2020 Liang et al., 2020), adding regularization terms (Song et al. 2023 Su et al. 2024b). using
confidence thresholding NNu et al. [022] 2023), updating models with exponential moving average architectures (Wang et al. 2022 Döbler et al. (2023), partially updating model weights (Wang et al.)
2020, Yuan et al.] 2023), and augmenting testing samples (Zhang al. 2022 Döbler et al. 2023), all
aimed at minimizing sample distribution discrepancies. However, adaptation during the testing stage remains highly risky. In this work, we aim to investigate the risk of TTA posed by data poisoning.
Robustness in Test- Time Adaptation: Recent research has increasingly focused on the robustness of TTA in realistic deployments. Wang al. (2022) and BBrahma & Rail (2023) tackle issues of catastrophic forgetting due to changing test data distributions. Niu et (2023) Gong et al. (2022) and
Yuan et (2023) address non-i.i.d. and shifting label distributions in test data, [i al. (2023) and
Zhou et al. 2023) introduce open-world scenarios in TTA, where test data may include novel classes not present in the source domain. Furthermore, Wu et al. (023) and Cong et al. (2023) investigate the threat of data poisoning in TTA, where attackers alter test data to exploit vulnerabilities in TTA methods. DIA (Wu et al. 2023) uses bi-level optimization to degrade target sample performance, assuming white-box attack access. In contrast, TePA (Cong et al. 2023) includes exclusive offline stage for poisoning data on the source model prior to the TTA process. Inspired by these works, we focus on the severe threat of data poisoning. We examine the adversarial robustness of TTA under realistic conditions, with access only to the source model and partial test samples for data poisoning. Our attack, despite these constraints, outperforms previous methods, highlighting the significant adversarial risks that TTA methods face.
Ad Adversarial Attack & Data Poisoning: Adversarial risk is a crucial concern for models address for safe deployment, which can be divided into two categories: ad adversarial attacks and data poisoning.
Adversarial attacks (Szegedy et al. 2014 Akhtar Mian 2018 Goodfellow et al. 2014 Madry et al. 20088: Croce & Hein, 2020; Chen et al. [022]] Chakraborty et al. 20188) manipulate models during inference by adding small perturbations to input data. White-box attacks (Szegedy et al..
2014 Tramer et al. (0188) assume full access to the victim model, creating adversarial examples by
maximizing loss gradients. In contrast, grey-box attacks (Chen et al., 20177; 7, [yyas et al. 2018 Ru ct al. (2119) assume no model access, generating adversarial samples by estimating gradients through
intensive querying. Data poisoning (Biggio et al. 2012 Yang et al. 2017 Shafaai et al. 2018 Alfeld
al. 2016 Huang al. 2021 Fowl al. 2021 Fan et al. 2022) compromises models by injecting
3

---

Published as a conference paper at ICLR 2025
manipulated data into the training set, misleading the training process. Traditional data poisoning assumes that the attacker can observe and poison the entire training set at once. Recently, online poisoning Zhang et al. (020) relaxes this assumption by requiring knowledge of the model updating strategies. In this work, we explore a more realistic scenario of TTA's adversarial robustness, focusing on data poisoning without online model access and without multiple queries for the same data.
3 SETTING: REALISTIC TEST-TIME DATA POISONING
3.1 OVERVIEW OF TEST-TIME DATA POISONING
We first provide generic overview of test-time adaptation for a K-way classification task. We denote the testing data as D = {x; ] and pre-trained model as 6-- TTA methods often employ unsupervised loss, Lita, to update model parameters upon observing a minibatch of testing samples B, = { at timestamp L. Usually, subset of model parameters 0g S 0, is subject to update at timestamp f, and we denote the BN statistics as (15.) [u(B),, o² (B.) and the frozen parameters as 07, The posterior of the sample 2; towards the model parameter 0 is f(z;;;; 0) E [0, 1J The adversarial risk arises when a subset of testing samples are poisoned, e.g. through adding an adversarial noise ă = x + c, s.t. ||c||o I b. The model trained on poisoned data exhibit poor performance clean/benign testing samples. In a typical TTA scenario, the online model is queried by both adversary and benign users. Thus, we denote the query data from adversary as adversary poisoned subset B, [5:²] ]. The poisoned subset could be generated from arbitrary clean testing data, denoted as Bas = { [x (e.g. use any public clean images). The generation follows an additive noise, i.e. T. I; + Es, s.t. 34 E Bg X, Bad, where the noise e; is the data poisoning to be learned. The query data from benign users is denoted as benign subset BB = {æih. where Bas n BB = 0. Generally, we form the combination of both, B, B. U Bb, as a single TTA minibatch. The effectiveness of test-time poisoning is evaluated at the attack success rate (classification error) on benign subset B,, An illustration of batch split is presented the Appendix
[.... A.I.I
3.2 REALISTIC TEST-TIME DATA POISONING PROTOCOL
The adversarial risk of TTA methods must be assessed under realistic attacks. Existing works may have exaggerated the adversarial risk when attack is implemented under an overly strong assumption. We summarize the criteria that define a more realistic attack as follows.
White-Box V.s. Grey-Box Attack: Access to real-time TTA model parameters is a key factor for crafting realistic test-time data poisoning. Contrary to the assumption adopted by white-box adversarial attack (Szegedy et al.] 2014 Biggio et al., 2013) that a frozen model is deployed for inference, the TTA model parameters experience constant updating at inference stage. The update is performed on the cloud side, thus the attacker doesn't 't normally have access to real-time model weights. Such realistic assumption prompts us explore more relaxed grey-box test-time data poisoning, i.e. only the model architecture and initial model weights are available to the adversary, such the open-source famous pre-trained models (He al., 2016 Dosovitskiy al. 202 and
popular foundation models (Radford et al., 2021 Kirillov et al. 2023 Oquab et al. 22224).
The efffectiveness of TTDP is evaluated on the benign subset B3 submitted by benign users. In a standard cloud service, users are generally restricted from accessing the queries of other users. Thus the adversary should only have access to adversary subset Ba. This assumption prohibits the practice of crafting poisoned data by directly optimizing (minimizing) the loss on benign subset (Wu et al. 2023), on which the attack success rate (performance) is calculated.
Access To Benign Subset:
Attack Order: Finally, attacking TTA model in a realistic way should be implemented during the adaptation stage. Attacking the model before TTA begins is deemed less practical (Cong et al. 2023).
Based on the aforementioned key criteria, we provide summary of existing test-time data poisoning methods in Tab. [ Our analysis indicates that none of the current methods fully satisfy all the estab- lished criteria. Specifically, DIA (Wu et al. 2023) employs a white-box attack strategy, generating poisoned samples by maximizing the error rate on a subset of benign data. TePA (Cong et al. 2023) attacks the pre-trained model with an offline surrogate model prior the commencement of test-time adaptation. In the rest of the paper, we stick to the most realistic assumptions, i.c. online grey-box poisoning and no access benign subset, named as Realistic Test- Time Data Poisoning (RTTDP).
4

---

Published as a conference paper at ICLR 2025
Table 1: Taxonomy of methods based on the criteria for realistic test-time data poisoning.
Setting
Grey-box White-box
Access to Benign Subset
Atlack Order
DIA (W al
2023
White-box
Online
ToPA Cong ot al.
2023
Grey-box
Offline
RITDP (Ours)
Grey-box
Online
Adversarial Attacks on Standard Image Classification: Common grey-box black-box adversar- ial attack techniques are often impractical for RTTDP due to two primary challenges. First, existing adversarial attack methods operate
stafic model and inherently require multiple queries
for gradient approximation or fitness evaluation, as seen in query-based attacks (Li et al. 2020 Xu
al. 202T), }. genetic algorithms (Chen et al. 2019), and black-box optimization (Qiu al. 2021).
However, in the test-time udaptation setting, the online model is continuously updated with each query during the inference phase. Thus, repetitive querying the model for gradient approximation or fitness evaluation is unavailable. Second, traditional adversarial attack methods focus on crafting adversarial samples to degrade the performance of the attacker's own input data. In contrast, in realistic test-time data poisoning scenario, poisoned samples are introduced into the test-time adaptation process to degrade the performance of benign samples submitted by other users.
METHODOLOGY
4.1
GREY-BOX ATTACK BY SURROGATE MODEL DISTILLATION
To tackle the challenge of access TTA model parameters, propose to maintain surrogate model, denoted 0, at timestamp for the purpose of synthesizing poisoned data. To ensure good approximation, we distill the target model 0, into the surrogate model Ó, by leveraging the feedback of poisoned data from the online target model. Specifically, for each query to the target model, we minimize the symmetric KL-Divergence between the posteriors of the target and surrogate models, as Eq. [ Our empirical observations demonstrate that utilizing the ad adversarial subset Bg for distillation yields performance comparable to a white-box attack, as illustrated in Fig. (a).
Cdist
1
Σ
KLD (M(a;; (,)1((;; + KLD ((MM3
(1)
B.
2
8. EB,
4.2 IN-DISTRIBUTION TEST-TIME DATA POISONING
In this section, we further address the challenge of attacking TTA model without access to benign user's testing samples. In the first place, we revisit DIA (Wu al. (023), which formulated test-time poisoned data generation as bi-level optimization problem, in Eq
min 15.
Σ Z,ER,
Catk (2;; (B..)
Bb
s.t. B, B. Bs; 8 =
(2)
gur arg min Luta (B.)) (B.) = 0:* U 8d ue
8:¹
The above bi-level optimization problem employed in DIA (Wu al. 2023) aims to generate adversarially poisoned data B, by minimizing the loss function Latk, which is computed the benign samples B,, Although DIA approximates 0%* ze 0% to discard the inner TTA gradient update loop and reduce the number of queries to the online model, several realistic concerns still persist under the RTTDP protocol. First, as discussed in Sec. 3.2, DIA, as a white- box attack method, must query the online TTA model 0r for generating poisoned samples. To mitigate this issue, propose to leverage a surrogate model ê, as an proxy model, which distilled by Eq. with the last feedback of Ba,t-5. where 5 denotes the time interval between two injected poisoned subsets. Second, DIA evaluates the outer optimization by employing the benign samples B, (assuming access to benign users' query samples). The adversarial risk mainly arises from the injection of poisoned samples into the TTA training process. However, employing the validation (benign) samples as the outer optimization objective may result in an overestimation of this risk, as the specific poisoned samples could be tailored for the attack on the inference of the current batch of benign samples (Park et al. 2024), e.g., 0² rather than for attacking the TTA process, i.e., 0"
5

---

Published as a conference paper at ICLR 2025
Merte Seepho
Source Mocel Sumoguda Madee
NHE Alback
Fas Mag
MiL Allack
Onitne Hodel
0.4
samples
4.2
posamee samp un (6as)
polsosed camp ed UGua Aeupp
TE35
+N00 910 XIN -.12 Fat Syp A.Am
E471
Car ACwp
K.S.
Figure 2: (a) The attack performance comparison about the poisoned samples generated on Source (pretrained) Model, our proposed Surrogate Model and Online target Model (white box). (b) The T-SNE visualization of the feature points (before FC layer). Without Greg. common attack losses (c.g. maximizing cross-entropy) produce poisoned samples (orange dots) that are far from benign ones (blue dots), leading to less effective attacks. (c) The attack performance comparison between w.o. and w. Lreg. (d) The average prediction entropy of the poisoned samples generated by our proposed two different attack objectives, respectively.
To prevent from using benign users' samples for optimization and assuming grey-box attack, one possible solution is to swap the benign users' samples B, with the adversary's ss clean sample Bab based on the assumption that Bas and B, are drawn from similar distributions, and replacing the white-box model 0 with distilled model 0 and discarding the inner TTA loop 0 ss Or as adopted by DIA (Wu al. )023). resulting in the following formulation.
Σ Lock (1:; (BB))
min B.
|abl
I.EBuo
s.t. B. BB U Bub: (M=) ((.)) 0 U
Despite the above formulation alleviates the assumption of the available to access the benign users' samples and the online model parameters, it still remains elusive to tackle. First, as most TTA methods leverage the current batch statistics to estimate the statistics on target domain, forwarding B, results in esimating the BN statistics on the combination of B, and Bas which bias update of BN parameters. This may results in mismatch between the feature distribution of Bab and B,, if B, is queried independently. Thus, the poisoning effect may fail to generalize to B, This is evidenced by an empirical study into the distribution of poisoned data in Fig. 2 (b) where the distribution of generated poisoned data (orange dots) deviate substantially from the benign samples (blue dots) if the is is directly attacked. Therefore, we are prompted to explore a solution that does not explicitly require forward pass for both B3 and Bab simultaneously and is able to transfer the attacked effect from B, to the benign subset i.e. Bar or Br.
To address this challenge, we propose introducing additional constraints and integrating the optimized target Bas with the optimizing objective B, into unified objective Ba, as presented in Eq. 4 where D(P1, P2) is a metric of two distributions and P. and Pos refer to the feature distributions of B, and Bas. We have the follow reason why the formulation is effective. If B, and B& have the similar distribution, can have Pa + Pab E.) (Latx (2)] - (p [aut (2)) according to the Probably Approximately Correct (PAC) learning framework (Valiant). Thus attack against B, has a high chance to generalize Bab.
min &
B
Σ Z,E8.
Lack (B::6((Ba)): s.t. D(Pa, =0
(4)
Feature Consistency Regularization for In-Distribution Attack: To achieve indistinguish- able distribution between P. and Pa.. we propose to measure the discrepancy at feature level and introduce the discrepancy as a constraint to the optimization problem. Crucially, since the attack loss is defined in the representation extracted by the backbone network, the distri- bution consistency is idcally imposed on the intermediate features except for the final seman- tic one. Specifically, we denote the 1-th intermediate feature map before each normalization layer as 2! f'(x;) E Hi Di, A single Gaussian distribution is fitted to intermedi- ate layer features, as 1 H.W, Enw d "thw Σ = 1 H.W. Σn. (+ - (1) - x4) and (2) (2 ( - -A/)T, where & and 2, refer to the
A = 1. sample features from B, and Bab. respectively. The KL-Divergence between feature distributions introduced as the constraint.
n,W, Σn,w thum Σ/ H,W, En w.w
Σ BEDDNNGEEWWNI ED Greg L
(5)

---

Published as a conference paper at ICLR 2025
With the introduced constraint Creg O, we finally formulate the problem Eq. The problem now degenerates to a single level optimization with constraints which can be easily converted into an unconstrained optimization problem via Lagrangian multiplier (Lag) 2008). The unconstrained problem can be solved in an iterative fashion with detailed algorithm presented in the Appendix AA.1.
min B.
Σ
Lark
I;j O((())
Ba
E.EBB
s.t.
@hr {u((B), (Bu)]:
((((o)) 0e U Creg =0
Σ Lack ((a:)(((a)) + >
(6)
=> min max
Ba
[Bal
Z.EB.
Σ Z,EB.
Lack (x:: (((a)) + T. Σ LALDINAAEEIINN ED),
= >min
max
B.
(Ag
1)
where L is the number of intermediate feature layers. Through optimizing the above objective, now we could craft the effective poisoned data B, that satisfies Ez Lutk (3r;; 0.) Rs Ex,e8, Lau (x;;; 0z), since the approximation equation Pa - Pab + Pa now holds. Fig. (c). demonstrates the effective- ness of this in-distribution attack, where the attack performance of the attack objective combined with the feature regularization (orange bars) is always higher than that of the corresponding attack objective alone (blue bars) in different TTA methods. Next, we would introduce the attack objectives batk that we designed to effectively generate different kinds of poisoned samples.
4.3 TTA-AWARE ATTACK OBJECTIVE
The specific design of attack objective Latk warrants careful consideration. The existing works examined both targeted and indiscriminative attacks, demonstrating that both are effective against state-of-the-art TTA methods (Wu al. (023). However, we argue that attack objective is not universally effective against all TTT methods. Therefore, we investigate two types of attack objectives as follows, and the prediction entropy of poisoned samples generated via the proposed attack losses can be compared in Fig. (d).
High Entropy Attack Objective: Self-Training based TTA methods, e.g. TENT, RPL, are vulnerable for out-of- -istiitiitttion samples with high entropy (Niu et al., 2023), and therefore, a straightforward way to generate poisoned data attack TTA model is by maximizing the entropy of poisoned data as proposed in TePA (Cong et al. (223)) since these poisoned samples with high prediction entropy would induce high updating gradient. However, maximizing the prediction entropy does not guarantee wrong pseudo labels. Therefore, we propose a stronger high entropy attack objective called Notch High Entropy Attack (NHE Attack). Based on the uniform distribution, we set the probability in the ground-truue label to zero and construct the target distribution Q. Then minimize the cross-entropy against target distribution Q
k yi others
CHEE C
(1:2) =
Σ Qik logh s.l.
Qik
K
Low Entropy Attack Objective: High entropy attack objective is particular effective against self- training based TTA methods because of high updating gradient, yet they could be easily defended by some defense strategies, e.g. entropy thresholding. Therefore, we further explore new low entropy based attack objective. DIA (Wu et al. 2023) proposed to maximize the cross-entropy loss on the benign samples (indiscriminate attack) to generate the other poisoned samples. However, we empirically found that maximizing the cross-entropy loss without any constraints on one sample is prone to maximizing the probability of the most confident class (except the ground-truth, of one model, and feeding these samples into TTA model would bring up the following issues. i) The model will quickly bias towards the most confident class and collapse if without any class diversity constraints. ii) If class diversity constraints are applied (assembled in several TTA methods, e.g. EATA, ROID), this objective will become less ineffective, since class-biased poisoned samples will obtain a less updating weighting than other benign samples. Therefore, we propose class-balanced low entropy attack, termed Balanced Low Entropy Attack (BLE Attack). Specifically, we maintain an moving average probability confusion C' E 00, 1 to store the prediction bias in each class and find a global optimal label mapping M E {0, 1) KxX such that each class is attacked towards the most probable ground-truth class. Details of deriving label mapping M is deferred to the
7

---

Published as a conference paper at ICLR 2025
Appendix A.1.2 Finally, the BLE Attack objective is calculated as Eq. 8
Lakk (2:) - Σ1 arg max My..g) log (T:)) gfy
(8)
Overall Attack Strategy: We craft poisoned data by attacking the aforementioned in-distribution at- tack objective with regularization. Following the practice that poisoned data should be less discernible by human, employ a 40 steps Projected Gradient Descent algorithm (Boyd & Vandenberghe 2004) on the combined objective in Eq. 6 with budget b. More details of the whole data poisoning algorithm are deferred to the Appendix A.1.6
5
EXPERIMENT
5.1 EXPERIMENT DETAILS
Benchmark Poisoning Methods: We evaluated the following methods under our proposed RTTDP setting. Unlearnable Examples (Huang al. 202I) generates the poisoned noise by minimizing the cross-entropy of B, on a randomly initialized model. Adversarial Poisoning (Fowl et al. 2021 proposed to minimize the cross-entropy between the posterior probabilities of poisoned samples, B.. and their corresponding incorrect labels, D. where the incorrect labels are defined as
y/ 1. the poisoned data via maximizing the cross-entropy of other benign data. We adapt DIA to the realistic evaluation protocol by splitting Bas into two subsets of 50% each, i.e. 5% : B%, = 1: 1, and craft BE to maximize the cross-entropy of B%, TePA (Cong et al. 2023) proposed to maximize entropy to generate poisoned data and performed attack betore TTA starts. We adapt TePA to generate the poisoned data based on the source model and inject them into TTA pipeline on-the-fly. MaxCE (Madry al. 20188) an established way to create adversarial samples by maximizing the cross-entropy loss. Finally, we evaluate the two attack objectives proposed in this paper, i.e high entropy attack (NHE Attack) and low entropy attack against most probable and balanced non ground- truth class (BLE Attack). For both NHE Attack and BLE Attack, we evaluate the attack objective subject to the constraint of our proposed feature consistency (Eq. For all methods that require gradient-based optimization, we employ the Projected Gradient Descent (PGD) algorithm (Boyd & Vandenberghe 2004) to perform the constrained optimization. We use 40 steps PGD for all methods for fair comparison.
DIA (Wu et al. 2023) is one of the first approaches towards test-time data poisoning, generating
Datasets: We evaluate on three datasets, widely adopted for TTA benchmarking. CIFARIO-C. CIFAR100-C and ImageNet-C are synthesized from the original clean validation set by adding
various types of corruptions to simulate natural distribution shifts (Hendryeks & Dietterich 2019 We
choose corruption level 5 and perform continual test- time adaptation setting (Wang et al. (022) for
evaluation. Following prior works (Wang et al., 2022 Döbler et al.) 2023), we adopt the pre-trained
WedeResNet-28 (Zagoruyko & Komodakis (016), ResNeXt-29 (Xie et al. 2017), and ResNet-50 (He)
ct al. 2016) models for experiments on the CIFARIO-C CIFARI00-C and ImageNet-C datasets, respectively, More experiment details can be found in the Appendix A.3
5.2 EVALUATION ON TEST- TIME DATA POISONING
We present the results of comparing different attack objectives against state-of-the-art TTA methods in Tab. 2 Tab. 3 and Tab. for CIFAR10-C. CIFAR100-C and ImageNet-CC respectively. We make the following observations from the results. i) Contrary to the claims that TTA methods are extremely vulnerable data poisoning, under the realistic data poisoning protocol, without accessing to benign data, it's not trivial to transfer the adversarial risk from poisoned data to benign data, especially for the TTA methods using EMA model such as CoTTA and ROID. In particular, existing methods do not pose too much risk to more advanced TTA methods without feature consistency regularization. DIA and TePA are more effective on TENT and RPL. than other TTA methods. We attribute this to the fact that both TENT and RPL. are naive self-training methods without filtering testing samples, hence, poisoned data could easily mislead model update. ii) Our proposed two attack objectives generally perform better than existing poisoning methods, demonstrating better average ranking and a higher average еттог rate. This is attributed to the combination of the well-designed attack objective and the regularization of feature consistency. On the other hand, low entropy attack (BLE) obtains significantly improved with our proposed feature consistency compared with the similar low entropy attack i.e. MaxCE. iii) "Non-uniform" " attack in general yields higher attack success rate
8

---

Published as a conference paper at ICLR 2025
Table 2: Evaluation of test-time data poisoning under the RTTDP protocol for CIFARIO-C. We report the attack success rate (higher the better) for each TTA method and the average ranking (lower the better) for each attack objective. indicates that the method is modified to align with the RTTDP protocol.
Attack Freq
Attack Objecttiee
Source TENT
EATA
TTAC
CoTTA
ROD
Arrg. Exc (1)
Arg. Rack (1)
No Altock
19.72
18.03
17.41
16:37
Unlearnable Examples
[haan etal NOT
32.61
25.62
20:11
18.43
19:23
17.27
17.80
21:72
1.36
Adverssrial Pricuming
19:50
19.90
18.94
18.69
19.50
18.34
19:12
19:21
3.86
DIA'
Ww Atw.aaaaa 204
26.04
21.87
18.94
18.56
19.46
17.72
17:27
20.05
4.36
Unifomm
TeP/
Tikk & the
43.81
33.78
22:36
23.37
17.75
19:53
16.57
18.75
21.75
4.43
MaxCE
Madly 3 2015
18:55
20.51
18.17
18.50
19.50
16.85
18.57
18.71
6.001
B.F.E Altack (Chin
54.07
51.99
45.20
34,00
26,50
18.12
19:06
35.61
1.57
NHE Attack (Ours
73,86
72.40
29.73
15.67
24.56
17.54
17.00
36.25
2.86
No Altikk
19:20
20.36
17:25
16.89
18:74
16.18
15.81
17.86
5.71
DEA" TePA'
Cont al 2013
22.54
25.70
19:75
18.35
19.42
19.18
17.79
20.45
4.00
40:31
32:32
23:00
18.06
19:49
18.07
18.50
24.25
3:43
Non- Uniform
MasCE
Maary 2013
18.64
20.01
18.29
18.45
19.47
18.01
18.94
18.85
4.43
BLE Attack (Our)
56.17
45.65
50.97
34,55
27.54
19.25
20.12
56.42
1.43
NHE Attack (Ours)
74.93
73.65
27.56
18.75
24.95
20.96
16.77
36.78
2.00
Table 3: Evaluation of test-time data poisoning under the RTTDP protocol for CIFARI0O-C. We report the attack success rate (higher the better) for each TTA method and the average ranking (lower the better) for each attack objective. indicates that the method is modified to align with the RTTDP protocol.
Attack Freq.
Attack Ubjective
Source
TENT
RPL
EATA
TTAC
SAR
CoTTA
RUID
Avrg. Eir. (1)
Arg. Rank (2)
No Attock
611.55
47.12
32.20
31.93
31.62
52.13
29:11
57.77
7:29
Unlesrnable Examples Hhune al
2011
75.31
72.74
17.5
33.66
41.75
33.66
31.31
46. 46.45
3.71
Advamsrial Poisuning Post
2011
13.87
34:27
32.11
35.49
32:27
52.76
30.15
$2.99
5.1
DIA"
14.79
33.77
32.57
35.20
32,68
30.10
43.65
5:29
Unitoom
TePA*
46,23
15.79
81.98
26.74
33.41
35.79
32.41
32.24
46.91
3.57
MasC3:
Madly
34,08
40.75
35.15
BLC Altack (Ours)
13:93
76.71
47.30
34.58
43.25
32,81
32.27
48.69
2.28
Altack
33.85
56.09
31.48
No
49:60
31,45
52.07
32.56
25.39
58.26
DIA"
203
33.74
32,63
35.24
30.07
44.44
TePA"
82:70
36.40
34.89
33.27
Non-Uniform
MaxCE
46.33
41.89
38.05
33.20
42.13
31.90
32.78
31.66
35.91
B.FFF Attsck (Qurs)
21:30
60
94,46
94.58
40:05
34.14
33.53
31.45
Table 4: Evaluation of test-time data poisoning under the RTTDP protocol for ImageNet-C. We report the attack success rate (higher the better) for each TTA method and the average ranking (lower the better) for each attack objective. indicates that the method is modified to align with the RTTDP protocol.
Attack Freq.
Attack Objective
Source
TENT
SAR
CoTTA
ROID
Avg. Err. (1)
Avg. Rank (L)
No Atlack
63.49
61.26
63.02
53.42
60.30
5.50
DIA
Wu et al. 2023
67.18
62.91
64.09
56.97
62.79
4.00
TePA
Long el 4024
75.36
64,90
62.84
59.78
65.72
3.00
Uniform
MaxCE
Madry ét 2018
82.08
62.64
61.66
68.83
59.89
63.26
3.25
BLE Attack (Ours)
68.04
64.31
66.40
57.10
63.96
3.00
NHE Attack (Ours)
78.03
72.58
63.84
57.72
68.04
2.25
No Attack
61.81
59.52
62.51
50.36
58.55
6.00
DIA
Wu 2023
66.61
62.88
63.40
55.18
62.02
4.25
TePA*
Cong al. 2214
74.98
62.31
62.78
59.29
64,84
3.50
Non-Uniform
MaxCE
Madry et al 2018
81.98
62.09
65.58
72.39
59.76
64.96
2.50
BLE Attack (Ours)
67.61
65.95
65.75
56,23
63.89
2.75
NHE Attack (Ours)
77.49
73.67
64.02
57.09
68.07
2.00
than "Uniform" attack. This is probably due to consecutive attack being more effective in misleading model's 's update.
5.3 ABLATION STUDY ON ATTACK MODULES
In this section, we ablate our proposed modules including the surrogate model, the feature consistency regularization and two attack objectives to demonstrate their their indispensable contribution to the final results. We conduct the experiments on CIFAR1O-C and CIFAR 100-C datasets as shown in Tab. 5 First, comparing the use of the source model v.s. surrogate model for generating poisoned data, the surrogate model consistently delivers superior results, often approaching or even slightly surpassing those obtained with the online model, regardless of the attack objective. It demonstrates the effectiveness of our proposed surrogate model that is leveraged for generating on-the-fly poisoned data. Second, our proposed NHE attack objective is effective though using source model and without feature consistency regularization, that could be attributed to the high entropy samples easily mislead the model update and cause strong perturbation to the source knowledge. Third, under the surrogate model or online model, both BLE and NHE are significantly improved with the help of feature

---

Published as a conference paper at ICLR 2025
Table 5: The ablation study of our proposed modules under the RTTDP protocol on CIFARIO/100-C datasets.
Artack Model
Amtakk Objective
Feat. Cons.
CIFARIO
(1)
CIFARIODC
(1)
Source Model
BLE
19.26
18.82
19.80
19.29
34.21
31.62
32.17
32.67
Source Model
BLE
38,09
23.28
21.86
27.74
67.11
34.57
33.78
45.15
Source Model
NHIE
63.00
23.76
19:52
35.43
85.48
39.37
38.76
$4.54
Model
54.94
Samrogiic Model (Ours)
BLE
20.22
20:16
19.71
20:03
38.71
31.87
Surrogule Model (Ohrs)
RIF
54.07
45.20)
26,80
42.02
73,93
47.30
43.25
54.33
Surmgote Model (Ours)
NHE
72:77
21.93
19.51
38.07
$1.49
32. 16
31.68
48.44
Sumogotc Model (Ours)
NHE
Online Model
BLE
25.14
25.50
19.71
23.45
42.09
32.82
32.05
35.65
Online Model
BLE
56,75
52.32
27.38
45.48
77.92
49.91
44.01
57.28
Online Modil
NEIE
72.01
21.91
19
37.81
80
31
31.69
48.09
Online Model
NHE
28.73
consistency regularization, empirically demonstrating the reasonableness and effectiveness of our method.
5.4 EXPLORING EFFECTIVE DEFENSE PRACTICES
In this section, we explore effective defense practices. We conduct an ablation study for each of the above practices on top of a simple entropy minimization baseline method (Min. Ent.). As seen in Tab. 6 we make the observation that without any hypothesized defense practice, directly minimizing entropy is very sensitive to poisoned data, especially the high entropy attack (NHE). When entropy thresholding (Ent. Thresh.) is applied, we observe significant improvement in robustness under high entropy attack, suggesting rejecting high entropy testing samples from TTA is an effective defense practice. Furthermore, both data augmentation (Data Aug.) and exponential moving average update (EMA) are very effective defense practices. The former could perturb the testing sample towards non-adversarial direction while the latter prevents the model from updating too quickly, thus less sensitive to poisoned data. Finally, one might expect stochastic parameter restoration (Stoch. Resto.) to be an effective defense method. Despite exhibiting improved adversarial robustness alone, parameter restoration does not further improve the robustness when combined with other effective defense methods.
Table 6: Ablation study of hypothesized defense practices on CIFAR 10-C dataset.
Table 7: The generalisation of modules CIFARIO-C.
Thresh.
Aug.
Update
Swoch. Resto.
Atack Objective
TENT
EATA
43.81
54.07
DIA + Ours
27.02
19.41
46.24
35.86
TrA
33.78
23.37
24.01
20.05
TePA Ours
38.79
21.37
20.22
19.76
29. 68
50.68
MaxCE
20.41
20.30
MaxCE + Ouns
23.61
19.44
5.5 GENERALISATION OF OUR MODULES
In this section, to demonstrate the generalization of our method, we combine the existing attack objec- tives with proposed modules including the surrogate model and feature consistency regularization. The comparisons are shown in Tab. 7 First, we can observe that DIA and MaxCE could get improved additional with proposed method. Second, TePA could obtain improvement under TENT method but slightly degraded under EATA method. It could be that TePA using maximizing entropy as an attack objective, and with the help of the surrogate model, the poisoned data would have very high prediction entropy because the attack reference model is more approximate to the online model, but fail to pass the entropy threshold and class diverse weighting using in EATA. Overall, our proposed in nn-distribttion attack could generalize to most of the existing attacking objectives.
CONCLUSION
In this work, we reviewed the assumptions adopted by existing works for generating poisoned data at test-time and propose few criteria that define more realistic test-time data poisoning. Specifically, we approach from the angles of attack transparency, access to other users' benign data, attack budget, and attack order. To craft realistic poisoned data, proposed grey-box in-distribution attack with attack objective tailored for TTA methods. Through extensive evaluations under the realistic evaluation protocol, reveal that the adversarial risk of TTA method might be over estimated and, importantly, certain practices in TTA methods are empirically proven to be effective and should be considered for designing adversarial robust TTA methods in the future.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
This research work is supported by the National Natural Science Foundation of China (NSFC) (Grant Number: 62106078), the Agency for Science, Technology and Research (A *STA)) under its MTC Programmatic Funds (Grant Number: M23L7b0021) and the Guangdong R&D key project of China (Grant Number: 2019B010155001) This work was done during Yongyi Su's attachment with Institute for Infocomm Research (I2R), funded by China Scholarship Council (CSC).
REFERENCES
Lagrange Multiplier, pp. 292-294. Springer New York, New York, NY, 2008. ISBN 978-0-387- 32833-1. doi: (0.1007/978-00-88-3233333 0-387-32833-1 1_21...
Naveed Akhtar and Ajmal Mian. Threat adversarial attacks on deep learning in computer vision: A survey, Ieee Access, 6: 6:14410-14430, 2018.
Scott Alfeld, Xiaojin Zhu, and Paul Barford. Data poisoning attacks against autoregressive models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Eric Arazo, Diego Ortego, Paul Albert, Noel O' DoConoo,, and Kevin McGuinness. Pseudo-labeling and confirmation bias in deep semi-supervised learning. In Proceedings of 2020 International Joint Conference on Neural Networks, pp. 1-8. IEEE, 2020.
Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. arXiv preprint arXiv: 220. 6389, 2012.
Battista Biggio, Igino Corona, Davide Maiorca, Blaine Nelson, Nedim Šmdić, Pavel Laskov. Giorgio Giacinto, and Fabio Roli. Evasion attacks against machine learning at test time. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases, 2013.
Stephen Boyd and Lieven Vandenberghe. Convex optimization. Cambridge university press, 2004.
Dhanajit Brahma and Piyush Rai. A probabilistic framework for lifelong test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3582 3591, 2023.
Anirban Chakraborty, Manaar Alam, Vishal Dey, Anupam Chattopadhyay, and Debdeep Mukhopad- hyay. Adversarial attacks and defences: A survey. arXiv preprint arXiv: /8/ /8/000066, 2018.
Jiefeng Chen, Xi Wu, Yang Guo, Yingyu Liang, and Somesh Jha. Towards evaluating the robustmess of neural networks learned by transduction. In International Conference on Learning Representations,
2022. URL /ttttp::///perreviiee ew net. forum? _d=___________ rx11
Jinyin Chen, Mengmeng Su, Shijing Shen, Hui Xiong, and Haibin Zheng. Poba-ga: Perturbation optimized black-box adversarial attacks via genetic algorithm. Computers & Security, 2019.
Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM workshop on artificial intelligence and security, pp. 15-26, 2017.
Yijin Chen, Xun Xu, Yongyi Su, and Kui Jia. Stfar: Improving object detection robustness at test-time by self-training with feature alignment regularization. arXiv preprint arXiv:2303. 17937, 2023.
Zhixiang Chi, Li Gu. Tao Zhong, Huan Liu, YUANHAO YU, Konstantinos N Plataniotis, and Yang Wang. Adapting to distribution shift by visual domain prompt generatio.. In The Twelfth International Conference on Learning Representations, 2024.
Tianshuo Cong, Xinlei He, Yun Shen, and Yang Zhang. Test-time poisoning attacks against test-time adaptation models. In IEEE Symposium Security and Privacy, 2023.
Francesco Croce and Matthias Hein. Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-fii free attacks. In International conference on machine learning, pp. 2206-2216 PMLR, 2020.
11

---

Published as a conference paper at ICLR 2025
Mario Döbler, Robert A Marsden, and Bin Yang. Robust mean teacher for continual and gradual test-time adaptation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7704-7714, 2023.
Alexey Dosovitskiy. Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn. Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2021.
Jiaxin Fan, Qi Yan, Mohan Li, Guanqun Qu and Yang Xiao. A survey data poisoning attacks and defenses. In 2022 7th IEEE International Conference on Data Science in Cyherspace (DSC), pp. 48 55. IEEE, 2022.
Liam Fowl, Micah Goldblum, Ping-yeh Chiang, Jonas Geiping, Wojciech Czaja, and Tom Goldstein. Adversarial examples make strong poisons. Advances in Neural Information Processing Systems, 34:30339-30333 2021.
Yunhe Gao, Xingjian Shi, Yi Zhu, Hao Wang, Zhiqiang Tang, Xiong Zhou, Mu Li, and Dimitris N Metaxas. Visual prompt tuning for test-time domain adaptation. arXiv preprint arXiv:2210.04811, 2022.
Taesik Gong, Jongheon Jeong, Taewon Kim, Yewon Kim, Jinwoo Shin, and Sung-Ju Lee. NOTE: Robust continual test-time adaptation against temporal correlation. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022.
lan J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Dan Hendrycks and Thomas Dietterich. Benchmarking neural network robustness to common corruptions and perturbations. arXiv preprint 19XXv:1903.122000 2019.
Hanxun Huang, Xingjun Ma, Sarah Monazam Erfani, James Bailey, and Yisen Wang. Unlearn- able examples: Making personal data unexploitable. In International Conference on Learning Representations, 2021. URL htwpt:ttt//ccm /tttpps://pperrvviiii wtt.:eeeooorrrmm1ii1immmleellll
Andrew Ilyas, Logan Engstrom, Anish Athalye, and Jessy Lin. Black-box adversarial attacks with limited queries and information. In International conference on machine learning, pp. 2137-2146. PMLR, 2018.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Huichen Li, Xiaojun Xu, Xiaolu Zhang, Shuang Yang, and Bo Li. Qeba: Query-efficient boundary- based blackbox attack. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020.
Yushu Li, Xun Xu, Yongyi Su, and Kui Jia. On the robustness of open- world test-time training: Self-training with dynamic prototype expansion. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 11836-1 1846, 2023.
Jian Liang, Dapeng Hu, and Jiashi Feng. Do we really need to access the source data? source hypothesis transfer for unsupervised domain adaptation. In Proceedings of International conference on machine learning, pp. 6039, 2020.
Jian Liang, Ran He, and Tieniu Tan. A comprehensive survey on test-time adaptation under distribu- tion shifts. International Journal of Computer Vision, pp. 1-34, 2024.
12

---

Published as a conference paper at ICLR 2025
Nanqing Liu, Xun Xu, Yongyi Su, Haojie Zhang, and Heng- Beng-Chha Li. Pointsam: Pointly- supervvsed segment anything model for remote sensing images. IEEE Transactions on Geoscience and Remote Sensing, 2025.
Yuejiang Liu, Parth Kothari, Bastien Van Delft, Baptiste Bellot-Gurlet, Taylor Mordan, and Alexandre Alahi. Tit++: When does self--upervised test- time training fail or thrive? Advances in Neural Information Processing Systems, 34:21808-211 180--2882, 2021.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In International Conference Learning Representations, 2018.
Robert A Marsden, Mario Döbler, and Bin Yang. Universal test-time adaptation through weight ensembling, diversity weighting, and prior correction. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 2555-2565, 2024.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang. Yaofo Chen, Shijian Zheng, Peilin Zhao, and Mingkui Tan. Efficient test-time model adaptation without forgetting. In Proceedings of International conference on machine learning, pp. 16888-16905, 2022.
Shuaicheng Niu, Jiaxiang Wu, Yifan Zhang, Zhiquan Wen, Yaofo Chen, Peilin Zhao, and Mingkui Tan. Towards stable test-time adaptation in dynamic wild world. In Proceedings of International Conference on Learning Representations, 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research Journal, pp. 1-31, 2024.
Hyejin Park, Jeongyeon Hwang, Sunung Mun, Sangdon Park, and Jungseul Ok. Medbn: Robust test-time adaptation against malicious test samples. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5997-6007, 2024.
Hao Qiu, Leonardo Lucio Custode, and Giovanni Iacca. Black-box adversarial attacks using evolution strategies. In Proceedings of the Genetic and Evolutionary Computation Conference Companion, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748 8763. PMLR, 2021.
Binxin Ru, Adam Cobb, Arno Blaas, and Yarin Gal. Bayesopt adversarial attack. In International Conference on Learning Representations, 2019.
Evgenia Rusak, Steffen Schneider, George Pachitariu, Luisa Eck, Peter Vincent Gehler, Oliver Bringmann, Wieland Brendel, and Matthias Bethge. If your data distribution shifts, use self- learning. Transactions on Machine Learning Research, 2022.
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted elean-lahel poisoning attacks on neural networks. Advances in neural information processing systems, 31, 2018
Junha Song, Jungsoo Lee, In So Kweon, and Sungha Choi. Ecotta: Memory--fficient continual test-time adaptation via self-distilled regularization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11920-11929, 2023.
Yongyi Su, Xun Xu, and Kui Jia. Revisiting realistic test-time training: Sequential inference and adaptation by anchored clustering. Proceedings of Advances in Neural Information Processing Systems, 35:17543-175555 2022.
Yongyi Su, Xun Xu, and Kui Jia. Towards real-world test-time adaptation: Tri-net self-training with balanced normalization. In Proceedings of the AAAI Conference on Artificial Intelligence, 2024a.
13

---

Published as a conference paper at ICLR 2025
Yongyi Su, Xun Xu, Tianrui Li, and Kui Jia. Revisiting realistic test-time training: Sequential inference and adaptation by anchored clustering regularized self felf-training.. IEEE Transactions on Pattem Analysis and Machine Intelligence, 2024b.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In International Conference on Learning Representations, 2014.
Florian Tramèr, Alexey Kurakin, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick Mc- Daniel. Ensemble adversarial training: Attacks and defenses. In International Conference on Learning Representations, 2018.
Leslie G Valiant. A theory of the learnable. Communications the ACM.
Dequan Wang. Evan Shelhamer, Shaoteng Liu, Bruno Olshausen, and Trevor Darrell. Tent: Fully test-time adaptation by entropy minimization. Proceedings of International Conference on Learning Representations, 2020.
Qin Wang. Olga Fink, Lue Van Gool, and Dengxin Dai. Continual test-time domain adaptation. In Proceedings of the IEEE/CVF Conference Computer Vision and Puttern Recognition, pp. 7201-7211, 2022.
Ziqiang Wang. Zhixiang Chi, Yanan Wu, Li Gu. Zhi Liu, Konstantinos Plataniotis, and Yang Wang. Distribution alignment for fully test-time adaptation with dynamic online data streams. In European Conference Computer Vision, pp. 2223499. Springer, 2025.
Tong Wu, Feiran Jia, Xiangyu Qi, Jiachen T Wang, Vikash Sehwag, Saeed Mahloujifar, and Pra- teek Mittal. Uncovering adversarial risks test-time adaptation. In Proceedings of the 40th International Conference on Machine Learning, 2023.
Yanan Wu, Zhixiang Chi, Yang Wang, Konstantinos N Plataniotis, and Songhe Feng. Test-time domain adaptation by learning domain awwre batch normalization. In Proceedings the AAAI Conference on Artificial Intelligence, volume 38, pp.
2024.
Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1492-1500, 2017.
Ying Xu, Xu Zhong, Antonio Jimeno Yepes, and Jey Han Lau. Grey-box adversarial attack and defence for sentiment classification. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 4078-4087, 2021.
Chaofei Yang, Qing Wu, Hai Li, and Yiran Chen. Generative poisoning attack method against neural networks. arXiv preprint arXiv: 1703.0 703.000333333 33400, 2017. 2017.
Longhui Yuan, Binhui Xie, and Shuang Li. Robust test-time adaptation in dynamic scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2023.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Procedings of the British Machine Vision Conference, 2016.
Marvin Zhang, Sergey Levine, and Chelsea Finn. Memo: Test time robustness via adaptation and augmentation. Advances in neural information processing systems, 35:38629-38642, 2022.
Xuezhou Zhang, Xiaojin Zhu, and Laurent Lessard. Online data poisoning attacks. In Learning for
Dynamics and Control, pp. 201-210. PMLR, 2020.
Tao Zhong, Zhixiang Chi, Li Gu, Yang Wang. Yuanhao Yu, and Jin Tang. Meta-dmoe: Adapting to domain shift by meta-distillation from mixture-of-experts. Advances' in Neural Information Processing Systems, 35:22243-22255 2022.
Zhi Zhou, Lan-Zhe Guo, Lin-Han Jia, Dingchu Zhang, and Yu-Feng Li. Ods: test-time adaptation in the presence of open-world data shift. In Proceedings of International Conference on Machine Learning, pp. 42574 42588. 2023.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
A.1 ADDITIONAL DETAILS FOR METHODOLOGY
A.1.1 ILLUSTRATION TEST-TIME DATA POISONING BATCH SPLIT
We visualize the batch split under realistic test-time data poisoning Fig. present the batch split scheme for both "Uniform" and "Non-Unifori" Uniform" attack frequencies. Under "Uniform" attack frequency, the poisoned minibatch is uniformly presented the test data stream while Aion-Uniform attack protocol simulates the situation the adversary attacks the TTA model in short period of time with huge amount of poisoned data.
Domain
Domain
Domain N
Ba,
Bp,
Ba2
Ba2
Ba,
Bo,3
B.
Brz
Poisoned Subset
(a) Uniform Data Poisoning
Domain
Domain
Domain
Ba.1
Ba
Bas
Bn,
Bb
Bb
Br
B:2
(b) Non-Uniform Data Poisoning
Figure 3: Illustration of test-time data poisoning batch split.
A.1.2 GLOBAL OPTIOMAL LABEL MAPPING
For Balanced Low Entropy Attack, we need to obtain label mapping which mainly addresses the following issues, i) maps the GT label to one wrong label; ii) the label mapping is bijective; iii) the sum of the mapping cost is the minimal. To achieve it, we first define probability confusion between all class pairs as C E [0, 1|K* The probability confusion is updated in exponentially moving average fashion using the current posterior predictions. The label mapping M (0, []jXX X is then obtained by optimizing the following linear assignment problem. Efficient solver, e.g. Hungarian method, can employed to solve this problem, as Eq. 9
ΣΣ Mrg arg max M
K 9
s.l. Ma #1, Mkg 11 M (0, KxK
(9)
c = 3ct + (1 -1-53 (1-8) E Ei,e8, 1(y, k) h()))) Σx,0,, h(x,)
Here, we also provide the pseudo code of the implementation of BLE Attack, as follow,
def attack beecs-oobeetiieeee (self, x, y):
// (B, K) the predicted logit of poisoned data // y: (B, ) the ground true label of poisoned data // return the attack loss value
15

---

Published as a conference paper at ICLR 2025
with torch.no_grace
// EMA update crt_k
curr_prab_term scatter_ment ww..toamax (1), y Nonel, dim-0, aut-ttrch.zeeos_likeee ott prob)) prob)))))
new new_ema_prob = self.class setssswiiee__eeeeeeeeee wttse_mmmeetu__probo.oooo
new new_ema_prob [y. (y.
self o haa...ommtttmnceeefiiiee....
new gew_ema_problyyuriiee self. selff.moomenttmmmoooffffiiiiie
+ (1
curr_prob_termlyyom
new_ema_prob__eeeeeaeaaae prob_select new new_ema_prob.clor
diag_mask =
torch.diag (torch. ones (new hea__ema_aeeeeeeeee hat___ea__rrb_sseee
ones
[hape(0]))
rev_ema_prob_select [diag_mask.bool ()] = 0.
// Find the global optimal mapping M
label gabel_mapping
peroserros (new_ema ema prob lttplpppooeeeppleleeeeeeepeeeeee select. shape (0),
for dtype=torch.
range haa__ema_probsseeecee
[0])
biased_probs biased_class =
- F.normalize (tae select, dim=-1,
p=1) max (dim=-1)
max_item = blased__pobb.rrgmae (aaizgaaxxdimm=->> (dim=-1)
label habellmappinglieee [max item] biased class [max item]
new_ema_pzos sebleettteeeeecttttceeeeeeeeeee_ select [max _itm, :] 0.
don_ema_prob_selet selobb_seleeee
biased_cr cl ass imax_item 0.
// BLE attack loss
loss = F.cross_end accrssseettropy/ (x, label_mapping [y])
self. wwwrent_proottt return loss
www__ema_probbheeee detach
A.1.3 DETAILS OF DEFENSE PRACTICES
Here, we provide the details about the defense practices evaluated in Tab of the main text.
Entropy Thresholding implemented through filtering the entropy below 0.05 log K),
where K is the class number of the dataset.
Data Augmentation performs the data augmentation used into CoTTA over the input samples and constrain the consistent predictions between the augmented samples and the corresponding original samples.
EMA Update module allow us maintain an exponentially moving average updated model to generate robust predictions and used to supervise the online model update, and the EMA momentum 0.999.
Stochastic Parameter Restoration is implemented the module used in CoTTA. We randomly reset the network weights to source model weights with probability and is set 0.01.
A.1.4 DISTINCTION BETWEEN RTTDP AND TEPA
We would like to highlight the key differences between proposed RTTDP and TePA protocols as follows,
TePA (Cong et al. 2023) employs fixed surrogate model before test-time adaptation begins for generating poisoning, which qualifies the method as an offline method. The surrogate model is obtained by training separate model (different architecture from the target model) using the same
16

---

Published as a conference paper at ICLR 2025
source dataset. For example, on TTA for CIFARIO-C, if the target model, i.c. the model deployed for inference and is subject to test-time adaptation, is ResNet18, TePA employs VGG-11 as the surrogate model and trains VGG- 11 on the same source training dataset (CIFAR 10 clean training set). This is evidenced from the source code released by official repository | and the descriptions in TePA "we assume that the adversary has background knowledge of the distribution of the target model's training dataset. This knowledge allows the adversary to construct surrogate model with a similar distribution dataset".
TePA employs the fixed surrogate model to generate poisoned dataset x'. Then generated poisoned dataset is fed to test-time adaptation update model weights. Afterwards, TTA is further conducted on clean testing data for model update and performance evaluation. The segregation of data poisoning and TTA steps further support the claim that TePA should be classified as an offline approach.
Finally, to ensure fair comparison between TePA with our proposed methods under RTTDP protocol, TePA could be adapted online fashion and we made such an adaptation to TePA for comparison in Tab.2 Tab. and Tab, of the main text. Specifically, initial surrogate model and inject the generated poisoning into the testing data stream, i.e. placing poisoning in between benign testing batches. In this way, poisoning will affect TTA in an online fashion. We believe this is the most fair way to compare RTTDP with TePA.
use TePA to generate poisoning against the
A.1.5 MORE ANALYSIS AND DERIVATIONS ABOUT THE OPTIMIZATION OBJECTIVE
Here, we discuss the transition from the original bi-level optimization objective (Eq. 2 to our proposed single-level optimization objective (Eq. with feature consistency constraint.
The original optimization objective for test-time data poisoning is formulated as bi-level optimization problem, as shown below (equivalent the meaning of Eq. 2
B₁ arg min ((,,)))B [Luck (h( (B U B:)), y)] B.
(10)
s.t. (B, U BB) = arg min Lita (h(B, U Bb; 0))
8,
The above optimization involves an inner loop where the model adapts to test samples, including poisoned and benign samples, and an outer loop to optimize the attack objective. This structure computationally intensive and impractical under the constraints of the RTTDP setting. To address this, we provide a detailed step-by-step derivation and explanation below.
1. Discarding the Inner Optimization: In DIA (Wu et al., 2023), the inner optimization is approximated by assuming 0r x 9r. where 07 represents the parameters after a full adaptation step, and 0, represents the current parameters. This approximation is justified TTA models typically update minimally during single minibatch iteration, resulting in minor perturbations to 6. Thus, the approximation retains practical relevance while simplifying the problem. The formula is derived as (this is also the DIA's objective),
B₁ = argming, (i( (h(x;; (Ba U BB))), v)]
(11)
2. Surrogate Model for Online Parameters: In the RTTDP protocol, direct access to online model parameters 9₂ is unrealistic. Instead, we replace 9 with the surrogate model parameters ô,, which accessible and trained to approximate the online model's behavior.
B, = aarggiingg E(r.)) (h(a; (, U (B)))
(12)
3. Removing the access to B&: In the RTTDP protocol, the adversary is prohibited from observing benign users' samples when generating poisoned samples. Consequently. the B, term is excluded from the optimization objective. In the main text, introduce to leverage Bab replace Bb, where Bas represents the adversary benign samples before they are poisoned.
hhaps//gtthhh.commiinshcccooooo
17

---

Published as a conference paper at ICLR 2025
Bu arg ming, Bl (h(: u Bat)). v)
(13)
where (B, Bb) indicates forwarding Bg U Bd to update the BN statistics. This objective would lead to trivial solution that B is effective only for the current Ba data through easily introducing biased normalization in each BN layer, and has little effect while B, and Bab are seperated batch. Therefore, it would waste a half of attack query budget for forwarding these poisoned samples (the benign samples take up half of the batch size).
4. Introducing feature consistency constraint to improve query utilization: In the main text, observed the feature distributions of B, and B and found out that they obviously do not overlap, so we introduced feature consistency constraint to regularize their distributions according to the PAC learning framework in order to merge the two subsets into a single one, and to improve the utilization of the poisoned data query. The final objective is derived as follows, where Pa and Pas the shallow feature distributions of B, and Bab, respectively.
B. = arg mins, (( (b(x;( 0, v)) s.t. D(Pa, Pab) = 0 (14)
To this end, we can fully utilize the budget of all poisoned data query generate poisoned data. The experimental results show that the attack performance of the attack objective will be significantly improved after using this regularization term.
Our proposed objective leverages the dependence of TTA models self-training mechanisms, which aim to maximize confidence on pseudo- labels for adaptation. When the TTA model adapts to poisoned samples, it learns and reinforces incorrect associations. This creates a vulnerability, as future test samples with similar shallow feature distributions are more likely to be misclassified by the online model. Since TTA methods iteratively adapt using incoming test samples, our approach leverages this dependency to propagate the error induced by poisoned samples throughout the adaptation process.
In-distribution Attack Objective from a TTA Perspective:
1.1.6 DETAILED POISONING & TRAINING ALGORITHM
We present the overall algorithm for generating poisoned data and surrogate model update in Alg.
A.2 EXPERIMENTAL SETUPS OF DIA, TEPA AND RTTDP
We revisit the experimental setups of the previous methods, i.c. DIA (Wu et al. (2023) and TePA Cong al. (2023), explain the differences under our RTTDP protocol, and justify the adaptations we made to ensure fair comparisons.
Commonalities among the different protocols: The three protocols, TePA, DIA, and RTTDP, share several overarching goals and assumptions. First, all protocols aim to evaluate the adversarial risks posed to Test-Time Adaptation (TTA) by injecting poisoned samples into the test data stream. Second, all protocols allow the adversary to obtain the source model, since the source model is usually the well-known pre-trained model, e.g., ImageNet pre-trained ResNet, and the open-source foundation model, e.g., DINOv2, SAM.
Key Differences among Protocols: Despite sharing some commonalities, the protocols diverge significantly in their attack setups.
In the TePA protocol, poisoned samples are generated by maximizing the entropy of the adversary's crafted samples with respect to the source model's predictions. These poisoned samples injected into the TTA pipeline before any benign users' samples are processed, simulating an offline attack scenario. However, this approach is unrealistic in real-world settings, where adversaries cannot fully control the sequence of test samples in advance.
In contrast, the DIA protocol generates poisoned samples by optimizing them to maximize the cross-ent cross-entropy loss of benign samples belonging to other users. DIA assumes direct access to the online model's parameters and the ability observe other users" benign samples. Poisoned samples
18

---

Published as a conference paper at ICLR 2025
Algorithm 1: The pipeline of RTTDP
input :A minibatch of testing samples B, Bab U Bb where Bas is the adversary benign subset that is preparing for crafting poisoned data and Bi other users' benign subset. Test-time adaptation model: h(x;0; 0c),
Surrogate model used by adversary: h(:;; 0z).
Attack Objective: Catk
I/ generate attack samples through surrogate model.
B, # / then initialize = Bxx x3, where B |Babl.
initialize A {0),, where L is the number of feature layers.
for i = to 40 do
B, [5;; ā.
+ )
calculate the attack loss: loss: 1 Ei,B, Latk (ft)-
Bal
obtain all feature maps (= before the normalization layers.
calculate the feature consistency regularization term as Eq.
loss? Es.gs, KLD(N(u) [ (W Σ) construct the final optimized objective Eq. 6
= loss, + + Et X -loss/
update the adversarial noise:
=- sign [v.C], where a is PGD attack step size of 0.01.
clamp(z; + c 0, 1) Zí, 24 E B..
update the Ap: À A 0.001 C
// feed into TTA model and obtain the prediction.
V: hx (t), 24 B,
// update the surrogate model if B, # 0.
B. # 0 then
0.,0 = 0,
for j : 1 to ilers do
ff h(a 0c), x.e E B..
pf = h(x; -1)) x E B.. calculate the distillation loss Ldist Eq.
Org P Ir V Goiist
0.41 = 0r,iters.
19

---

Published as a conference paper at ICLR 2025
injected into the TTA pipeline alongside the corresponding benign users' samples. However, this protocol has sigmificant limitations in realistic settings. In practice, adversaries typically lack access to or control over the online model's 's parameters. Additionally, it is highly improbable for adversaries to observe henign users' samples, let alone the validation samples required for optimizing poisoning objectives.
The proposed RTTDP protocol addresses these limitations by operating under more realistic assump- tions. In RTTDP, the adversary neither has access to other users' benign samples nor the parameters of the online model. Instead, RTTDP employs a surrogate model, initialized as the source model, to generate poisoned samples. This surrogate model is iteratively updated based on feedback from previously injected poisoned samples. Poisoned samples are then injected into the TTA pipeline, either uniformly or non-unifornly, depending on the attack frequency in RTTDP protocol.
A Adapaations of Competing Methods to RTTDP protocol: To ensure fair comparisons under RTTDP protocol, we made the following adjustments the competing methods:
For TePA method, we preserved TePA's original poisoning objective, i.e. maximizing entropy, but adapted the poisoned data injection strategy from an offline manner to an online manner, i.e. placing poisoning in between benign testing batches according to RTTDP.
For DIA method, (1) Replacing Online Model Parameters: DIA's original objective relies on online model parameters, which are inaccessible in RTTDP. We replaced these parameters with the initial surrogate model, i.e. source model. (2) No Access to Benign Users' Samples for Optimization: DIA uses benign users' samples optimization targets in its original setup. To meet RTTDP's s constraints, we split Bas into two equal subsets, 5% and 8% with a 1:1 ratio. We then generate poisoned samples B by maximizing the cross-entropy loss of 8% The specific formula can be found in Eq. 15
A.3 ADDITIONAL DETAILS FOR EXPERIMENT
Benchmark TTA Methods: We investigate several state-of-the-ar art TTA methods under our RTTDP protocol to evaluate their adversarial robustness. Source serves as the baseline for inference perfor- mance without adaptation. TENT (Wang et al.) 2020) updates BN parameters through minimizing entropy. RPL (Rusak et al. 2022) performs self-training with generalized cross-entropy (GCE) loss, which aids in more robust adaptation under label noise. FATA (Niu et al., 2022) minimizes entropy with the Fisher regularization term to prevent forgetting knowledge from the source do- main. TTAC (Su et al.) 2022) adapts all backbone parameters by jointly optimizing global and class-wise distribution all alignment with the source distribution. SAR (Niu et al. 2023) updates BN parameters with sharpness-aware optimizer to filter out noisy labels and help escape local minima. COTTA (Wang et al. (022) leverages a teacher-student structure, optimizing all parameters of the student model and updating the teacher model using exponential moving average. To better prevent forgetting during continual adaptation, it incorporates parameter random resetting and data augmentation methods. ROID (Marsden et al., (024)) updates BN parameters with loss of self-label refinement (SLR) weighed by certainty and diversity while continually weighting the online model and the source model to prevent forgetting.
Evaluation Protocol:
We devise a evaluation plan respecting the realistic test-time data poisoning criteria. First, we investigate the frequency of injecting poisoned data. The "Uniform" scheme indicates that the poisoned minibatch is uniformly present in the test data stream, simulating the scenario that the adversary is periodically injecting the poisoned data. We further evaluate "Non- Uniform" scheme by allowing the adversary to concentrate the attack budget within short period of time. We fix the overall attack budget as r [8. throughout the experiments. We report the attack success rate as the evaluation metric, which is measured as the percentage of misclassified benign samples in the benign subset B,. Additionally, for casier comparison, we also calculate the average error rate (higher the better) and the average ranking (lower the better) for each poisoning method.
B.1
Hyperparameters: For all competing methods, we employ the 40 steps Loe PGD attack to generate poisoned data. The maximum perturbation budget is 0.3 and the attack step size is 0.01. Additionally, within each PGD iteration, we update x via X; = x + 0.001 V,,C for Eq. where À is initialized as zero before PGD attack. Unless otherwise noted, the overall attack budget is 50% throughout the
20

---

Published as a conference paper at ICLR 2025
experiment. For surrogate model distillation module, adopt SGD optimizer with 0.1 learning rate for 10 iterations to update the surrogate model in each update stage.
Implementation Details: For fair comparison, implement various data poisoning methods within a unified poisoning framework. This framework utilizes a 40-step Projected Gradient Descent (PGD Boyd & Vandenberghe (2004)) optimization process tailored to the respective objectives of each method. The poisoned samples generated are then injected into the TTA pipeline in an online manner, adhering to the RTTDP protocol. Specifically, the respective objectives of different competing poisoning methods are shown as follows,
* DIA (Wu et al. 2023
CrosEEnrrppLLoss((i: (B)) w)
B
arg min E (1.))
(15)
TePA (Cong al. )023):
Bg arg min ( B.
(16)
MaxCE (Madry et al. 2018):
B, = arg min BB B.
(17)
Unlearnable Examples (Huag et al.
2021
B₂ arg min E (a,)),,
(CrssEEntroyyLLos(httt (m)))
(18)
6,
Adversarial Poisoning FFowl et al. 2021
B, = arg min (a,,)) (hooseEntroyyLoos((hh: (B( where ÿ (v + 1)%K.
(19)
NHE Attack (Ours):
B, = arg min max E (a, [Cark (B.)) + ACreg]
(20)
8,
BLE Attack (Ours):
B, arg min max B(aa)EB. Lask (): (B:)) + Xrrea
(21)
where ig indicates the initial surrogate model parameters, Onnit indicates the randomly initialized parameters and K is the number of the category in the dataset. Since under the RTTDP protocol the real-time model parameters are unavailable to access, we leverage the initial surrogate model, replace the online model they might have used in their original paper, as the threat model for the competing methods. The surrogate model is initialized as source model. For our proposed methods, we employ the proposed surrogate model distillation module update the surrogate model, and during each PGD iteration, the variables B,, and À are updated simultaneously using gradient descent for B,, and gradient ascent for X, respectively. More details about our proposed methods can be found in Alg. T
A.4 ADDITIONAL EMPIRICAL ANALYSIS
A.4.1 COMPARISON WITH ADVERSARIAL ATTACK METHODS
Adversarial attack methods are designed to generate perturbations on input samples to mislead the model into making incorrect predictions. Here, we aim to investigate whether the adversarial effects of poisoned samples, generated using advanced adversarial attack methods (Madry al.) 2018] Croce Hein! 2020; Chen et al. 2022), can effectively transferred to benign users" samples under the RTTDP protocol.
21

---

Published as a conference paper at ICLR 2025
We conduct the experiments on CIFARIO-C and ImageNet-C datasets with a Uniform attack frequency under our proposed RTTDP protocol. The results are shown in Tab. 8 and Tab. [ We make the following observations. (i) The objectives of adversarial attack methods and data poisoning methods differ fundamentally. Adversarial attack methods focus on generating adversarial noise to mislead model predictions the perturbed test samples. In contrast, data poisoning methods aim to inject carefully crafted poisoned samples to degrade the model's ' performance on subsequent benign samples after adaptation. (ii) While AutoAttack (Croce & Hein 2020) represents more advanced adversarial attack method, its performance is inferior to that of MaxCE-IGD (Madry et al. 2018)) on ImageNet-C. (iii) Furthermore, certain complex adversarial attack methods, such as GMSA-MIN and GMSA- AVG (Chen et al.) 2022), require generating adversarial perturbations separately for each class, process that incurs substantial computational costs and limits scalability (which why these methods are excluded from comparison on ImageNet-C), and still fall short compared to the efficacy of our proposed data poisoning methods.
Table 8: Comparison with different adversarial attack methods with our proposed data poisoning methods on CIFARIO-C dataset under the RTTDP protocol.
Attack Objective
TENT
EATA
SAR
ROID
Avg
NoAttack
19.72
18.03
18.94
16.37
18.27
MaxCE-PGD (Madry et al.
2018
18.55
18.17
19.50
18.57
18.70
AutoAttack (Croce & Hei
2020
26.29
19.12
19.56
18.67
20.91
GMSA-MIN (Chen et al.
2022
35.92
22.78
19.99
18.65
24.33
GMSA-AVG
(Chen et al.
2022
38.80
21.89
19.95
18.51
24.79
BLE Attack (Ours) NHE Attack (Ours)
54.07
45.20
26.80
19.06
36.28
73.86
29.73
24.56
17.00
36.29
Table 9: Comparison with different adversarial attack methods with our proposed data poisoning methods on ImageNet-C dataset under the RTTDP protocol.
Attack Objective
TENT
SAR
CoTTA
ROID
Avg
NoAttack
63.49
61.26
63.02
53.42
60.30
MaxCE-PGD (Madry al.
2018)
62.64
61.66
68.83
59.89
63.26
AutoAttack (Croce & Hei
2020)
64.48
61.42
63.03
54.78
60.93
BLE Attack (Ours)
68.04
64.31
66.40
57.10
63.96
NHE Attack (Ours)
78.03
72.58
66.40
57.72
68.68
A.4.2 ABLATION STUDY ON QUERY COUNTS
Regarding varying query attempts, we add additional evaluation follows. Nonetheless, want to highlight that the query attempts do not have to be limited for our method because all queries are submitted to the surrogate model rather than the online model. More queries simply makes generating poisoning slower. In this study, we vary the query steps from 10 to 60 for the projected gradient descent optimization (Boyd & Vandenberghe 2004). We evaluate varying attack query counts for two TTA methods under their respective strongest attack objectives. The results in the Tab. ar obtained on CIFARIO-C dataset with Uniform attack frequency. We make the following observations. (i) Increasing the number of queries could improve the performance at low query budget. (ii) When the budget increased to beyond 40 queries, the performance saturates. We draw the conclusion that allowing sufficient queries to the surrogate model is necessary for generating effective data poisoning, and, importantly, this procedure will not create alert to the online model.
A.4.3 ANALYSIS ON SYMMETRIC KLD USED FOR DISTILLING SURROGATE MODEL
In this work, we adopt the common practice of symmetrizing the Kullback- Leibler Divergence (KLD) to ensure balanced alignment between distributions in the surrogate model distillation. Following the
22

---

Published as a conference paper at ICLR 2025
Table 10: The ablation study on query counts. These results obtained CIFAR10-C dataset with a Uniform attack frequency under RTTDP protocol. We choose 40 queries throughout the experiments.
TTA Method
10
20
30
40
50
60
TENT (NHE Attack)
66.95
74.36
74.34
73.86
73.51
73.66
EATA (BLE Attack)
35.73
39.70
42.36
45.20
45.99
45.89
definitions provided in the main text, the forward KLD is expressed as (LD(h(a;;0))) 0.)). while the reverse KLD is defined as (h ( (zxy; 0t)).
Forward KLD emphasizes penalizing discrepancies where the distilled (surrogate) model Ú, assigns low probability to samples that the source (real-time target) model deems important. encourages the distilled model to mimic the behavior of the target model by focusing on areas of high confidence in 0's posterior.
Reverse KLD, in contrast, focuses on matching I's predictions where á assigns high probabilities. This can result in sharper, more focused distributions but might dismiss less probable regions of l's posterior.
The symmetric KLD balances the above two objectives. The forward KL.D may be more suitable when surrogate model is significantly smaller than the target model and the objective is to allow the surrogate model to mimic the target model's certainty. When the surrogate model is of the same capacity with target model, using the symmetric KLD may better align the two models in both high confident and low confident predictions. this work, the capacity of surrogate is similar to target model. Thus, we hypothesize that the symmetric KLD could be better.
We further use empirical observations in the Tab. II below to support the hypothesis. With symmetric KLD the performance is slightly better than using the forward KLD.
Table 11: Comparison between Symmetric KLD and Forward KLD used for surrogate model distillation. These results are obtained on CIFARIO-C dataset with Uniform attack frequency under RTTDP protocol.
TTA Method
Symmetric KLD KLD(h(;;; (h),
TENT (NHE Attack)
73.86
74.35
EATA (BLE Attack)
45.20
43.99
SAR (BLE Attack)
26.80
26.35
Nevertheless, we do acknowledge that both symmetric KLD and forward KLD give competitive results. The choice depends on computation affordability and empirical observations.
A.4.4 DIFFERENT ATTACK BUDGETS
We further evaluate the effectiveness of proposed poisoning approach under different attack budgets. Specifically, evaluated at 0.1, r 0.2 and r = 0.5. We clearly observe that both high entropy and low entropy attacks are effective regardless of attack budgets.
A.4.5 VISUALIZATION OF POISONED SAMPLES
We visualize selected samples before and after test-time data poisoning in Fig. The high corruption level makes the adversarial noise less noticeable, suggesting the poisoned data could even evade human inspection.
23

---

Published as a conference paper at ICLR 2025
Table 12: Comparing the attack performance of test-time data poisoning under different attack budgets.
Attack Obj.
0.1
0.2
0.5
No Attack
20.72
20.39
19.72
TENT
BLE Attack
22.44
27.60
54.07
NHE Attack
39.20
62.19
73.86
No Attack 17.99
17.76
18.03
EATA
BLE Attack
22.20
28.29
45.20
NHE Attack
19.59
20.10
29.73
No Attack 18.95
18.90
18.94
SAR
BLE Attack
19,90
21.30
26,80
NHE Attack
19.33
20.74
24.56
24

---

Published as a conference paper at ICLR 2025
Poisoned
Brightness
Benign
Poisoned
Defocus Blur
Benign
Poisoned
Gaussian Noise
Benign
Poisoned
Snow
Benign
Poisoned
Pixelate
Benign
Figure 4: Visualizing of selected samples before and after test-time data poisoning.
25


Reviews:
[{"review_id": "4mrR52eqsW", "reviewer": "Reviewer_8xpV", "summary": "This paper investigates adversarial risks in test-time adaptation (TTA), which updates model weights during inference to counter distribution shifts. The authors argue that the assumed threat model of prior work is unrealistic and gives the attacker too much access. They propose a more restricted attack model called Realistic Test-Time Data Poisoning (RTTDP), assuming grey-box access to the model and no access to the benign samples of other users. The authors propose a new attack method outperforming previous work in this new setting.", "questions": "**Questions**\n\nL181: Why is repeated querying prohibited? I understand that the model is constantly updated throughout the querying, but is this really a problem?\n\nSection 3.1: How exactly is the model distilled? L181 says repeated querying is prohibited, but my understanding is that training a surrogate model requires precisely such querying? How can you query for a single $\\theta_t$ if it changes by the very fact that it is being queried?\n\nL232/233: How can we assume to know $\\mathcal{B}_{a, t-1}$? Other users could have queried the model an unknown number of times between training the surrogate and launching the attack.\n\nTable 2-4: What do “Uniform” and “Non-Uniform” Attack Freq. refer to? I could not find this mentioned anywhere in the paper. The appendix has a figure, but also no sufficient explanation. Since this is a central part of the main results tables, it should be carefully explained in the main part of the paper.\n\nTable 2-4: What does “Source” refer to? this should also be explained.\n\nL259/260: I don’t understand what this sentence means. What is “the optimized”, what is the “optimizing objective”, what is the “problem”?\n\nL431: I do not understand this sentence. Can you please rephrase?\n\nL149/150: “the attacker is assumed to have no access to real-time model weights.” By whom is this assumed?\n\nTable 1: what does “Offline” attack order refer to?\n\n**Suggestions for Improvement**\n\nFigure 1: The overview figure 1 gives over the method is great! I appreciate the difficulty in illustrating the complex system with many aspects. I would like to make some suggestions to simplify the figure, as it took me a long time to work through it, and I believe it would be much more helpful with slightly less detail that distracts from the core parts:\n\n- I suggest entirely removing the two boxes “TTA Model” and “Attack Objectives”. They don’t add much information over the text, and it would allow the viewer to concentrate on the three parties in the system: the adversary, the benign user, and the TTA server.\n- Removing the symbols for the attacker and user would further reduce visual clutter, and they are redundant to the titles of the boxes.\n- I would stylize the images from the distributions - e.g. squares of one color per distribution - instead of using actual images from the dataset. This has several advantages: (i) it reduces visual clutter, (ii) it makes it immediately apparent which distribution they belong to, (iii) the “poison” symbol becomes more apparent (it took me a while to see the little devils).\n\nThere may be more opportunities for improvements, e.g., thicker lines for arrows, but I believe the three points above will already make the figure significantly easier to understand.\n\nFigure 2: Please consider breaking these plots into individual figures.\n\n- They are currently way too small, to the point where labels and points are barely readable when printed. At least doubling their size would be required.\n- The several subplots are unrelated and even referenced from different sections of the paper. It would be much more helpful to have the figure close to the text that it refers to.\n\nL298: You refer to Fig 2(c), which contains many acronyms that have not been introduced. E.g., TENT, EATA, SAR, NHE, BLE, …\n\nTables 5-7: These tables would be easier to read without the grey shading.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**More Realistic Threat-Model**\n\nThe paper critically examines the assumptions made in prior work regarding the threat model of poisoning attacks against TTA systems, particularly regarding the adversary’s capabilities. In particular, the relaxation of (i) white-box access to gray-box access and (ii) restricting access to the targeted samples makes the setting much more realistic and, therefore, the findings more significant and relevant.\n\n**A New Attack Rooted in Theoretical Insights**\n\nThe proposed method cleverly extends on previous attacks, making it effective in the new, more restricted attack setting. The method is rooted in theoretical insights that nicely explain the various parts and modifications. It demonstrates that TTA is still vulnerable to poisoning attacks even with more restrictive assumptions.\n\n**Thorough Evaluation**\n\nThe experimental evaluation is thorough. It spans three different datasets with increasing complexity (CIFAR10-C, CIFAR1-C, ImageNet-C), multiple different TTA methods, and compares against two different baselines. The evaluation also includes an ablation study, as well as first look at potential defenses, which I appreciate. The defenses show promise but are unable to fully recover the functions of the original models.", "weaknesses": "**Improved Assumptions Still Strong**\n\nWhile the gray-box assumption is a significant improvement over the white-box assumption in some prior work, it still assumes the adversary's access to the original model (before TTA) and data from the same distribution as the victim benign user. Both of these assumptions are fairly strong and could ideally be relaxed further to, e.g., black-box access.\n\n**Unclear Details**\n\nSeveral important details of the method remain unclear and should be included in the paper. The most important ones include: why repeated querying of the model is forbidden (L181), how the surrogate model can be trained without querying (Section 3.1), why we can assume to know $\\mathcal{B}_{a, t-1}$, and how “Uniform” and “Non-Uniform” attacks are executed. Please refer to “Questions” for more details and additional comments.\n\n**The Presentation Can Be Improved**\n\nThe method’s presentation could be improved in multiple ways:\n\n- Figure 1 gives a good overview of the method. It could benefit from some simplifications to reduce visual clutter; see “Suggestions for Improvements” for detailed suggestions.\n- Figure 2 is very small and should be enlarged. The four subplots are also unrelated and referred to from different parts of the paper. I suggest splitting it into individual figures and moving those to the corresponding sections.\n- In general, the paper’s English is easy to read. However, some parts are hard to understand due to grammatically wrong or overly complex sentences (e.g. L431, L259, …). The paper would benefit significantly from a careful revision, possibly with an English language tool.\n\n**No Code Available**\n\nNo code was available for review, and the authors did not specify whether it will be released upon publication.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gPalBi00Lc", "reviewer": "Reviewer_f4Wt", "summary": "This paper examines the adversarial risks in test-time adaptation (TTA), highlighting that TTA’s exposure to test-time data poisoning may be less severe under realistic attack assumptions. The authors propose two TTA-aware attack objectives and a new in-distribution poisoning method that operates without access to benign data, revealing that TTA methods show greater robustness than expected and identifying defense strategies to enhance TTA’s adversarial resilience.", "questions": "1. Has the author investigate the effect of the ratio of between the adversarial example and the benign examples? \n2. Did the author experiment and compare with harder attack like in [1] that requires access to benign examples. Or is there quantitative measurement on the tradeoff when relaxing to realistic attacks. \n3. Is it possible to derive any formal guarantee on the attack effectiveness when relaxing the different constraints?\n\n\n\n[1]. Chen, J., Wu, X., Guo, Y., Liang, Y., and Jha, S. Towards evaluating the robustness of neural networks learned by transduction. In Int. Conf. Learn. Represent., 2022", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper is overall well-written with clear math definitions and extensive experiments and studies an important problem of data poisoning under weaker/realistic assumptions. \n2. The paper studies the popular test-time attack problem from a novel perspective of weaker threat model.", "weaknesses": "1. Line 680, citation format wrong. Missing conference name.\n2. Figure 1 is helpful but visually confusing and ovewhelming. Please explain what B_a, B_ab, B_t, B_b in the figure description section. A term (-\\lambda L_{reg} and L_{atk}) is confusing to put with the full equation . Also, it is better to present the attack objective in separate part than to fit it in Figure 1.\n3. It will be more helpful to have a graph that assign the current popular attack methods into different buckets, where each bucket has different threat model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "xSY9N2kdA8", "reviewer": "Reviewer_AP94", "summary": "The paper deals with data poisoning in the test time adaptation setting. The paper exposes some issues with existing poisoning attacks in this setting, and goes on to propose certain alterations/additions to existing poisoning objectives, and evaluates results on several datasets, with several test time adaptation methods.", "questions": "* Does this only work against losses $\\mathcal{L}_{tta}$ that are unsupervised? It would be nice to explain this a bit more and give some examples of what this loss function looks like in the main body. \n* Do you ever specify the attack budget? I couldn't find it in the paper. In A.2, you define this quantity, $r$, but I don't ever see details for it. Is this the same thing as $b$ in Table 8? \n* What are the \"Source\" numbers listed in Tables 2,3,4? Is this just the success of standard adversarial attacks?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "* The experimentation is thorough, and generally well presented. \n* The authors do a great job of pointing out issues with existing poisoning attacks against TTA training. \n* The authors motivate and propose a poisoning scheme of their own, and show its effectiveness in several settings.", "weaknesses": "* The area of data poisoning in the TTA setting is a bit niche, and to be honest, doesn't seem to present many challenges beyond existing data poisoning settings. So called \"availability\" attacks have been introduced in [1,2], and several other works. These should at least be cited, and probably compared against as there's a decent amount of overlap. \n* The real \"value add\" for the authors, in my view, is the addition of the feature clustering regularizer. The other contributions (BLE, notch loss, etc.) seem to be very slight modifications of existing poisoning attacks, or even just existing adversarial attacks. \n* The related work shouldn't be at the end. It would be very helpful for readers to have some more info on TTA at the beginning of the work. But if you're going to keep it at the end, you NEED to cite things earlier, as it's totally unclear what things like TENT, RPL, etc. are when they're introduced. It seems like in section 4.2, only the acronyms are introduced, with no explanation, and no citation to click on and find more information. Note: I didn't deduct any \"points\" for this weakness, but it really needs to be addressed. \n\n[1] Huang, Hanxun, et al. \"Unlearnable examples: Making personal data unexploitable.\" arXiv preprint arXiv:2101.04898 (2021).\n\n[2] Fowl, Liam, et al. \"Adversarial examples make strong poisons.\" Advances in Neural Information Processing Systems 34 (2021): 30339-30351.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1UN4i6vSU2", "reviewer": "Reviewer_hmy9", "summary": "This paper explores test-time data poisoning (TTDP) under realistic conditions, proposing a grey-box attack that reflects practical scenarios where adversaries lack full access to benign data and real-time model updates. The study introduces two attack objectives based on entropy manipulation to effectively poison data while remaining within realistic constraints. Through extensive experimentation on state-of-the-art test-time adaptation (TTA) methods, the authors assess the vulnerability of these methods and propose feature consistency regularization as a countermeasure. This work highlights the persistent adversarial risks in TTA setups under real-world scenarios.", "questions": "How does your approach transition from a bilevel optimization problem to a single-level one, and could you provide additional theoretical support for this methodology?\n\nCan you elaborate on how your benchmarking setup differs from prior works and how these design choices may influence your results?\n\nGiven that MaxCE is primarily an adversarial example attack, what was the rationale behind including it in your comparisons, and have you considered additional comparisons with more advanced adversarial example attacks like PGD or AutoAttack?", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The study is well-written;\n- It covers an interesting setting that deserve much more attention from the security aspect.", "weaknesses": "While the study is well-written and formally structured, there are several areas where clarity and methodological rigor could be improved.\n\n- Distinction between RTTDP and TePA in Table 1. In Table 1, the difference between RTTDP and TePA, particularly in terms of online adaptability, is not clear. It would be beneficial to clarify whether TePA is prevented from online applicability, thus highlighting RTTDP’s novelty.\n\n- Inconsistent comparison between poisoning and adversarial attacks. The paper compares the proposed poisoning attack with MaxCE, an adversarial example attack applied at test time rather than during training. This inclusion creates confusion about the setting being considered, especially as MaxCE’s performance is close to that of the proposed attacks, despite being a simpler, suboptimal approach. Additionally, this inconsistency leads to general confusion throughout the paper regarding the exact threat model, setting, and practical cases under consideration. The title’s emphasis on “Realistic” test-time data poisoning also adds to this confusion, as data poisoning traditionally pertains to training rather than test inference. To clarify, the authors should explain that, while poisoning data are collected during test inference, they are subsequently used to update the model through adaptation techniques, which occurs as an offline process. Better descriptions and visualizations of the threat model, attack setting, and relevant practical scenarios would improve clarity and accurately convey the scope and applicability of the proposed approach.\n\n- Not impactful results. MaxCE’s performance, although designed as an adversarial example attack rather than a poisoning attack, remains close to that of the proposed attacks, which raises questions about the actual impact and effectiveness of the proposed methods. The results do not demonstrate a clear or significant advantage of the new attack over MaxCE, suggesting that the proposed methods may lack the robustness or advancement intended. Furthermore, more sophisticated adversarial example attacks (e.g., PGD, C&W, and AutoAttack). I recommend including comparisons with these advanced adversarial attacks to provide a clearer perspective on whether the proposed approach offers meaningful improvements. Currently, the lack of a notable performance gap reduces the impact of the results, leaving it uncertain whether the proposed attacks represent a substantial advancement.\n\n- Unsupported and unclear methodology. The transition from a bilevel optimization formulation to a single-level optimization problem lacks theoretical clarity. Such transitions are complex, and previous research has shown limitations in achieving optimal results through this method. Expanding on this approach and its theoretical basis would improve the rigor of the paper’s claims.\n\n- Confusing experimental setup. The design choices in the \"Benchmarking Poisoning Methods\" section lack clarity. The authors do not specify how these choices impact prior works or how they differ from established benchmarking methods, such as DIA and TePA. This lack of clarity makes it difficult to understand the setup and its implications for reproducibility.\n\n- Missing details on attack queries. The number of queries used by each attack is a critical factor, yet the experimental comparisons lack this information. Since query efficiency is a key point raised by the authors, detailing query usage across attacks would be essential for a fair comparison.\n\n\n**Minor Points:**\n\n- In Equation 1, the authors include two Kullback-Leibler Divergence (KLD) terms. To clarify their necessity, the authors should explain why both terms are required rather than just one, even if asymmetry in KLD is a factor. Providing justification in the text would enhance methodological clarity.\n\n- I suggest the authors break Equation 2 into multiple lines to improve readability.\n\n- Expanding the background discussion presented in Section 2 and related work with further background knowledge on adversarial examples, and on white-box, grey-box, and black-box threat models in the context of poisoning. The authors could reference relevant surveys that would help to clarify their threat model and position at the SoA and facilitate non-expert readers understand these distinctions. \n\n- Figure 1 is overly technical and lacks clarity regarding the threat model and attacker strategy. Simplifying it could enhance its value as a conceptual overview.\n\n\n**References**\n- [AutoAttack] Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks.\n- [PGD] Towards deep learning models resistant to adversarial attacks\n- [C&W] Towards evaluating the robustness of neural networks", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)