On the Adversarial Risk of Test Time Adaptation: An Investigation into Realistic Test-Time Data Poisoning

## Abstract

Abstract Test-time adaptation (TTA) updates the model weights during the inference stage using testing data to enhance generalization. However, this practice exposes TTA to adversarial risks. Existing studies have shown that when TTA is updated with crafted adversarial test samples, also known as test-time poisoned data, the performance on benign samples can deteriorate. Nonetheless, the perceived adversarial risk may be overstated if the poisoned data is generated under overly strong assumptions. In this work, we first review realistic assumptions for test-time data poisoning, including white-box versus grey-box attacks, access to benign data, attack budget, and more. We then propose an effective and realistic attack method that better produces poisoned samples without access to benign samples, and derive an effective in-distribution attack objective. We also design two TTA-aware attack objectives. Our benchmarks of existing attack methods reveal that the TTA methods are more robust than previously believed. In addition, we analyze effective defense strategies to help develop adversarially robust TTA methods.

## 1 Introduction

Test-time adaptation (TTA) emerges as an effective measure to counter distribution shift at inference stage (Wang et al., ; Liu et al., ; Su et al., ; Song et al., ). Successful TTA methods leverage the testing data samples for self-training (Wang et al., ; Su et al., ), distribution alignment Su et al. (); Liu et al. () or prompt tuning (Gao et al., ). Despite the continuing efforts into developing computation efficient and high caliber TTA approaches, the robustness of TTA methods has not picked up until recently, leading to studies examining the robustness of TTA methods under constant distribution shift (Song et al., ), correlated testing data stream (Su et al., ; Niu et al., ), open-world testing data (Li et al., ), adversarial robustness (Wu et al., ; Cong et al., ), etc. Among these risks, adversarial vulnerability warrants particular attention due to its potential for evading human inspection and the significant consequences of admitting malicious samples during TTA.

Existing research frames the adversarial risk of Test-Time Adaptation (TTA) as the crafting of poisoned testing data, resulting in models updated with such data performing poorly on clean testing samples (Wu et al., ; Cong et al., ). Consequently, this task is also referred to as Test-Time Data Poisoning (TTDP). The pioneering work DIA (Wu et al., ) introduced a poisoning approach by crafting malicious data with access to all benign samples within a minibatch, leveraging real-time model weights for explicit gradient computing, i.e., a white-box attack. Another concurrent study (Cong et al., ) implements poisoning by preemptively injecting all poisoned data to attack the model even before TTA starts. While these explorations conclude that TTA methods are susceptible to poisoned data, evaluations based on unrealistic assumptions may exaggerate the adversarial risk for several reasons. i) Access to real-time model weights (white-box attack) is often considered overly optimistic, especially given that models are constantly updated during adaptation. Therefore, a grey-box or even black-box attack is preferred for TTDP. ii) The adversary is typically assumed only to be aware of the query samples submitted by themselves. Thus, benign samples submitted by other users should not be utilized for crafting poisoned data, for instance, through bi-level optimization (Wu et al., ). iii) Crafting poisoned data requires querying the model with testing samples. Repeatedly querying the model from a single user could easily trigger alerts in defensive systems. Therefore, any query sample, whether adversarial or benign, should be counted towards the attack budget.
iv) Following the above concern, the adversary should not monopolize the entire testing bandwidth. This constraint translates to a scenario where poisoned data only partially occupies a minibatch, and the adversary is not allowed to inject all poisoned data at once even before TTA starts (Cong et al., ).

To the best of our knowledge, existing attempts at test-time poisoning have not fully addressed the above realistic concerns. In this work, we aim to propose a threat model that advances towards more Realistic Test-Time Data Poisoning (RTTDP). Firstly, we formulate the threat model under a grey-box attack scenario, where initial model weights are visible. We distill a simple surrogate model from the online model using only the adversary’s queries, enabling efficient gradient-based synthesis of poisoned data. Empirical analysis demonstrates that the distilled surrogate provides sufficient information for crafting effective poisoned data. Moreover, to constrain the attack budget, we reformulate the bi-level optimization objective proposed in prior work (Wu et al., ) by replacing benign samples with poisoned data only. Through reasoning on generalization error, we illustrate that the attack loss defined on poisoned data can be generalized to benign samples if the distributions between poisoned and benign samples are identical. This insight motivates us to introduce a feature distribution consistency regularization for in-distribution attacks, eliminating the need for additional benign samples to construct the outer objective. Finally, we devise two alternative attack objectives tailored to the unique features of Test-Time Adaptation (TTA) methods. We first propose a high-entropy oriented attack to generate poisoned samples biased towards high entropy. This approach proves effective in compromising TTA methods based on entropy minimization (Wang et al., ). However, high-entropy attacks may become less effective with the introduction of simple defense techniques, such as confidence thresholding. Therefore, we explore a low-entropy based attack objective aimed at attacking towards a non-ground-truth class. The combined threat model is applied to a diverse range of TTA methods, resulting in more effective outcomes compared to existing threat models. An overview of the overall framework is presented in Figure 1.

In addition to crafting effective threat models, we delve into exploring practices conducive to enhancing Test-Time Adaptation (TTA)’s adversarial robustness. Contrary to the reliance on adversarially trained models (Madry et al., ) and robust batch normalization estimation (Wu et al., ), we draw inspiration from empirical observations of robust TTA methods. Our validation reveals that confidence thresholding, data augmentation, exponential moving averaging (EMA), and random parameter restoration represent potential directions for improving the adversarial robustness of TTA methods.

We summarize the contributions of this work as follows.

- •
We argue the unrealistic assumptions, e.g. white-box attack and overusing attack budget, made in existing attempts at TTDP may overestimate the adversarial risk of TTA methods. To address this, we first propose key criteria for defining realistic test-time data poisoning scenarios.
- •
Under our proposed realistic test-time protocol, we analyze the generalization error and introduce an in-distribution attack strategy with feature distribution consistency regularization. This strategy eliminates the need for additional benign samples in evaluating the outer objective. Additionally, we tailor attack objectives specifically for TTA methods, resulting in more effective poisoning.
- •
We conduct extensive evaluations on state-of-the-art TTA methods, demonstrating the efficacy of our proposed in-distribution attack strategy. Furthermore, we identify certain practices that are conducive to improving realistic adversarial robustness.

Figure: Figure 1: Illustration of the proposed test-time data poisoning pipeline. Adversary generates poisoned data by attacking a regularized objective without accessing benign samples from other users. Model is attacked when carrying out TTA on testing data stream mixed with benign and poisoned data.
Refer to caption: x1.png

## 2 Setting: Realistic Test-time Data Poisoning

### 2.1 Overview of Test-Time Data Poisoning

We first provide a generic overview of test-time adaptation for a $K$-way classification task. We denote the testing data as $\mathcal{D}=\{x_{i}\}_{i=1}^{N_{t}}$, a pre-trained model as $\theta_{0}$. TTA methods often employ unsupervised loss, $\mathcal{L}_{tta}$, to update model parameters upon observing a minibatch of testing samples $\mathcal{B}_{t}=\{x_{i}\}_{i=1}^{N_{b}}$ at timestamp $t$. Usually, a subset of model parameters $\theta_{t}^{u}\subseteq\theta_{t}$ is subject to update at timestamp $t$, and we denote the BN statistics as $\theta^{b}(\mathcal{B}_{t})=\{\mu(\mathcal{B}_{t}),\sigma^{2}(\mathcal{B}_{t})\}$ and the frozen parameters as $\theta^{f}$. The posterior of the sample $x_{i}$ towards the model parameter $\theta$ is $h(x_{i};\theta)\in[0,1]^{K}$. The adversarial risk arises when a subset of testing samples are poisoned, e.g. through adding an adversarial noise $\tilde{x}=x+\epsilon,\;s.t.\;||\epsilon||_{\infty}\leq b$. The model trained on poisoned data exhibit poor performance on clean/benign testing samples. In a typical TTA scenario, the online model is queried by both adversary and benign users. Thus, we denote the query data from adversary as adversary poisoned subset $\mathcal{B}_{a}=\{\tilde{x}_{i}\}$. The poisoned subset could be generated from arbitrary clean testing data, denoted as $\mathcal{B}_{ab}=\{x_{i}\}$ (e.g. use any public clean images). The generation follows an additive noise, i.e. $\tilde{x}_{i}=x_{i}+\epsilon_{i},\;s.t.\;\tilde{x}_{i}\in\mathcal{B}_{a}\;x_{i
}\in\mathcal{B}_{ab}$, where the noise $\epsilon_{i}$ is the data poisoning to be learned. The query data from benign users is denoted as benign subset $\mathcal{B}_{b}=\{x_{i}\}$, where $\mathcal{B}_{ab}\cap\mathcal{B}_{b}=\emptyset$. The combination of both, $\mathcal{B}_{t}=\mathcal{B}_{a}\cup\mathcal{B}_{b}$, forms a single TTA minibatch. The effectiveness of test-time poisoning is evaluated at the attack success rate (classification error) on benign subset $\mathcal{B}_{b}$.

### 2.2 Realistic Test-Time Data Poisoning Protocol

The adversarial risk of TTA methods must be assessed under realistic attacks. Existing works may have exaggerated the adversarial risk when attack is implemented under an overly strong assumption. We summarize the criteria that define a more realistic attack as follows.

White-Box v.s. Grey-Box Attack: Access to real-time TTA model parameters is a key factor for crafting realistic test-time data poisoning. Contrary to the assumption adopted by white-box adversarial attack (Szegedy et al., ; Biggio et al., ) that a frozen model is deployed for inference, the TTA model parameters experience constant updating at inference stage. The update is carried out on the cloud side, thus the attacker is assumed to have no access to real-time model weights. Such a realistic assumption prompts us to explore a more relaxed grey-box test-time data poisoning, i.e. only the model architecture and initial model weights are available to the adversary, such as the open-source famous pre-trained models (He et al., ; Dosovitskiy et al., ) and popular foundation models (Radford et al., ; Kirillov et al., ; Oquab et al., ).

Access To Benign Subset: The effectiveness of test-time data poisoning is evaluated on the benign subset $\mathcal{B}_{b}$ submitted by benign users. In a standard cloud service, users are generally restricted from accessing the queries of other users. Thus the adversary should only have access to adversary subset $\mathcal{B}_{a}$. This assumption prohibits the practice of crafting poisoned data by directly optimizing (minimizing) the loss on benign subset (Wu et al., ), on which the attack success rate (performance) is calculated.

Attack Order: Finally, attacking TTA model in a realistic way should be implemented during the adaptation stage. Attacking the model before TTA begins is deemed less practical (Cong et al., ).

Based on the aforementioned key criteria, we provide a summary of existing test-time data poisoning methods in Tab. 1. Our analysis indicates that none of the current methods fully satisfy all the established criteria. Specifically, DIA (Wu et al., ) employs a white-box attack strategy, generating poisoned samples by maximizing the error rate on a subset of benign data. TePA (Cong et al., ) attacks the pre-trained model with an offline surrogate model prior to the commencement of test-time adaptation.
In the rest of the paper, we stick to the most realistic assumptions, i.e. online grey-box poisoning and no access to benign subset, named as Realistic Test-Time Data Poisoning (RTTDP).

**Table 1: Taxonomy of methods based on the criteria for realistic test-time data poisoning.**
| Setting | Grey-box v.s. White-box | Access Benign Subset | Attack Order |
| --- | --- | --- | --- |
| DIA (Wu et al., ) | White-box | ✓ | Online |
| TePA (Cong et al., ) | Grey-box | ✗ | Offline |
| RTTDP (Ours) | Grey-box | ✗ | Online |

Adversarial Attacks on Standard Image Classification: Common grey-box or black-box adversarial attack techniques are often impractical for RTTDP due to two primary challenges. First, existing adversarial attack methods operate on a static model and inherently require multiple queries for gradient approximation or fitness evaluation, as seen in query-based attacks (Li et al., ; Xu et al., ), genetic algorithms (Chen et al., ), and black-box optimization (Qiu et al., ). However, in the test-time adaptation setting, the online model is continuously updated with each query during the inference phase. Thus, repetitive querying the model for gradient approximation or fitness evaluation is prohibited. Second, traditional adversarial attack methods focus on crafting adversarial samples to degrade the performance of the attacker’s own input data. In contrast, in a realistic test-time data poisoning scenario, poisoned samples are introduced into the test-time adaptation process to degrade the performance of benign samples submitted by other users.

## 3 Methodology

### 3.1 Grey-Box Attack by Surrogate Model Distillation

To tackle the challenge of access to TTA model parameters, we propose to maintain a surrogate model, denoted as $\hat{\theta}_{t}$ at timestamp $t$, for the purpose of synthesizing poisoned data. To ensure good approximation, we distill the target model $\theta_{t}$ into the surrogate model $\hat{\theta}_{t}$ by leveraging the feedback of poisoned data from the online target model. Specifically, for each query to the target model, we minimize the symmetric KL-Divergence between the posteriors of the target and surrogate models, as Eq. 1. Our empirical observations demonstrated that utilizing the adversarial subset $\mathcal{B}_{a}$ for distillation yields performance comparable to a white-box attack, as illustrated in Fig. 2 (a).

$$ $\mathcal{L}_{dist}=\frac{1}{|\mathcal{B}_{a}|}\sum_{x_{i}\in\mathcal{B}_{a}} \frac{1}{2}\left[KLD\left(h(x_{i};\theta_{t})||h(x_{i};\hat{\theta}_{t})\right )+KLD\left(h(x_{i};\hat{\theta}_{t}||h(x_{i};\theta_{t}))\right)\right]$ (1) $$

### 3.2 In-Distribution Test-Time Data Poisoning

In this section, we further address the challenge of attacking TTA model without access to benign user’s testing samples.
In the first place, we revisit DIA (Wu et al., ), which formulated test-time poisoned data generation as a bi-level optimization problem, in Eq. 2.

$$ $\begin{aligned} &\min_{\mathcal{B}_{a}}\frac{1}{|\mathcal{B}_{b}|}\sum_{x_{i} \in\mathcal{B}_{b}}\mathcal{L}_{atk}\left(x_{i};\theta_{t}^{*}(\mathcal{B}_{t} )\right)\\ &s.t.\;\mathcal{B}_{t}=\mathcal{B}_{a}\cup\mathcal{B}_{b};\;{\theta^{b}}^{ \prime}=\{\mu(\mathcal{B}_{t}),\sigma^{2}(\mathcal{B}_{t})\};\;\theta_{t}^{u*} =\arg\min_{\theta_{t}^{u}}\mathcal{L}_{tta}(\mathcal{B}_{t};\theta_{t}^{*}( \mathcal{B}_{t}));\;\theta_{t}^{*}(\mathcal{B}_{t})=\theta_{t}^{u*}\cup{\theta ^{b}}\cup\theta^{f}\end{aligned}$ (2) $$

The above bi-level optimization problem employed in DIA (Wu et al., ) aims to generate adversarially poisoned data $\mathcal{B}_{a}$ by minimizing the loss function $\mathcal{L}_{atk}$, which is computed on the benign samples $\mathcal{B}_{b}$. Although DIA approximates $\theta_{t}^{u*}\approx\theta_{t}^{u}$ to discard the inner TTA gradient update loop and reduce the number of queries to the online model, several realistic concerns still persist under the RTTDP protocol. First, as discussed in Sec. 2.2, DIA, as a white-box attack method, must query the online TTA model $\theta^{*}_{t}$ for generating poisoned samples. To mitigate this issue, we propose to leverage a surrogate model $\hat{\theta}_{t}$ as an proxy model, which is distilled by Eq. 1 with the last feedback of $\mathcal{B}_{a,t-1}$. Second, DIA evaluates the outer optimization by employing the benign samples $\mathcal{B}_{b}$ (assuming access to benign users’ query samples). The adversarial risk mainly arises from the injection of poisoned samples into the TTA training process. However, employing the validation (benign) samples as the outer optimization objective may result in an overestimation of this risk, as the specific poisoned samples could be tailored for the attack on the inference of the current batch of benign samples (Park et al., ), e.g., ${\theta^{b}}^{\prime}$,
rather than for attacking the TTA process, i.e., $\theta^{u}$.

Figure: Figure 2: (a) The attack performance comparison about the poisoned samples generated on Source (pretrained) Model, our proposed Surrogate Model and Online target Model (white box). (b) The T-SNE visualization of the feature points (before FC layer). Without $\mathcal{L}_{reg}$, common attack losses (e.g. maximizing cross-entropy) produce poisoned samples (orange dots) that are far from benign ones (blue dots), leading to less effective attacks. (c) The attack performance comparison between w.o. and w. $\mathcal{L}_{reg}$. (d) The average prediction entropy of the poisoned samples generated by our proposed two different attack objectives, respectively.
Refer to caption: x2.png

To prevent from using benign users’ samples for optimization and assuming grey-box attack, one possible solution is to swap the benign users’ samples $\mathcal{B}_{b}$ with the adversary’s clean sample $\mathcal{B}_{ab}$ based on the assumption that $\mathcal{B}_{ab}$ and $\mathcal{B}_{b}$ are drawn from similar distributions, and replacing the white-box model $\theta$ with distilled model $\hat{\theta}$ and discarding the inner TTA loop $\hat{\theta}_{t}^{*}\approx\hat{\theta}_{t}$ as adopted by DIA (Wu et al., ), resulting in the following formulation.

$$ $\begin{split}&\min_{\mathcal{B}_{a}}\frac{1}{|\mathcal{B}_{ab}|}\sum_{x_{i}\in \mathcal{B}_{ab}}\mathcal{L}_{atk}\left(x_{i};\hat{\theta}_{t}^{\prime}( \mathcal{B}_{t})\right)\\ &s.t.\quad\mathcal{B}_{t}=\mathcal{B}_{a}\cup\mathcal{B}_{ab};\quad{\hat{ \theta}^{b\prime}}=\{\mu(\mathcal{B}_{t}),\sigma^{2}(\mathcal{B}_{t})\};\quad \hat{\theta}_{t}^{\prime}(\mathcal{B}_{t})=\hat{\theta}_{t}^{u}\cup{\hat{ \theta}^{b\prime}}\cup\hat{\theta}^{f}\end{split}$ (3) $$

Despite the above formulation alleviates the assumption of the available to access the benign users’ samples and the online model parameters, it still remains elusive to tackle. First, as most TTA methods leverage the current batch statistics to estimate the statistics on target domain,
forwarding $\mathcal{B}_{t}$ results in esimating the BN statistics on the combination of $\mathcal{B}_{a}$ and $\mathcal{B}_{ab}$ which bias update of BN parameters. This may results in a mismatch between the feature distribution of $\mathcal{B}_{ab}$ and $\mathcal{B}_{b}$, if $\mathcal{B}_{b}$ is queried independently. Thus, the poisoning effect may fail to generalize to $\mathcal{B}_{b}$. This is evidenced by an empirical study into the distribution of poisoned data in Fig. 2 (b) where the distribution of generated poisoned data (orange dots) deviate substantially from the benign samples (blue dots) if the Eq. 3 is directly attacked. Therefore, we are prompted to explore a solution that does not explicitly require forward pass for both $\mathcal{B}_{a}$ and $\mathcal{B}_{ab}$ simultaneously and is able to transfer the attacked effect from $\mathcal{B}_{a}$ to the benign subset i.e. $\mathcal{B}_{ab}$ or $\mathcal{B}_{b}$.

To tackle this challenge, we would like to approach from introducing additional constraints and combining the optimized and optimizing objectives into one objective to the problem, as Eq. 4, where $D(P_{1},P_{2})$ is a metric of two distributions and $P_{a}$ and $P_{ab}$ refer to the feature distributions of $\mathcal{B}_{a}$ and $\mathcal{B}_{ab}$. We have the follow reason why the formulation is effective. If $\mathcal{B}_{a}$ and $\mathcal{B}_{ab}$ have the similar distribution, we can have $P_{a}\rightarrow P_{ab}\Rightarrow\mathbbm{E}_{P_{a}(x)}[\mathcal{L}_{atk}(x)]
\rightarrow\mathbbm{E}_{P_{ab}(x)}[\mathcal{L}_{atk}(x)]$ according to the Probably Approximately Correct (PAC) learning framework (). Thus attack against $\mathcal{B}_{a}$ has a high chance to generalize to $\mathcal{B}_{ab}$.

$$ $\begin{split}&\min_{\mathcal{B}_{a}}\frac{1}{|\mathcal{B}_{a}|}\sum_{x_{i}\in \mathcal{B}_{a}}\mathcal{L}_{atk}\left(x_{i};\hat{\theta}_{t}^{\prime}( \mathcal{B}_{a})\right);\quad s.t.\quad D(P_{a},P_{ab})=0\end{split}$ (4) $$

Feature Consistency Regularization for In-Distribution Attack:
To achieve indistinguishable distribution between $P_{a}$ and $P_{ab}$, we propose to measure the discrepancy at feature level and introduce the discrepancy as a constraint to the optimization problem. Crucially, since the attack loss is defined in the representation extracted by the backbone network, the distribution consistency is ideally imposed on the intermediate features except for the final semantic one. Specifically, we denote the $l$-th intermediate feature map before each normalization layer as $z_{i}^{l}=f^{l}(x_{i})\in\mathbbm{R}^{H_{l}\times W_{l}\times D_{l}}$. A single Gaussian distribution is fitted to intermediate layer features, as $\mu_{i}^{l}=\frac{1}{H_{l}W_{l}}\sum_{h,w}z_{ihw}^{l},\;\Sigma_{i}^{l}=\frac{1
}{H_{l}W_{l}}\sum_{h,w}(z_{ihw}^{l}-\mu_{i}^{l})(z_{ihw}^{l}-\mu_{i}^{l})^{\top}$ and $\tilde{\mu}_{i}^{l}=\frac{1}{H_{l}W_{l}}\sum_{h,w}\tilde{z}_{ihw}^{l},\;\tilde
{\Sigma}_{i}^{l}=\frac{1}{H_{l}W_{l}}\sum_{h,w}(\tilde{z}_{ihw}^{l}-\tilde{\mu
}_{i}^{l})(\tilde{z}_{ihw}^{l}-\tilde{\mu}_{i}^{l})^{\top}$, where $\tilde{z}_{i}$ and ${z}_{i}$ refer to the sample features from $\mathcal{B}_{a}$ and $\mathcal{B}_{ab}$, respectively. The KL-Divergence between feature distributions is introduced as the constraint.

$$ $\begin{split}&\mathcal{L}_{reg}=\frac{1}{L}\sum_{l}KLD(\mathcal{N}(\mu_{i}^{l} ,\Sigma_{i}^{l})||\mathcal{N}(\tilde{\mu}_{i}^{l},\tilde{\Sigma}_{i}^{l}))\\ \end{split}$ (5) $$

With the introduced constraint $\mathcal{L}_{reg}=0$, we finally formulate the problem as Eq. 6.
The problem now degenerates to a single level optimization with constraints which can be easily converted into an unconstrained optimization problem via Lagrangian multiplier (Lag, ). The unconstrained problem can be solved in an iterative fashion with detailed algorithm presented in the Appendix.

$$ $\begin{split}&\min_{\mathcal{B}_{a}}\frac{1}{|\mathcal{B}_{a}|}\sum_{x_{i}\in \mathcal{B}_{a}}\mathcal{L}_{atk}\left(x_{i};\hat{\theta}_{t}^{\prime}( \mathcal{B}_{a})\right)\\ &s.t.\quad\hat{\theta}^{b\prime}=\{\mu(\mathcal{B}_{a}),\sigma^{2}(\mathcal{B} _{a})\};\quad\hat{\theta}_{t}^{\prime}(\mathcal{B}_{a})=\hat{\theta}_{t}^{u} \cup{\hat{\theta}^{b\prime}}\cup\hat{\theta}^{f};\quad\mathcal{L}_{reg}=0\\ \Rightarrow&\min_{\mathcal{B}_{a}}\max_{\lambda}\frac{1}{|\mathcal{B}_{a}|} \sum_{x_{i}\in\mathcal{B}_{a}}\mathcal{L}_{atk}\left(x_{i};\hat{\theta}_{t}^{ \prime}(\mathcal{B}_{a})\right)+\lambda\mathcal{L}_{reg}\\ \Rightarrow&\min_{\mathcal{B}_{a}}\max_{\{\lambda_{0}\cdots\lambda_{L-1}\}} \frac{1}{|\mathcal{B}_{a}|}\sum_{x_{i}\in\mathcal{B}_{a}}\mathcal{L}_{atk} \left(x_{i};\hat{\theta}_{t}^{\prime}(\mathcal{B}_{a})\right)+\frac{1}{L}\sum_ {l}\lambda_{l}KLD(\mathcal{N}(\mu_{i}^{l},\Sigma_{i}^{l})||\mathcal{N}(\tilde{ \mu}_{i}^{l},\tilde{\Sigma}_{i}^{l}))\\ \end{split}$ (6) $$

Through optimizing the above objective, now we could craft the effective poisoned data $\mathcal{B}_{a}$ that satisfies $\mathbb{E}_{x_{i}\in\mathcal{B}_{a}}\mathcal{L}_{atk}(x_{i};\hat{\theta}_{t})
\approx\mathbb{E}_{x_{i}\in\mathcal{B}_{b}}\mathcal{L}_{atk}(x_{i};\hat{\theta
}_{t})$, since the approximation equation $P_{a}\rightarrow P_{ab}\rightarrow P_{b}$ now holds. Fig. 2 (c) demonstrates the effectiveness of this in-distribution attack.

### 3.3 TTA-Aware Attack Objective

The specific design of attack objective $\mathcal{L}_{atk}$ warrants careful consideration. The existing works examined both targeted and indiscriminative attacks, demonstrating that both are effective against state-of-the-art TTA methods (Wu et al., ). However, we argue that an attack objective is not universally effective against all TTA methods. Therefore, we investigate two types of attack objectives as follows, and the prediction entropy of poisoned samples generated via the proposed attack losses can be compared in Fig. 2 (d).

High Entropy Attack Objective:
Self-Training based TTA methods, e.g. TENT, RPL, are vulnerable for out-of-distribution samples with high entropy (Niu et al., ), and therefore, a straightforward way to generate poisoned data to attack TTA model is by maximizing the entropy of poisoned data as proposed in TePA (Cong et al., ) since these poisoned samples with high prediction entropy would induce high updating gradient. However, maximizing the prediction entropy does not guarantee wrong pseudo labels. Therefore, we propose a stronger high entropy attack objective called Notch High Entropy Attack (NHE Attack). Based on the uniform distribution, we set the probability in the ground-truth label to zero and construct the target distribution $Q$. Then we minimize the cross-entropy against target distribution $Q$.

$$ $\mathcal{L}^{NHE}_{atk}(\tilde{x}_{i})=-\sum_{k}Q_{ik}\log h_{k}(\tilde{x}_{i} )\quad s.t.\quad Q_{ik}=\begin{cases}0&k=y_{i}\\ \frac{1}{K-1}&others\end{cases}$ (7) $$

Low Entropy Attack Objective:
High entropy attack objective is particular effective against self-training based TTA methods because of high updating gradient, yet they could be easily defended by some defense strategies, e.g. entropy thresholding. Therefore, we further explore a new low entropy based attack objective. DIA (Wu et al., ) proposed to maximize the cross-entropy loss on the benign samples (indiscriminate attack) to generate the other poisoned samples. However, we empirically found that maximizing the cross-entropy loss without any constraints on one sample is prone to maximizing the probability of the most confident class (except the ground-truth) of one model, and feeding these samples into TTA model would bring up the following issues. i) The model will quickly bias towards the most confident class and collapse if without any class diversity constraints. ii) If class diversity constraints is applied (assembled in several TTA methods, e.g. EATA, ROID), this objective will become less ineffective since the class-biased poisoned samples will obtain the less updating weighting than other benign samples. Therefore, we propose a class-balanced low entropy attack, termed Balanced Low Entropy Attack (BLE Attack). Specifically, we maintain an moving average probability confusion $C\in[0,1]^{K\times K}$ to store the prediction bias in each class and find a global optimal label mapping $M\in\{0,1\}^{K\times K}$ such that each class is attacked towards the most probable non ground-truth class. Details of deriving label mapping $M$ is deferred to the supplementary. Finally, the BLE Attack objective is calculate as Eq. 8.

$$ $\mathcal{L}_{atk}^{BLE}(\tilde{x}_{i})=-\sum_{k}\mathbbm{1}(k=\arg\max_{q\neq y _{i}}M_{y_{i},q})\log h(\tilde{x}_{i})$ (8) $$

Overall Attack Strategy: We craft poisoned data by attacking the aforementioned in-distribution attack objective with regularization. Following the practice that poisoned data is less discernible by human, we employ PGD attack (Madry et al., ) on the combined objective in Eq. 6 with a budget $b$. More details of the training and evaluation algorithm are deferred to the Appendix.

## 4 Experiment

### 4.1 Experiment Details

Benchmark Poisoning Methods: We evaluated the following methods under our proposed RTTDP setting. DIA (Wu et al., ) is one of the first approaches towards test-time data poisoning, generating the poisoned data via maximizing the cross-entropy of other benign data. We adapt DIA to the realistic evaluation protocol by splitting $\mathcal{B}_{ab}$ into two subsets of $50\%$ each, i.e. $\mathcal{B}^{p}_{ab}:\mathcal{B}^{b}_{ab}=1:1$, and craft $\mathcal{B}^{p}_{a}$ to maximize the cross-entropy of $\mathcal{B}^{b}_{ab}$. TePA (Cong et al., ) proposed to maximize entropy to generate poisoned data and performed attack before TTA starts. We adapt TePA to generate the poisoned data based on the source model and inject them into TTA pipeline on-the-fly.
MaxCE (Goodfellow et al., ) is an established way to create adversarial samples by maximizing the cross-entropy loss. Finally, we evaluate the two attack objectives proposed in this paper, i.e high entropy attack (NHE Attack) and low entropy attack against most probable and balanced non ground-truth class (BLE Attack). For both NHE Attack and BLE Attack, we evaluate the attack objective subject to the constraint of our proposed feature consistency (Eq. 6).

Datasets: We evaluate on three datasets, widely adopted for TTA benchmarking. CIFAR10-C, CIFAR100-C and ImageNet-C are synthesized from the original clean validation set by adding various types of corruptions to simulate natural distribution shifts (Hendrycks & Dietterich, ). We choose corruption level 5 and perform continual test-time adaptation setting for evaluation.
Following prior works (Wang et al., ; Döbler et al., ), we adopt the pre-trained WideResNet-28 Zagoruyko & Komodakis (), ResNeXt-29 (Xie et al., ), and ResNet-50 (He et al., ) models for experiments on the CIFAR10-C, CIFAR100-C, and ImageNet-C datasets, respectively. More experiment details can be found in the Appendix.

### 4.2 Evaluation on Test-Time Data Poisoning

We present the results of comparing different attack objectives against state-of-the-art TTA methods in Tab. 2, Tab. 3 and Tab. 4 for CIFAR10-C, CIFAR100-C and ImageNet-C respectively. We make the following observations from the results.
i) Contrary to the claims that TTA methods are extremely vulnerable to data poisoning, under the realistic data poisoning protocol, without accessing to benign data, it’s not trivial to transfer the adversarial risk from poisoned data to benign data, especially for the TTA methods using EMA model such as CoTTA and ROID. In particular, existing methods do not pose too much risk to more advanced TTA methods without feature consistency regularization. DIA and TePA are more effective on TENT and RPL than other TTA methods. We attribute this to the fact that both TENT and RPL are naive self-training methods without filtering testing samples, hence, poisoned data could easily mislead model update.
ii) Our proposed two attack objectives generally perform better than existing poisoning methods, demonstrating a better average ranking and a higher average error rate. This is attributed to the combination of the well-designed attack objective and the regularization of feature consistency. On the other hand, low entropy attack (BLE) obtains significantly improved with our proposed feature consistency compared with the similar low entropy attack i.e. MaxCE.
iii) “Non-uniform ” attack in general yields higher attack success rate than “Uniform” attack. This is probably due to consecutive attack being more effective in misleading model’s update.

**Table 2: Evaluation of test-time data poisoning under the RTTDP protocol for CIFAR10-C. We report the attack success rate (higher the better) for each TTA method and the average ranking (lower the better) for each attack objective.**
| Attack Freq. | Attack Objective | Source | TENT | RPL | EATA | TTAC | SAR | CoTTA | ROID | Avg. Err. ($\uparrow$) | Avg. Rank ($\downarrow$) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Uniform | No Attack | 43.81 | 19.72 | 21.00 | 18.03 | 17.41 | 18.94 | 16.46 | 16.37 | 18.28 | 5.71 |
| DIA (Wu et al., ) | 26.04 | 21.87 | 18.94 | 18.56 | 19.46 | 17.72 | 17.77 | 20.05 | 3.71 |  |  |
| TePA (Cong et al., ) | 33.78 | 22.36 | 23.37 | 17.75 | 19.53 | 16.57 | 18.76 | 21.73 | 3.43 |  |  |
| MaxCE (Goodfellow et al., ) | 18.55 | 20.81 | 18.17 | 18.50 | 19.50 | 16.88 | 18.57 | 18.71 | 4.57 |  |  |
| BLE Attack (Ours) | 54.07 | 51.99 | 45.20 | 34.00 | 26.80 | 18.12 | 19.06 | 35.61 | 1.29 |  |  |
| NHE Attack (Ours) | 73.86 | 72.40 | 29.73 | 18.67 | 24.56 | 17.54 | 17.00 | 36.25 | 2.29 |  |  |
| Non-Uniform | No Attack | 43.55 | 19.29 | 20.36 | 17.75 | 16.89 | 18.74 | 16.18 | 15.81 | 17.86 | 5.71 |
| DIA (Wu et al., ) | 22.84 | 25.70 | 19.75 | 18.35 | 19.42 | 19.18 | 17.79 | 20.43 | 4.00 |  |  |
| TePA (Cong et al., ) | 40.31 | 32.32 | 23.03 | 18.06 | 19.49 | 18.07 | 18.50 | 24.25 | 3.43 |  |  |
| MaxCE (Goodfellow et al., ) | 18.64 | 20.01 | 18.29 | 18.43 | 19.47 | 18.01 | 18.94 | 18.83 | 4.43 |  |  |
| BLE Attack (Ours) | 56.17 | 46.66 | 50.97 | 34.25 | 27.54 | 19.23 | 20.12 | 36.42 | 1.43 |  |  |
| NHE Attack (Ours) | 74.93 | 73.65 | 27.56 | 18.75 | 24.95 | 20.86 | 16.77 | 36.78 | 2.00 |  |  |

**Table 3: Evaluation of test-time data poisoning under the RTTDP protocol for CIFAR100-C. We report the attack success rate (higher the better) for each TTA method and the average ranking (lower the better) for each attack objective.**
| Attack Freq. | Attack Objective | Source | TENT | RPL | EATA | TTAC | SAR | CoTTA | ROID | Avg. Err. ($\uparrow$) | Avg. Rank ($\downarrow$) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Uniform | No Attack | 46.23 | 60.25 | 47.12 | 32.20 | 31.93 | 31.62 | 32.13 | 29.11 | 37.77 | 5.71 |
| DIA (Wu et al., ) | 74.79 | 68.42 | 33.77 | 32.57 | 33.20 | 32.68 | 30.10 | 43.65 | 4.00 |  |  |
| TePA (Cong et al., ) | 75.79 | 81.98 | 36.74 | 33.41 | 35.79 | 32.41 | 32.24 | 46.91 | 2.86 |  |  |
| MaxCE (Goodfellow et al., ) | 42.36 | 39.04 | 34.08 | 40.73 | 32.01 | 32.18 | 31.08 | 35.93 | 4.43 |  |  |
| BLE Attack (Ours) | 73.93 | 76.71 | 47.30 | 34.58 | 43.25 | 32.81 | 32.27 | 48.69 | 2.00 |  |  |
| NHE Attack (Ours) | 92.08 | 91.72 | 37.86 | 33.85 | 56.09 | 32.50 | 31.48 | 53.65 | 2.00 |  |  |
| Non-Uniform | No Attack | 46.33 | 62.30 | 49.69 | 31.45 | 32.07 | 31.37 | 32.56 | 28.39 | 38.26 | 5.71 |
| DIA (Wu et al., ) | 76.83 | 71.44 | 33.74 | 32.63 | 33.24 | 33.13 | 30.07 | 44.44 | 4.14 |  |  |
| TePA (Cong et al., ) | 82.29 | 88.48 | 36.40 | 34.89 | 35.56 | 33.44 | 33.27 | 49.19 | 2.43 |  |  |
| MaxCE (Goodfellow et al., ) | 41.89 | 38.05 | 33.20 | 42.12 | 31.90 | 32.78 | 31.66 | 35.94 | 4.43 |  |  |
| BLE Attack (Ours) | 71.30 | 72.97 | 47.21 | 35.26 | 41.22 | 33.50 | 32.60 | 47.72 | 2.29 |  |  |
| NHE Attack (Ours) | 94.46 | 94.58 | 40.05 | 34.14 | 56.48 | 33.52 | 31.45 | 54.95 | 2.00 |  |  |

**Table 4: Evaluation of test-time data poisoning under the RTTDP protocol for ImageNet-C. We report the attack success rate (higher the better) for each TTA method and the average ranking (lower the better) for each attack objective.**
| Attack Freq. | Attack Objective | Source | TENT | SAR | CoTTA | ROID | Avg. Err. ($\uparrow$) | Avg. Rank ($\downarrow$) |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Uniform | No Attack | 82.08 | 63.49 | 61.26 | 63.02 | 53.42 | 60.30 | 5.50 |
| DIA (Wu et al., ) | 67.18 | 62.91 | 64.09 | 56.97 | 62.79 | 4.00 |  |  |
| TePA (Cong et al., ) | 75.36 | 64.90 | 62.84 | 59.78 | 65.72 | 3.00 |  |  |
| MaxCE (Goodfellow et al., ) | 62.64 | 61.66 | 68.83 | 59.89 | 63.26 | 3.25 |  |  |
| BLE Attack (Ours) | 68.04 | 64.31 | 66.40 | 57.10 | 63.96 | 3.00 |  |  |
| NHE Attack (Ours) | 78.03 | 72.58 | 63.84 | 57.72 | 68.04 | 2.25 |  |  |
| Non-Uniform | No Attack | 81.98 | 61.81 | 59.52 | 62.51 | 50.36 | 58.55 | 6.00 |
| DIA (Wu et al., ) | 66.61 | 62.88 | 63.40 | 55.18 | 62.02 | 4.25 |  |  |
| TePA (Cong et al., ) | 74.98 | 62.31 | 62.78 | 59.29 | 64.84 | 3.50 |  |  |
| MaxCE (Goodfellow et al., ) | 62.09 | 65.58 | 72.39 | 59.76 | 64.96 | 2.50 |  |  |
| BLE Attack (Ours) | 67.61 | 65.95 | 65.75 | 56.23 | 63.89 | 2.75 |  |  |
| NHE Attack (Ours) | 77.49 | 73.67 | 64.02 | 57.09 | 68.07 | 2.00 |  |  |

### 4.3 Ablation Study on Attack Modules

In this section, we ablate our proposed modules including the surrogate model, the feature consistency regularization and two attack objectives to demonstrate their their indispensable contribution to the final results. We conduct the experiments on CIFAR10-C and CIFAR100-C datasets as shown in Tab. 5. First, comparing using source model and surrogate model to generate poisoned data, using surrogate model would give superior results which approach or even surpass slightly the ones using online model, regardless of the attack objective. It demonstrates the effectiveness of our proposed surrogate model that is leveraged for generating on-the-fly poisoned data. Second, our proposed NHE attack objective is effective though using source model and without feature consistency regularization, that could be attributed to the high entropy samples easily mislead the model update and cause strong perturbation to the source knowledge. Third, under the surrogate model or online model, both BLE and NHE are significantly improved with the help of feature consistency regularization, empirically demonstrating the reasonableness and effectiveness of our method.

**Table 5: The ablation study of our proposed modules under the RTTDP protocol on CIFAR10/100-C datasets.**
| Attack Model | Attack Objective | Feat. Cons. Reg. | CIFAR10-C | CIFAR100-C |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| TENT | EATA | SAR | Avg. Err. ($\uparrow$) | TENT | EATA | SAR | Avg. Err. ($\uparrow$) |  |  |  |
| Source Model | BLE | - | 19.26 | 18.82 | 19.80 | 19.29 | 34.21 | 31.62 | 32.17 | 32.67 |
| Source Model | BLE | ✓ | 38.09 | 23.28 | 21.86 | 27.74 | 67.11 | 34.57 | 33.78 | 45.15 |
| Source Model | NHE | - | 63.00 | 23.76 | 19.52 | 35.43 | 85.48 | 39.37 | 38.76 | 54.54 |
| Source Model | NHE | ✓ | 37.27 | 20.79 | 21.07 | 26.38 | 81.81 | 35.96 | 47.06 | 54.94 |
| Surrogate Model (Ours) | BLE | - | 20.22 | 20.16 | 19.71 | 20.03 | 38.71 | 31.87 | 31.95 | 34.18 |
| Surrogate Model (Ours) | BLE | ✓ | 54.07 | 45.20 | 26.80 | 42.02 | 73.93 | 47.30 | 43.25 | 54.83 |
| Surrogate Model (Ours) | NHE | - | 72.77 | 21.93 | 19.51 | 38.07 | 81.49 | 32.16 | 31.68 | 48.44 |
| Surrogate Model (Ours) | NHE | ✓ | 73.86 | 29.73 | 24.56 | 42.72 | 92.08 | 37.86 | 56.09 | 62.01 |
| Online Model | BLE | - | 25.14 | 25.50 | 19.71 | 23.45 | 42.09 | 32.82 | 32.05 | 35.65 |
| Online Model | BLE | ✓ | 56.75 | 52.32 | 27.38 | 45.48 | 77.92 | 49.91 | 44.01 | 57.28 |
| Online Model | NHE | - | 72.01 | 21.91 | 19.52 | 37.81 | 80.62 | 31.96 | 31.69 | 48.09 |
| Online Model | NHE | ✓ | 73.62 | 28.73 | 25.03 | 42.46 | 91.15 | 39.01 | 54.67 | 61.61 |

### 4.4 Exploring Effective Defense Practices

In this section, we explore effective defense practices.
We conduct an ablation study for each of the above practices on top of a simple entropy minimization baseline method (Min. Ent.). As seen in Tab. 7, we make the observation that without any hypothesized defense practice, directly minimizing entropy is very sensitive to poisoned data, especially the high entropy attack (NHE). When entropy thresholding (Ent. Thresh.) is applied, we observe a significant improvement in robustness under high entropy attack, suggesting rejecting high entropy testing samples from TTA is an effective defense practice. Furthermore, both data augmentation (Data Aug.) and exponential moving average update (EMA) are very effective defense practices. The former could perturb the testing sample towards non-adversarial direction while the latter prevents the model from updating too quickly, thus less sensitive to poisoned data. Finally, one might expect stochastic parameter restoration (Stoch. Resto.) to be an effective defense method. Despite exhibiting improved adversarial robustness alone, parameter restoration does not further improve the robustness when combined with other effective defense methods.

**Table 6: Ablation study of hypothesized defense practices on CIFAR10-C dataset.**
| Min. Ent. | Ent. Thresh. | Data Aug. | EMA Update | Stoch. Resto. | BLE | NHE |
| --- | --- | --- | --- | --- | --- | --- |
| - | - | - | - | - | 43.81 |  |
| $\checkmark$ | - | - | - | - | 54.07 | 73.86 |
| $\checkmark$ | $\checkmark$ | - | - | - | 46.24 | 35.86 |
| $\checkmark$ | $\checkmark$ | $\checkmark$ | - | - | 24.01 | 20.05 |
| $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | - | 20.22 | 19.76 |
| - | - | - | - | $\checkmark$ | 29.68 | 50.68 |
| $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | $\checkmark$ | 20.41 | 20.30 |

### 4.5 Generalisation of our modules

In this section, to demonstrate the generalisation of our method, we combine the existing attack objectives with our proposed modules including the surrogate model and feature consistency regularization. The comparisons are shown in Tab. 7. First, we can observe that DIA and MaxCE could get improved additional with our proposed method. Second, TePA could obtain improvement under TENT method but slightly degraded under EATA method. It could be that TePA using maximizing entropy as an attack objective, and with the help of the surrogate model, the poisoned data would have very high prediction entropy because the attack reference model is more approximate to the online model, but fail to pass the entropy threshold and class diverse weighting using in EATA. Overall, our proposed in-distribution attack could generalize to most of the existing attacking objectives.

## 5 Related Work

Test-Time Adaptation (TTA)
(Wang et al., ; ; Niu et al., ; Su et al., ; Li et al., ) have shown significant success in bridging the domain gap by using stream-based testing samples to dynamically update models in real time. The success of TTA is mainly attributed to the self-supervised learning on testing data. While it has proved sensitive to confirmation bias (Arazo et al., ), many solutions were proposed to minimize the influences of the wrong pseudo-labels.
These strategies include minimizing sample entropy (Wang et al., ; Liang et al., ), adding regularization terms (Song et al., ; Su et al., ), using confidence thresholding (Niu et al., ; ), updating models with exponential moving average architectures (Wang et al., ; Döbler et al., ), partially updating model weights (Wang et al., ; Yuan et al., ), and augmenting testing samples (Zhang et al., ; Döbler et al., ), all aimed at minimizing sample distribution discrepancies.
However, adaptation during the testing stage remains highly risky, as it directly impacts the final results. In this work, we aim to demonstrate that most existing TTA methods are highly susceptible to data poisoning.

Robustness in Test-Time Adaptation:
Recent research has increasingly focused on the robustness of TTA in realistic deployments. Wang et al. () and Brahma & Rai () tackle issues of catastrophic forgetting due to changing test data distributions. Niu et al. (), Gong et al. () and Yuan et al. () address non-i.i.d. and shifting label distributions in test data. Li et al. () and Zhou et al. () introduce open-world scenarios in TTA, where test data may include novel classes not present in the source domain.
Furthermore, Wu et al. () and Cong et al. () investigate the threat of data poisoning in TTA, where attackers alter test data to exploit vulnerabilities in TTA methods. DIA (Wu et al., ) uses bi-level optimization to degrade target sample performance, assuming white-box attack access. In contrast, TePA (Cong et al., ) includes an exclusive offline stage for poisoning data on the source model prior to the TTA process.
Inspired by these works, we focus on the severe threat of data poisoning. We examine the adversarial robustness of TTA under realistic conditions, with access only to the source model and partial test samples for data poisoning. Our attack, despite these constraints, outperforms previous methods, highlighting the significant adversarial risks that TTA methods face.

Adversarial Attack & Data Poisoning:
Adversarial risk is a crucial concern for models to address for safe deployment, which can be primarily divided into two categories: adversarial attacks and data poisoning.
Adversarial attacks (Szegedy et al., ; Akhtar & Mian, ; Goodfellow et al., ; Madry et al., ) manipulate models during inference by adding small perturbations to input data. White-box attacks (Szegedy et al., ; Tramèr et al., ) assume full access to the victim model, creating adversarial examples by maximizing loss gradients. In contrast, grey-box attacks (Chen et al., ; Ilyas et al., ; Ru et al., ) assume no model access, generating adversarial samples by estimating gradients through intensive querying.
Data poisoning (Biggio et al., ; Yang et al., ; Shafahi et al., ; Alfeld et al., ) compromise models by injecting manipulated data into the training set, misleading the training process. Traditional data poisoning assumes that the attacker can observe and poison the entire training set at once. Recent advancements, such as online poisoning (Zhang et al., ), relax this assumption by requiring knowledge of the model updating strategies. In this work, we explore a more realistic scenario of adversarial robustness for TTA, focusing on data poisoning without online model access and without multiple queries for the same data.

## 6 Conclusion

In this work, we reviewed the assumptions adopted by existing works for generating poisoned data at test-time and propose a few criteria that define a more realistic test-time data poisoning. Specifically, we approach from the angles of attack transparency, access to other users’ benign data, attack budget, and attack order. To craft realistic poisoned data, we proposed a grey-box in-distribution attack with attack objective tailored for TTA methods. Through extensive evaluations under the realistic evaluation protocol, we reveal that the adversarial risk of TTA method might be over estimated and, importantly, certain practices in TTA methods are empirically proven to be effective and should be considered for designing adversarial robust TTA methods in the future.

Limitation:
We list the limitations as follows. The proposed test-time data poisoning strategy is based on grey-box attack, i.e. model architecture and initial weights are available to the adversary. The proposed attack does not guarantee huge improvement of attack success rate on all TTA methods, especially the ones integrated with multiple defense techniques.

## Appendix A Appendix

### A.1 Additional Details for Methodology

In this section, we present additional details for the Methodology, including an illustration of batch split, the way to construct the global optimal label mapping for low entropy attack, the overall poisoning & surrogate model update algorithm, as well as details of defense practices.

#### A.1.1 Illustration of Test-Time Data Poisoning Batch Split

We visualize the batch split under realistic test-time data poisoning in Fig. 3. We present the batch split scheme for both “Uniform” and “Non-Uniform” attack frequencies. Under “Uniform” attack frequency, the poisoned minibatch is uniformly presented in the test data stream while “Non-Uniform” attack protocol simulates the situation the adversary attacks the TTA model in a short period of time with a huge amount of poisoned data.

Figure: Figure 3: Illustration of test-time data poisoning batch split.
Refer to caption: x3.png

#### A.1.2 Global Optiomal Label Mapping

For Balanced Low Entropy Attack, we need to obtain a label mapping which mainly addresses the following issues, i) maps the GT label to one wrong label; ii) the label mapping is bijective; iii) the sum of the mapping cost is the minimal. To achieve it, we first define a probability confusion between all class pairs as $C\in[0,1]^{K\times K}$. The probability confusion is updated in an exponentially moving average fashion using the current posterior predictions. The label mapping $M\in\{0,1\}^{K\times K}$ is then obtained by optimizing the following linear assignment problem. Efficient solver, e.g. Hungarian method, can be employed to solve this problem, as Eq. 9.

$$ $\begin{split}&\hat{M}=arg\max_{M}\sum_{k}\sum_{q}C_{k,q}M_{k,q}\\ s.t.\quad&{M}_{k,k}\neq 1,\quad\sum_{q}{M}_{k,q}=1,\quad M\in\{0,1\}^{K\times K },\\ &C_{k}^{t}=\beta C_{k}^{t}+(1-\beta)\frac{\sum_{\tilde{x}_{i}\in\mathcal{B}_{a }}\mathbbm{1}(y_{i}=k)\cdot h(\tilde{x}_{i})}{\sum_{x_{i}\in\mathcal{B}_{a}}h( x_{i})}\end{split}$ (9) $$

Here, we also provide the pseudo code of the implementation of BLE Attack, as follow,

def attack_objective(self, x, y):    // x: (B, K) the predicted logit of poisoned data    // y: (B, )  the ground true label of poisoned data    // return the attack loss value    with torch.no_grad():        // EMA update C^t_k        curr_prob_term = scatter_mean(x.softmax(1), y[:, None], dim=0, out=torch.zeros_like(self.class_wise_momentum_prob))        new_ema_prob = self.class_wise_momentum_prob.clone()        new_ema_prob[y.unique()] = self.momentum_coefficient * new_ema_prob[y.unique()] + (1 - self.momentum_coefficient) * curr_prob_term[y.unique()]        new_ema_prob_select = new_ema_prob.clone()        diag_mask = torch.diag(torch.ones(new_ema_prob_select.shape[0]))        new_ema_prob_select[diag_mask.bool()] = 0.        // Find the global optimal mapping M        label_mapping = y.new_zeros(new_ema_prob_select.shape[0], dtype=torch.long)        for i in range(new_ema_prob_select.shape[0]):            biased_prob, biased_class = F.normalize(new_ema_prob_select, dim=-1, p=1).max(dim=-1)            max_item = biased_prob.argmax(dim=-1)            label_mapping[max_item] = biased_class[max_item]            new_ema_prob_select[max_item, :] = 0.            new_ema_prob_select[:, biased_class[max_item]] = 0.    // BLE attack loss    loss = F.cross_entropy(x, label_mapping[y])    self.current_prob = new_ema_prob.detach()    return loss

#### A.1.3 Detailed Poisoning & Training Algorithm

We present the overall algorithm for generating poisoned data and surrogate model update in Alg. 1

Figure: Algorithm 1 The pipeline of the proposed test time data poisoning

#### A.1.4 Details of Defense Practices

Here, we provide the details about the defense practices evaluated in Tab. 5 of the manuscript.

- •
Entropy Thresholding is implemented through filtering the entropy below $0.05*log(K)$, where K is the class number of the dataset.
- •
Data Augmentation performs the data augmentation used into CoTTA over the input samples and constrain the consistent predictions between the augmented samples and the corresponding original samples.
- •
EMA Update module allow us to maintain an exponentially moving average updated model to generate robust predictions and used to supervise the online model update, and the EMA momentum is 0.999.
- •
Stochastic Parameter Restoration is implemented as the module used in CoTTA. We randomly reset the network weights to source model weights with probability $p$ and $p$ is set to $0.01$.

### A.2 Additional Details for Experiment

Benchmark Poisoning Methods: We evaluated the following methods for test-time data poisoning. DIA (Wu et al., ) is one of the first approaches towards test-time data poisoning, generating the poisoned data via maximizing the cross-entropy of other benign data. We adapt DIA to the realistic evaluation protocol by splitting $\mathcal{B}_{ab}$ into two subsets of $50\%$ each, i.e. $\mathcal{B}^{p}_{ab}:\mathcal{B}^{b}_{ab}=1:1$, and craft $\mathcal{B}^{p}_{a}$ to maximize the cross-entropy of $\mathcal{B}^{b}_{ab}$. TePA (Cong et al., ) proposed to maximize entropy to generate poisoned data and performed attack before TTA starts. We adapt TePA to generate the poisoned data based on the source model and inject them into TTA pipeline on-the-fly.
MaxCE (Goodfellow et al., ) is an established way to create adversarial samples by maximizing the cross-entropy loss. Finally, we evaluate the two attack objectives proposed in this paper, i.e high entropy attack (NHE Attack) and low entropy attack against most probable and balanced non ground-truth class (BLE Attack). For both NHE Attack and BLE Attack, we evaluate the attack objective subject to the constraint of our proposed feature consistency (Eq. 6).

Benchmark TTA Methods:
We investigate several state-of-the-art TTA methods under our RTTDP protocol to evaluate their adversarial robustness.
Source serves as the baseline for inference performance without adaptation.
TENT (Wang et al., ) updates BN parameters through minimizing entropy.
RPL (Rusak et al., ) performs self-training with a generalized cross-entropy (GCE) loss, which aids in more robust adaptation under label noise.
EATA (Niu et al., ) minimizes entropy with the Fisher regularization term to prevent forgetting knowledge from the source domain.
TTAC (Su et al., ) adapts all backbone parameters by jointly optimizing global and class-wise distribution alignment with the source distribution.
SAR (Niu et al., ) updates BN parameters with a sharpness-aware optimizer to filter out noisy labels and help escape local minima.
CoTTA (Wang et al., ) leverages a teacher-student structure, optimizing all parameters of the student model and updating the teacher model using an exponential moving average. To better prevent forgetting during continual adaptation, it incorporates parameter random resetting and data augmentation methods.
ROID (Marsden et al., ) updates BN parameters with loss of self-label refinement (SLR) weighed by certainty and diversity while continually weighting the online model and the source model to prevent forgetting.

Datasets: We evaluate on three datasets, widely adopted for TTA benchmarking. CIFAR10-C, CIFAR100-C and ImageNet-C are synthesized from the original clean validation set by adding various types of corruptions to simulate natural distribution shifts (Hendrycks & Dietterich, ). We choose corruption level 5 and perform continual test-time adaptation setting for evaluation.
Following prior works (Wang et al., ; Döbler et al., ), we adopt the pre-trained WideResNet-28 Zagoruyko & Komodakis (), ResNeXt-29 (Xie et al., ), and ResNet-50 (He et al., ) models for experiments on the CIFAR10-C, CIFAR100-C, and ImageNet-C datasets, respectively.

Evaluation Protocol: We devise a evaluation plan respecting the realistic test-time data poisoning criteria. First, we investigate the frequency of injecting poisoned data. The “Uniform” scheme indicates that the poisoned minibatch is uniformly present in the test data stream, simulating the scenario that the adversary is periodically injecting the poisoned data. We further evaluate “Non-Uniform” scheme by allowing the adversary to concentrate the attack budget within a short period of time.
We fix the overall attack budget as $r=\frac{|\mathcal{B}_{a}|}{|\mathcal{B}_{a}|+|\mathcal{B}_{b}|}$ throughout the experiments. We report the attack success rate as the evaluation metric, which is measured as the percentage of misclassified benign samples in the benign subset $\mathcal{B}_{b}$. Additionally, for easier comparison, we also calculate the average error rate (higher the better) and the average ranking (lower the better) for each poisoning method.

Hyperparameters: For all competing methods, we employ the $40$ steps $L_{\infty}$ PGD attack to generate poisoned data. The maximum perturbation budget $b$ is $0.3$ and the attack step size is $0.01$. Additionally, within each PGD iteration, we update $\lambda_{l}$ via $\lambda^{\prime}_{l}=\lambda_{l}+0.001\cdot\nabla_{\lambda_{l}}\mathcal{L}$ for Eq. 6, where $\lambda_{l}$ is initialized as zero before PGD attack.
Unless otherwise noted, the overall attack budget $r$ is $50\%$ throughout the experiment. For surrogate model distillation module, we adopt SGD optimizer with $0.1$ learning rate for $10$ iterations to update the surrogate model in each update stage.

### A.3 Additional Empirical Analysis

#### A.3.1 Visualization of Poisoned Samples

We visualize selected samples before and after test-time data poisoning in Fig. 4. The high corruption level makes the adversarial noise less noticeable, suggesting the poisoned data could even evade human inspection.

Figure: Figure 4: Visualizing of selected samples before and after test-time data poisoning.
Refer to caption: x4.png

#### A.3.2 Different Attack Budgets

We further evaluate the effectiveness of proposed poisoning approach under different attack budgets. Specifically, we evaluated at $b=0.1$, $b=0.2$ and $b=0.5$. We clearly observe that both high entropy and low entropy attacks are effective regardless of attack budgets.

**Table 8: Comparing the attack performance of test-time data poisoning under different attack budgets.**
| TTA | Attack Obj. | 0.1 | 0.2 | 0.5 |
| --- | --- | --- | --- | --- |
| TENT | No Attack | 20.72 | 20.39 | 19.72 |
| BLE Attack | 22.44 | 27.60 | 54.07 |  |
| NHE Attack | 39.20 | 62.19 | 73.86 |  |
| EATA | No Attack | 17.99 | 17.76 | 18.03 |
| BLE Attack | 22.20 | 28.29 | 45.20 |  |
| NHE Attack | 19.59 | 20.10 | 29.73 |  |
| SAR | No Attack | 18.95 | 18.90 | 18.94 |
| BLE Attack | 19.90 | 21.30 | 26.80 |  |
| NHE Attack | 19.33 | 20.74 | 24.56 |  |

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: **Questions**

L181: Why is repeated querying prohibited? I understand that the model is constantly updated throughout the querying, but is this really a problem?

Section 3.1: How exactly is the model distilled? L181 says repeated querying is prohibited, but my understanding is that training a surrogate model requires precisely such querying? How can you query for a single $\theta_t$ if it changes by the very fact that it is being queried?

L232/233: How can we assume to know $\mathcal{B}_{a, t-1}$? Other users could have queried the model an unknown number of times between training the surrogate and launching the attack.

Table 2-4: What do “Uniform” and “Non-Uniform” Attack Freq. refer to? I could not find this mentioned anywhere in the paper. The appendix has a figure, but also no sufficient explanation. Since this is a central part of the main results tables, it should be carefully explained in the main part of the paper.

Table 2-4: What does “Source” refer to? this should also be explained.

L259/260: I don’t understand what this sentence means. What is “the optimized”, what is the “optimizing objective”, what is the “problem”?

L431: I do not understand this sentence. Can you please rephrase?

L149/150: “the attacker is assumed to have no access to real-time model weights.” By whom is this assumed?

Table 1: what does “Offline” attack order refer to?

**Suggestions for Improvement**

Figure 1: The overview figure 1 gives over the method is great! I appreciate the difficulty in illustrating the complex system with many aspects. I would like to make some suggestions to simplify the figure, as it took me a long time to work through it, and I believe it would be much more helpful with slightly less detail that distracts from the core parts:

- I suggest entirely removing the two boxes “TTA Model” and “Attack Objectives”. They don’t add much information over the text, and it would allow the viewer to concentrate on the three parties in the system: the adversary, the benign user, and the TTA server.
- Removing the symbols for the attacker and user would further reduce visual clutter, and they are redundant to the titles of the boxes.
- I would stylize the images from the distributions - e.g. squares of one color per distribution - instead of using actual images from the dataset. This has several advantages: (i) it reduces visual clutter, (ii) it makes it immediately apparent which distribution they belong to, (iii) the “poison” symbol becomes more apparent (it took me a while to see the little devils).

There may be more opportunities for improvements, e.g., thicker lines for arrows, but I believe the three points above will already make the figure significantly easier to understand.

Figure 2: Please consider breaking these plots into individual figures.

- They are currently way too small, to the point where labels and points are barely readable when printed. At least doubling their size would be required.
- The several subplots are unrelated and even referenced from different sections of the paper. It would be much more helpful to have the figure close to the text that it refers to.

L298: You refer to Fig 2(c), which contains many acronyms that have not been introduced. E.g., TENT, EATA, SAR, NHE, BLE, …

Tables 5-7: These tables would be easier to read without the grey shading.
soundness: 3
strengths: **More Realistic Threat-Model**

The paper critically examines the assumptions made in prior work regarding the threat model of poisoning attacks against TTA systems, particularly regarding the adversary’s capabilities. In particular, the relaxation of (i) white-box access to gray-box access and (ii) restricting access to the targeted samples makes the setting much more realistic and, therefore, the findings more significant and relevant.

**A New Attack Rooted in Theoretical Insights**

The proposed method cleverly extends on previous attacks, making it effective in the new, more restricted attack setting. The method is rooted in theoretical insights that nicely explain the various parts and modifications. It demonstrates that TTA is still vulnerable to poisoning attacks even with more restrictive assumptions.

**Thorough Evaluation**

The experimental evaluation is thorough. It spans three different datasets with increasing complexity (CIFAR10-C, CIFAR1-C, ImageNet-C), multiple different TTA methods, and compares against two different baselines. The evaluation also includes an ablation study, as well as first look at potential defenses, which I appreciate. The defenses show promise but are unable to fully recover the functions of the original models.
summary: This paper investigates adversarial risks in test-time adaptation (TTA), which updates model weights during inference to counter distribution shifts. The authors argue that the assumed threat model of prior work is unrealistic and gives the attacker too much access. They propose a more restricted attack model called Realistic Test-Time Data Poisoning (RTTDP), assuming grey-box access to the model and no access to the benign samples of other users. The authors propose a new attack method outperforming previous work in this new setting.
weaknesses: **Improved Assumptions Still Strong**

While the gray-box assumption is a significant improvement over the white-box assumption in some prior work, it still assumes the adversary's access to the original model (before TTA) and data from the same distribution as the victim benign user. Both of these assumptions are fairly strong and could ideally be relaxed further to, e.g., black-box access.

**Unclear Details**

Several important details of the method remain unclear and should be included in the paper. The most important ones include: why repeated querying of the model is forbidden (L181), how the surrogate model can be trained without querying (Section 3.1), why we can assume to know $\mathcal{B}_{a, t-1}$, and how “Uniform” and “Non-Uniform” attacks are executed. Please refer to “Questions” for more details and additional comments.

**The Presentation Can Be Improved**

The method’s presentation could be improved in multiple ways:

- Figure 1 gives a good overview of the method. It could benefit from some simplifications to reduce visual clutter; see “Suggestions for Improvements” for detailed suggestions.
- Figure 2 is very small and should be enlarged. The four subplots are also unrelated and referred to from different parts of the paper. I suggest splitting it into individual figures and moving those to the corresponding sections.
- In general, the paper’s English is easy to read. However, some parts are hard to understand due to grammatically wrong or overly complex sentences (e.g. L431, L259, …). The paper would benefit significantly from a careful revision, possibly with an English language tool.

**No Code Available**

No code was available for review, and the authors did not specify whether it will be released upon publication.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1. Has the author investigate the effect of the ratio of between the adversarial example and the benign examples? 
2. Did the author experiment and compare with harder attack like in [1] that requires access to benign examples. Or is there quantitative measurement on the tradeoff when relaxing to realistic attacks. 
3. Is it possible to derive any formal guarantee on the attack effectiveness when relaxing the different constraints?



[1]. Chen, J., Wu, X., Guo, Y., Liang, Y., and Jha, S. Towards evaluating the robustness of neural networks learned by transduction. In Int. Conf. Learn. Represent., 2022
soundness: 3
strengths: 1. The paper is overall well-written with clear math definitions and extensive experiments and studies an important problem of data poisoning under weaker/realistic assumptions. 
2. The paper studies the popular test-time attack problem from a novel perspective of weaker threat model.
summary: This paper examines the adversarial risks in test-time adaptation (TTA), highlighting that TTA’s exposure to test-time data poisoning may be less severe under realistic attack assumptions. The authors propose two TTA-aware attack objectives and a new in-distribution poisoning method that operates without access to benign data, revealing that TTA methods show greater robustness than expected and identifying defense strategies to enhance TTA’s adversarial resilience.
weaknesses: 1. Line 680, citation format wrong. Missing conference name.
2. Figure 1 is helpful but visually confusing and ovewhelming. Please explain what B_a, B_ab, B_t, B_b in the figure description section. A term (-\lambda L_{reg} and L_{atk}) is confusing to put with the full equation . Also, it is better to present the attack objective in separate part than to fit it in Figure 1.
3. It will be more helpful to have a graph that assign the current popular attack methods into different buckets, where each bucket has different threat model.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 2
questions: * Does this only work against losses $\mathcal{L}_{tta}$ that are unsupervised? It would be nice to explain this a bit more and give some examples of what this loss function looks like in the main body. 
* Do you ever specify the attack budget? I couldn't find it in the paper. In A.2, you define this quantity, $r$, but I don't ever see details for it. Is this the same thing as $b$ in Table 8? 
* What are the "Source" numbers listed in Tables 2,3,4? Is this just the success of standard adversarial attacks?
soundness: 3
strengths: * The experimentation is thorough, and generally well presented. 
* The authors do a great job of pointing out issues with existing poisoning attacks against TTA training. 
* The authors motivate and propose a poisoning scheme of their own, and show its effectiveness in several settings.
summary: The paper deals with data poisoning in the test time adaptation setting. The paper exposes some issues with existing poisoning attacks in this setting, and goes on to propose certain alterations/additions to existing poisoning objectives, and evaluates results on several datasets, with several test time adaptation methods.
weaknesses: * The area of data poisoning in the TTA setting is a bit niche, and to be honest, doesn't seem to present many challenges beyond existing data poisoning settings. So called "availability" attacks have been introduced in [1,2], and several other works. These should at least be cited, and probably compared against as there's a decent amount of overlap. 
* The real "value add" for the authors, in my view, is the addition of the feature clustering regularizer. The other contributions (BLE, notch loss, etc.) seem to be very slight modifications of existing poisoning attacks, or even just existing adversarial attacks. 
* The related work shouldn't be at the end. It would be very helpful for readers to have some more info on TTA at the beginning of the work. But if you're going to keep it at the end, you NEED to cite things earlier, as it's totally unclear what things like TENT, RPL, etc. are when they're introduced. It seems like in section 4.2, only the acronyms are introduced, with no explanation, and no citation to click on and find more information. Note: I didn't deduct any "points" for this weakness, but it really needs to be addressed. 

[1] Huang, Hanxun, et al. "Unlearnable examples: Making personal data unexploitable." arXiv preprint arXiv:2101.04898 (2021).

[2] Fowl, Liam, et al. "Adversarial examples make strong poisons." Advances in Neural Information Processing Systems 34 (2021): 30339-30351.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: How does your approach transition from a bilevel optimization problem to a single-level one, and could you provide additional theoretical support for this methodology?

Can you elaborate on how your benchmarking setup differs from prior works and how these design choices may influence your results?

Given that MaxCE is primarily an adversarial example attack, what was the rationale behind including it in your comparisons, and have you considered additional comparisons with more advanced adversarial example attacks like PGD or AutoAttack?
soundness: 3
strengths: - The study is well-written;
- It covers an interesting setting that deserve much more attention from the security aspect.
summary: This paper explores test-time data poisoning (TTDP) under realistic conditions, proposing a grey-box attack that reflects practical scenarios where adversaries lack full access to benign data and real-time model updates. The study introduces two attack objectives based on entropy manipulation to effectively poison data while remaining within realistic constraints. Through extensive experimentation on state-of-the-art test-time adaptation (TTA) methods, the authors assess the vulnerability of these methods and propose feature consistency regularization as a countermeasure. This work highlights the persistent adversarial risks in TTA setups under real-world scenarios.
weaknesses: While the study is well-written and formally structured, there are several areas where clarity and methodological rigor could be improved.

- Distinction between RTTDP and TePA in Table 1. In Table 1, the difference between RTTDP and TePA, particularly in terms of online adaptability, is not clear. It would be beneficial to clarify whether TePA is prevented from online applicability, thus highlighting RTTDP’s novelty.

- Inconsistent comparison between poisoning and adversarial attacks. The paper compares the proposed poisoning attack with MaxCE, an adversarial example attack applied at test time rather than during training. This inclusion creates confusion about the setting being considered, especially as MaxCE’s performance is close to that of the proposed attacks, despite being a simpler, suboptimal approach. Additionally, this inconsistency leads to general confusion throughout the paper regarding the exact threat model, setting, and practical cases under consideration. The title’s emphasis on “Realistic” test-time data poisoning also adds to this confusion, as data poisoning traditionally pertains to training rather than test inference. To clarify, the authors should explain that, while poisoning data are collected during test inference, they are subsequently used to update the model through adaptation techniques, which occurs as an offline process. Better descriptions and visualizations of the threat model, attack setting, and relevant practical scenarios would improve clarity and accurately convey the scope and applicability of the proposed approach.

- Not impactful results. MaxCE’s performance, although designed as an adversarial example attack rather than a poisoning attack, remains close to that of the proposed attacks, which raises questions about the actual impact and effectiveness of the proposed methods. The results do not demonstrate a clear or significant advantage of the new attack over MaxCE, suggesting that the proposed methods may lack the robustness or advancement intended. Furthermore, more sophisticated adversarial example attacks (e.g., PGD, C&W, and AutoAttack). I recommend including comparisons with these advanced adversarial attacks to provide a clearer perspective on whether the proposed approach offers meaningful improvements. Currently, the lack of a notable performance gap reduces the impact of the results, leaving it uncertain whether the proposed attacks represent a substantial advancement.

- Unsupported and unclear methodology. The transition from a bilevel optimization formulation to a single-level optimization problem lacks theoretical clarity. Such transitions are complex, and previous research has shown limitations in achieving optimal results through this method. Expanding on this approach and its theoretical basis would improve the rigor of the paper’s claims.

- Confusing experimental setup. The design choices in the "Benchmarking Poisoning Methods" section lack clarity. The authors do not specify how these choices impact prior works or how they differ from established benchmarking methods, such as DIA and TePA. This lack of clarity makes it difficult to understand the setup and its implications for reproducibility.

- Missing details on attack queries. The number of queries used by each attack is a critical factor, yet the experimental comparisons lack this information. Since query efficiency is a key point raised by the authors, detailing query usage across attacks would be essential for a fair comparison.


**Minor Points:**

- In Equation 1, the authors include two Kullback-Leibler Divergence (KLD) terms. To clarify their necessity, the authors should explain why both terms are required rather than just one, even if asymmetry in KLD is a factor. Providing justification in the text would enhance methodological clarity.

- I suggest the authors break Equation 2 into multiple lines to improve readability.

- Expanding the background discussion presented in Section 2 and related work with further background knowledge on adversarial examples, and on white-box, grey-box, and black-box threat models in the context of poisoning. The authors could reference relevant surveys that would help to clarify their threat model and position at the SoA and facilitate non-expert readers understand these distinctions. 

- Figure 1 is overly technical and lacks clarity regarding the threat model and attacker strategy. Simplifying it could enhance its value as a conceptual overview.


**References**
- [AutoAttack] Reliable evaluation of adversarial robustness with an ensemble of diverse parameter-free attacks.
- [PGD] Towards deep learning models resistant to adversarial attacks
- [C&W] Towards evaluating the robustness of neural networks

Meta-review: "On the Adversarial Risk of Test Time Adaptation: An Investigation into Realistic Test-Time Data Poisoning" re-investigates assumptions made in previous work on data poisoning during test time augmentation, propose a new gray-box setting and provide a careful investgation in this, more realistic, setting. 

Reviewers generally like this reinvestigation of attacks in the TTA setting, and the careful investigation in the submission. A number of concerns remain, such as the, still relatively large, attack budget necessary in this setting, compared to standard data poisoning, and the clarity of presentation of the results and the comparison to adversarial attacks. I do think it is in the authors' best interest to further improve the clarity of their presentation until the conference.

Decision: Accept (Poster)