On the Adversarial Vulnerability of Label-Free Test-Time Adaptation

Authors: Shahriar Rifat, Jonathan Ashdown, Michael J. De Lucia, Ananthram Swami, Francesco Restuccia
Primary area: alignment, fairness, safety, privacy, and societal considerations
Keywords: Test Time Adaptation, Adversarial Attack

Abstract:
Despite the success of Test-time adaptation (TTA), recent work has shown that adding relatively small adversarial perturbations to a limited number of samples leads to significant performance degradation. Therefore, it is crucial to rigorously evaluate existing TTA algorithms against relevant threats and implement appropriate security countermeasures. Importantly, existing threat models assume test-time samples will be labeled, which is impractical in real-world scenarios. To address this gap, we propose a new attack algorithm that does not rely on
access to labeled test samples, thus providing a concrete way to assess the security vulnerabilities of TTA algorithms. Our attack design is grounded in theoretical foundations and can generate strong attacks against different state of the art TTA methods. In addition, we show that existing defense mechanisms are almost ineffective, which emphasizes the need for further research on TTA security. Through extensive experiments on CIFAR10-C, CIFAR100-C, and ImageNet-C, we demonstrate that our proposed approach closely matches the performance of state-of-the-art attack benchmarks, even without access to labeled samples. In certain cases, our approach generates stronger attacks, e.g., more than 4% higher error rate on CIFAR10-C.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 3

Reviews and discussion:
<thread>
creation date:
1730367128035

thread_type:
official_review

<thread_object>
creator: reviewer_achd

<content>
summary:
To address the distribution shift of deep neural networks (DNNS), test-time adaptation (TTA) updates the model parameters using test samples. However, recent work reveals that TTA algorithms are vulnerable to adversarial attacks. This paper proposes a novel attack named Feature Collapse Attack (FCA). To be specific, the adversarial generated through FCA can be mixed with benign samples, thereby degrading the performance of the target model. A main feature of FCA is that it does not need the label of the test samples.

strengths:
- This paper is well-written and easy-to-follow
- TTA algorithms are the key developing trend of AI, and the evaluation of the adversarial robustness of TTA algorithms is timely and necessary.

weaknesses:
- The threat model of this paper is impractical.
- The experiments are insufficient.
- Important baseline attack is missing.

questions:
Thanks for submitting the paper to ICLR 2025!

This paper aims to address a timely and important problem, i.e., evaluating the adversarial robustness of TTA algorithms. As the authors claimed, launching adversarial attacks against TTA algorithms is not a novel question, but at an early stage. Meanwhile, mounting adversarial attacks against TTA algorithms is not trivial, as the capability of the attacker is limited. The authors demonstrate the effectiveness of their proposed FCA attack. However, I have the following questions.

- My biggest concern is the threat model of this paper.

     - One of the main claims of this article is the practicability of the attacks. The paper pointed out that previous attacks like DIA need the label of the test sample. The reviewer agrees that it is very valuable to design a label-free attack. However, in Section 4, the authors claimed that the attacker has white-box access to the latest updated model parameters provided by some malicious insider. In my opinion, this assumption is impractical. First, who is the malicious insider? Can the authors provide a real-world example? 
     - In the threat model of (Cong et al., 2024), they proposed a black-box attack named TePA. TePA does not need access to the parameters of the updated model, which is more practical.
     - The authors also claimed that "existing threat models assume test-time samples will be labeled", and TePA (Cong et al., 2024) relies on access to the entire source dataset to train a substitute model. Actually, TePA also is a label-free attack, and it does not need access to the entire source dataset. It only needs a surrogate dataset that has a similar distribution of the source dataset. I suggest that the authors reorganize the introduction of related work.

- In experiments like Table 1, this paper does not compare the attack success rate with TePA (Cong et al., 2024). 
- This paper considers five target TTA methods, which all need the test samples to be fed in a batch-by-batch manner. However, some TTA methods like TTT [R1] can accept only one sample to adjust the model. I am curious if FCA can attack such kinds of TTA methods.
- The proposed attack seems not to be stealthy. In other words, FCA needs a high poisoning rate (Table 3).
- How many batches does FCA need to effectively decrease the target model's performance?
- In (Cong et al., 2024). they also proposed several defenses, which are not discussed in this paper.

Reference

[R1] Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. ICML 2020.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730560293848

thread_type:
official_review

<thread_object>
creator: reviewer_k7py

<content>
summary:
The authors study the robustness of test-time adaptation (TTA) methods and propose a novel label-free attack. They argue that the assumption of the attacker having true labels on the test set is too strong for the real-life scenarios and without it the existing DIA attack is much less effective. However, the FCA attack proposed in the paper, that uses the idea of neural collapse, remains effective in the label-free scenario which emphasizes the need to further study the TTA robustness.

strengths:
**Originality**: label-free attacks on test-time adaptation seem to be a novel direction. The paper demonstrates the weaknesses of existing defence mechanisms (Section 7.1)

**Quality**: the results are supported with numerous experiments on different datasets (CIFAR10-Cm CIFAR100-C, ImageNet-C) and have theoretical justification (Theorem 1 with proof in Appendix). Ablation study with respect to different loss components is performed (Table 4). 

**Clarity**: the paper is in general well structured.

**Significance**: robustness of test-time adaptation is important in real life applications.

weaknesses:
**Quality:**

1)	The experiments are performed only with the ResNet architecture variants (line 316). Empirical results would benefit from including some other popular architecture e. g. one based on a vision transformer. At least in the Appendix. 

**Clarity:**

1)	Figure 1a: the process in the Figure 1a is hard to grasp for an outside reader. The diagram flow is confusing. Consider making it clearer where the flow starts and ends.  The overview figure has to be more intuitive to help the reader better understand the paper idea and not to confuse them. Consider simplifying it or breaking into several Figures. The top dotted line is missing an arrow on its end.

2)	Please elaborate on how the pseudo labels are defined or add a citation (line 364)

**Typos and minor comments:** 

1)	Line 090 - “as” not needed: This makes our proposed attack **as** a prevalent threat that requires countermeasure

2)	Line 134 - additional “the”: and **the** the set

3)	Line 172 - Why is “Nearest Neighbour Classifier” abbreviated as NCC and not NNC?

4)	Line 265 - the sentence is too complex and poorly constructed, it is hard to understand its point. The more the last layer activations collapse to class mean vector, the likelihood of having a region close to it where the actual prediction deviates if the sample is projected into that region decreases

5)	Line 314 - delete one “use”: we use the use the provided data.

6)	It would be good to include “w/o Attack” value (Table 1) in the Table 4 to better the relative improvement of each variant. The L_nc + L_col would also be good to include for completeness. Perhaps there is some synergy between these two terms that make their combination stronger than the sum of each term.

questions:
1) Why is the perturbation initialized with 0.5 and not e. g. as random noise? (Algorithm 1, step 2) Wouldn’t the optimization benefit from this?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730649861274

thread_type:
official_review

<thread_object>
creator: reviewer_jdou

<content>
summary:
The paper concentrates on the adversarial vulnerability in test-time adaptation (TTA). To explore more practical adversarial attacks, the paper designs a novel attack algorithm, named Feature Collapse Attack (FCA), which can achieve comparable performance without access to labeled test samples. This method consists of three loss designs, which are Nearest Centroid Loss, Feature Collapse Loss and Feature Scattering Loss. Experimental results demonstrate the outstanding performance of the proposed method.

strengths:
+ **New Scenario**: The paper explores a novel scenario, where the attacker has no access to the true labels.
+ **Simple Implementation**: The proposed method is relatively easy to implement and follow.

weaknesses:
+ The number of baseline attack methods is limited. More comparison with multiple attacks is expected.
+ In line 134, there is a repeated word "the".

questions:
The paper considers the worst-case scenario where the attacker has knowledge of the TTA algorithm and has white-box access to the latest updated model parameters provided by some malicious insider. Under this assumption, is it reasonable to hypothesize that the attacker has no access to the true labels?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730720696621

thread_type:
official_review

<thread_object>
creator: reviewer_fwbz

<content>
summary:
This paper proposes a novel label-free adversarial attack against Test-Time Adaptation (TTA) algorithms, called Feature Collapse Attack (FCA). The authors argue that existing TTA attack methods unrealistically assume access to test data labels, making them impractical for real-world scenarios. They address this gap by formulating a new attack that leverages the principles of Neural Collapse (NC) to degrade TTA performance without needing ground truth labels.

strengths:
1. The paper addresses a significant limitation of existing TTA attack research by introducing a label-free approach. This is crucial for practical security evaluations of TTA, as real-world adversaries are unlikely to have access to test labels.
2. The attack design is well-motivated by the theory of Neural Collapse, providing a clear rationale for the chosen loss functions.
3. FCA achieves competitive performance against state-of-the-art attacks that utilize labels, and even surpasses them in some cases. This demonstrates its effectiveness and potential impact.
4. The authors evaluate their attack across multiple datasets and TTA methods, providing a thorough assessment of its generalizability.

weaknesses:
See Questions

questions:
1. **Assumption of White-Box Access:** The attack assumes white-box access to the model parameters, which might not always be realistic. Analyzing the attack's performance under more restricted settings, such as black-box or gray-box access, would provide a more complete picture of its capabilities.
2. **Attack Algorithm:** The attack part of the FCA algorithm is almost entirely borrowed from the PGD in the adversarial sample attack, which will affect the contribution of this paper. Furthermore, does this suggest that the method could be well enhanced by using more powerful attacks?
3. **The Robustness of Victim Model:** The effectiveness of the proposed attack may be limited when targeting inherently robust models **[R]** or those with defense mechanisms (adversarial training), since it builds upon adversarial attack principles. How significantly does the attack's performance degrade when faced with robust or defended victim models?

**[R]** MEMO: Test Time Robustness via Adaptation and Augmentation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732482515575

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Common Response to all Reviewers

comment:
**Choice of White-box Attack Setting**  

Although we acknowledge that black-box attacks against TTA methods is more practical, in this paper we are aiming at understanding the fundamental robustness of  existing TTA methods, which can only be investigated by assuming the worst-case scenario.  We remark that understanding vulnerabilities in white-box settings will facilitate the creation of defense strategies that are also effective in black-box settings. We point out that our assumption of white-box setting perfectly aligns with prior work [1] on understanding the fundamental vulnerabilities of TTA as well as  possible defenses [2].  

  

Besides achieving a fundamental understanding of existing vulnerabilities, white-box attacks are extremely relevant in scenarios with malicious insiders, which are common in settings where the AI/ML algorithms are created locally and then shared with other users. For example, in the case of cellular networks using the Open Radio Access Network (O-RAN) paradigm [3], AI/ML models are trained by local O-RAN providers and then shared in common application marketplaces [4,5]. In such case, it is extremely likely that a malicious O-RAN provider will have perfect knowledge not only of the AI/ML model but also of its updated weights after TTA, for example, to perform additional fine-tuning with local data [12]. 

 
Finally, we have conducted additional experiments in  grey-box settings, i.e., where the attacker knows the model architecture and initial weights. We have presented these results in Table 6. Our proposed attack mechanism remains highly effective in this setting as well, reducing attack effectiveness by about 2\% only across different TTA methods. 


**Regarding exclusion of single sample TTA methods** 

  

We have not considered single-sample TTA methods such as [9,10] since they require fundamentally different considerations in terms of threat model and attack design. In stark opposition, our work studies how unperturbed, non-malicious samples are influenced by the presence of adversarially perturbed samples when true labels are unavailable. 

  

In single-sample scenarios, an adversary would need to craft attacks tailored to each test sample while accounting for augmentations, which aligns more closely with evasion attacks [6,7]. Conversely, our adversarial model is closely related to TTA poisoning [8]. The former case demands a specialized approach and attack mechanism, which deserves a separate investigation. We also point out that single-sample methods often incur substantial computational costs (e.g., multiplying the number of forward and backward passes by the batch size) without significantly improving robustness against distribution shifts [11], which limits their applicability. 

References:

[1] Uncovering adversarial risks of test-time adaptation, ICML 2023

[2] Medbn: Robust test-time adaptation against malicious test sample, CVPR 2024. 

[3] Understanding O-RAN: Architecture, Interfaces, Algorithms, Security, and Research Challenges. arXiv preprint arXiv:2202.01032, 2022.

[4] Managing o-ran networks: xapp development from zero to hero. arXiv preprint arXiv:2407.09619, 2024.

[5] Intelligence and Learning in O-RAN for Data-driven NextG Cellular Networks. IEEE Communications Magazine, October 2021

[6] Explaining and harnessing adversarial examples, arXiv preprint arXiv:1412.6572, 2014.

[7] Towards evaluating the robustness of neural network, IEEE SP 2017.

[8] Test-time poisoning attacks against testtime adaptation model, IEEE SP 2024.

[9] Test-time training with self-supervision for generalization under distribution shift, ICML 2020

[10] Memo: Test time robustness via adaptation and augmentation, NIPS 2022.

[11] Efficient test-time model adaptation without forgetting, ICML 2022

[12]  Open-RAN Gym: An Open Toolbox for Data Collection and Experimentation with AI in O-RAN, WCNC 2022
</content>
</thread_object>

</thread>
<thread>
creation date:
1733081163980

thread_type:
official_comment

<thread_object>
creator: area_chair_ap58

<content>
comment:
Dear Reviewers,

The authors have provided responses - do have a look and engage with them in a discussion to clarify any remaining issues as the discussion period is coming to a close in less than a day (2nd Dec AoE for reviewer responses).

Thanks for your service to ICLR 2025.

Best, 
AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper develops a new adversarial attack on test-time adaptation (TTA) algorithms that does not require access to ground truth labels at test-time. The attack is motivated by previous work on neural collapse and is shown to defeat current defense mechanisms in extensive evaluations on multiple datasets. 

Key strengths of the paper are that it addresses the robustness of an emerging class of methods with a novel approach that is well-motivated by theoretical arguments. Extensive experimental results in white-box and more realistic grey-box attack scenarios show that the attack is effective. On the other hand, the paper does not cover the class of single-sample test-time adaptation methods, and does not include a comprehensive evaluation on the axis of model architectures.

Overall the paper presents a novel, well-motivated attack for TTA that is effective in a reasonable grey-box setting (for which there are few existing attacks) and the AC recommends acceptance. The work will spur the development of improved defenses in the future. The AC also suggests that the grey-box attack results be included in the main text of the final version of the paper.