Breaking the False Sense of Security in Backdoor Defense through Re-Activation Attack
Authors: Mingli Zhu
Siyuan Liang
Baoyuan Wu
Venue: NeurIPS 2024 poster (2024)
Keywords: Backdoor Learning; Backdoor Attack; Backdoor Defense; Adversarial Machine Learning
Primary area: safety_in_machine_learning

Abstract: Deep neural networks face persistent challenges in defending against backdoor attacks, leading to an ongoing battle between attacks and defenses. While existing backdoor defense strategies have shown promising performance on reducing attack success rates, can we confidently claim that the backdoor threat has truly been eliminated from the model? To address it, we re-investigate the characteristics of the backdoored models after defense (denoted as defense models). Surprisingly, we find that the original backdoors still exist in defense models derived from existing post-training defense strategies, and the backdoor existence is measured by a novel metric called backdoor existence coefficient. It implies that the backdoors just lie dormant rather than being eliminated. To further verify this finding, we empirically show that these dormant backdoors can be easily re-activated during inference stage, by manipulating the original trigger with well-designed tiny perturbation using universal adversarial attack. More practically, we extend our backdoor re-activation to black-box scenario, where the defense model can only be queried by the adversary during inference stage, and develop two effective methods, i.e., query-based and transfer-based backdoor re-activation attacks. The effectiveness of the proposed methods are verified on both image classification and multimodal contrastive learning (i.e., CLIP) tasks. In conclusion, this work uncovers a critical vulnerability that has never been explored in existing defense strategies, emphasizing the urgency of designing more robust and advanced backdoor defense mechanisms in the future.

Breaking the False Sense of Security in Backdoor Defense through Re-Activation Attack
Mingli Zhu' Siyuan Liang'
Baoyuan Wu!
'School of Data Science,
The Chinese University of Hong Kong, Shenzhen, Guangdong, 518172, P.R. China National University of Singapore, Singapore
Abstract
Deep neural networks face persistent challenges in defending against backdoor attacks, leading to ongoing battle between attacks and defenses. While existing backdoor defense strategies have shown promising performance on reducing attack success rates, can confidently claim that the backdoor threat has truly been eliminated from the model? To address it, we re-investigate the characteristics the backdoored models after defense (denoted as defense models). Surprisingly, we find that the original backdoors still exist in defense models derived from existing post-training defense strategies, and the backdoor existence is measured by novel metric called backdoor existence coefficient. It implies that the backdoors just lie dormant rather than being eliminated. To further verify this finding, we empirically show that these dormant backdoors can be easily re-activated during inference stage, by manipulating the original trigger with well- delignes tiny perturbation using uni- versal adversarial attack. More practically, we extend our backdoor re- re-activation to black-box scenario, where the defense model can only be queried by the adversary during inference stage, and develop two effective methods, i.e., query-based and wransfer-base backdoor re-activation attacks. The effectiveness of the proposed methods are verified on both image classification and multimodal contrastive learn- ing (i.e., CLIP) tasks. In conclusion, this work uncovers a critical vulnerability that has never been explored in existing defense strategies, emphasizing the urgency of designing more robust and advanced backdoor defense mechanisms in the future.
Introduction
The pervasive application of Deep Neural Networks (DNNs) across safety-critical domains like facial recognition and autonomous driving [23, 36] has underlined their significance and profound impact in industrial and academic spheres. Despite their transformative potential, DNNs are known to be vulnerable to malicious threats [5, 27], which compromise the integrity and reliability of advanced systems. One the representative threats is backdoor attacks [18, 31], where an adversary pre-defines a "trigger" and embeds it within limited training data such that the backdoored model will misclassify trigger-containing inputs into specific target categories while appropriately processing benign inputs.
A successful backdoor attack consists of two stages: (1) the embedding of the backdoor within the model during training; and (2) its subsequent activation during inference stage [62]. To identify [14] and mitigate the harmful impacts of backdoor attacks, substantial efforts have been made ranging from dataset segmentation [7, 50), trigger inversion [53, 56], model pruning [64, 71], and fine-tuning based defenses [30, 67]. While these existing defense mechanisms aim at decreasing the attack success rates (ASR) [59] of corresponding backdoored models, a fundamental question arises: can
*Corresponds to Baoyuan Wu (wubaoyuan (ec.k.eeuuuc))
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

we confidently claim that the backdoor threat has truly been eliminated from the model? In this work, we use the term defense ms)el(s) to denote those models which have initially been poisoned to backdoored models and subsequently defended using some defensive techniques, for convenience.
To answer above question, we introduce an innovative con- cept, backdoor existence coefficient (BEC) to quantify the extent of backdoor presence within models. Using BEC, we can re-investigate the backdoor existence in existing de- fense models [30, 58, 67]. Specifically, the BEC measures the similarity of activation among backdoor-relatee
10
Backdoor Existence Coefficient
Backdoored models without defonse
Tackooored models after dafanse
neurons in the poisoned samples between the backdoored model and its corresponding defense model. Fig. 1 presents the rela- tionship between BEC and backdoor activation (indicated by ASR) across three different attack and defense methods for comparison. In this figure, distinct shapes and colors de- note various attack and defense methods, respectively. As depicted in the figure, even though the ASRs decline nearly to zero which implies that defense models perform com- parably to clean models, the BECs in the defense models remain significantly high. This notable observation implies that the original backdoors just lic dormant rather than being eliminated in defense models.
cantaddo
toput/oore tattack)
No delence
Raul idefenss)
0:0
misan icoterual
Clean model
model
-0.05
0.4
Backdoor Activation Rate
Figure 1: Comparative analysis of back- door existence coefficient and backdoor activation rate across different models.
Since the original trigger fails to activate the original backdoor, is it possible to unearth a variant of the original trigger that capable of re-activating the backdoor? Given that in real-world scenarios where the adversary cannot modify the defense model, our objective is to modify the original trigger, thereby facilitating backdoor re-activation in defense models during inference stage. To verify this feasibility, we formulate the backdoor re-activation task as constrained optimization problem with the goal of searching for minimal universal adversarial perturbation on the original trigger. Consequently, this general technique can be seamlessly combined with any prevailing backdoor attacks to re-activate backdoor effect in defense models in their inference time. To demonstrate the real-world threat posed by backdoor re-activation attack, we also expand our method to black-box and transfer attack scenarios, where adversaries limited querying the model without access to its internal mechanisms. Nowadays, multimodal contrastive learning (MMCL) has impressed us with its performance across range of tasks and backdoor threats in MMCL have also been broadly studied. In this work, we consider both image classification and multimodal tasks, demonstrating the universality and adaptability of our approach. Extensive experimental results on nine different attacks and eight state-of-the-arr defenses across four benchmark datasets and three model architectures demonstrate the effectiveness of our method. Our work reveals a new vulnerability in existing defense strategies, emphasizing the need for more robust and advanced defense mechanisms in the future.
Inspired by above observations, we pose question:
Our main contributions are threefold: 1) We re- ie-iessstigate existing defense methods, and reveal that the original backdoor still exists in the model even after defense, though it cannot be activated by the original trigger. 2) We develop a novel optimization problem to re-activate the original backdoor during inference by perturbing the original trigger, under white-box, black-box, and transfer attack scenarios. 3) We demonstrate the effectiveness of the proposed method with extensive experiments on both image classification and the emerging multi-modal contrastive learning tasks.
2 Related work
Backdoor attacks. Backdoor attacks [15, 16, 22, 48, 55, 59, 75] are a significant security threat in DNNs. As summarized by Wu et al. [60, 62], a successful backdoor attack consists of two components: backdoor injection during pre-training or training stage, and backdoor activation during inference stage. Backdoor injection could be divided into data poisoning attack at pre-training stage and controllable attack at training stage. During data poisoning attack, an adversary releases a poisoned dataset to plant backdoors. Representative works include BadNets [18], Blended [10], LF [68], SSBA [31], and Trojan [37]. For gaining-controllabe attack [70], an adversary takes control of the training process to optimize triggers and inject backdoors. Notable examples are Input-Aware [40] and WaNet [41]. In inference stage, the adversary uses the poisoned samples to activate backdoors in
the backdoored model, thereby achieving a successful attack.

---

While backdoor attacks are prevalent in supervised learning, backdoor threats also exist in domain of multi-modal contrastive learning (MMCL) [32, 33]. Carlini et al. [6] are the pioneers to unveil backdoor threats in MMCI,, demonstrating that as few as 0.0001 % of images can trigger a successful attack. More recently, sophisticated approaches have been introduced [2]. For instance, VQA [52] is designed for the multi-modal visual question answering task, while BadCLIP [35] shows that their attack can persist in effectiveness against backdoor defenses.
While a variety of attack methods have been proposed, they primarily focus on enhancing attack success rate during backdoor injection stage and employ the same trigger activate backdoors in inference stage. They did not consider that the model might be fine- tine-tuned or defended by users, and the original triggers fail activate backdoors in inference stage. Although Qi al. [42] attempted to enhance backdoor signal during inference stage, they did not consider defensive techniques in depth, and their attack lacks universality. this work, we focus on general backdoor attack method during inference time, researching on how to re-activate the dormant backdoors in defense models.
Backdoor defenses. A range of works [21, 24, 29, 39, 69] focusing on backdoor defenses have been put forward to address the threat of backdoor attacks. Considering defense stages, four main categories emerge: pre-prooe pre-proeesiing defenses, training- moining-stage defenses, post-trainne post-training defenses, and inference stage defenses [61]. Pre-processing defenses [7, 24, 76] aim to filter out poisoned samples from poisoned dataset. Training-stage strategies [9, 21, 29, 57] consider that the defender has access to both training samples and the model, and mitigates backdoor effects during training process. They leverage discrepancies between poisoned and benign samples to filter out suspicious instances. Post-training defenses [11, 39, 54, 69, 73] focus on removing backdoor effect from backdoored models through pruning potential backdoor neurons [64]. backdoor triggers reversion and unlearning [53], or enhancing fine-tuning processes for backdoor mitigation [72]. Inference stage defenses aim at preventing backdoor activation with samples detection samples recovery techniques [76]. In the domain of MMCL, there are a range of works [3, 34, 65]. CleanCLIP [3] is the first to defend the MMCL model using MMCL loss and self-supervised learning within each modality with clean samples. Additionally, RoCLIP [66] introduces a robust pre-training approach, which focuses on disrupting the link between poisoned image-caption pairs. In this work, focus backdoor re-activation attack and thus mainly consider our attack against post-training backdoor defenses.
3 Methodology
In this section, we introduce our threat model and methods for image classification task for clarity. For the formulation and methods for multimodal contrastive learning, please refer to Appendix A.
3.1 Threat model
Notations. For the image classification task, the training dataset is D ((x(0), XX where X' C Rd and y {1,..., K} are input space and label set, respectively. Given input æ. we define a deep neural network with I. layers as:
f(x) f(L) o f(l- o...o (x),
where (() is the function in the 1t layer of the network, S < T.. The feature map of the 1hh layer is denoted as m (x:) E IR Xwi, and fu represents the logit of the 1 class.
Before introducing methods, we first outline the pipeline of backdoor attack and defense. As summarized in [62] and shown in Tab. 1, the whole pipeline of backdoor attack and defense involves four stages:
I. Pre-training stage: An adversary conducts data poisoning backdoor attack, which involves revising small fraction of to generate poisoned dataset D, = ((xg (4) 1) by injecting trigger & into the image and changing the corresponding label into target label I.
II. Training stage: An adversary controls the training process to inject backdoors into model fe.-
III. Post-training stage: defender receives the poisoned model, and can gather some benign samples to remove the backdoor effect from the model, denoted as feo-
3

---

Table 1: Illustration of the pipeline of backdoor attack and defense.
Stage
Task description
Input/Output
Goal
Reference
Clean model training
Di Di Dr fo./fo, 2.C. (on
fo. (x)
y. fae ( # t
I: Pre-training II: Training
Backdoor injection
fa. (x) = y, fa. TE t
III: Post-training
Backdoor defense
fa, x = y, fop, [Eg t
IV: Inference
Backdoor re-activation
Jop (z) y. .op (1Fer)
IV. Inference stage: With the defense model fo,- the original trigger fails to activate the backdoor, i.e., fon (xt) # t. The goal is re-activate backdoors, L.e. fe, (zggr) = t, where g = + Ag.
Existing backdoor attacks primarily focus on achieving high attack success rates (ASR) in backdoor injection stages (I and II), with little consideration for the defensive impact in stage III. Given the failures of ag in attacking fe.. our work focuses on the backdoor re-activation attack in stage IV.
3.2 Backdoor existence coefficient
While the model performance in Tab. suggests that fo, and for analogous, argue that in terms of the backdoor effect, fop and So, are actually more closely aligned, which indicates the persistent existence of backdoor in model fop To verify this, need metric measure the quantity of backdoor existence within a model. An effective indicator should be capable of quantifying the similarity of backdoor effect between backdoored model fe, and the target defense model fe, across the entire models. To achieve this, we propose a new metric, Backdoor Existence Coefficient (BEC), which is calculated through the following three steps:
1. Backdoor neuron identification: Firstly, we need to identify backdoor-related neurons. Zheng et al. [71] proposed Trigger-activated Change (TAC) to quantify the correlation between backdoor impact and neurons (see Appendix C for details). With this metric, backdoor-related neurons in fa, are identified for each layer. Thus, the feature maps corresponding these (1)
neuron indices are selected for each model, denoted as M (xee), mD (ate). and mi (xx). respectively. Denote the feature maps across dataset Dp as m (1) (Dpp E IR",,x(
X(a xxxx))
2. Backdoor effect similarity metric: order to measure the backdoor effect similarity between models, we employ Centered Kernel Alignment (CKA) [25] (see Appendix C for details) to quantify the similarity between these matrices. The similarity backdoor effects between fo. and for, calculated through the use of corresponding features, be computed as:
S (Dp) = CKA (mb) (Dp), m (Dp)).
and { (Dp) is computed accordingly.
3. Backdoor existence coefficient computation: The BEC the average of normalized backdoor effect similarity across all layers. By assigning the BEC of So, a value of and fac value of O, the computation can proceed as follows:
(D,)) (Dpp -(())))()(Pp))) [0. 1].
1 N
PBEC(fer fer, fo.: Dp)
(3)
A (D,)) -())))
Remark. ( (Dp) = The second and third arguments in paEC serve as reference models to measure the backdoor existence of the model corresponding to the first argument fe.. Denote as Paac for simplicity. The higher the value (fr)(fe)): the stronger the
Paec(Jop so: existence of backdoors in the model. We utilize BEC to signify backdoor existence and employ ASR to quantify the extent of backdoor activation. As shown in Fig. 1, the BEC remains consistently high across various defenses, despite backdoor activation being low.
3.3 Backdoor re-activation attack
Motivated by the fact analyzed above that the original backdoor still exists the defense model for- here explore the possibility to re-activate the backdoor during inference. Since the adversary cannot

---

modify fe, during inference, one feasible solution is to modify the original trigger& &. Specifically, we propose to pursue a new trigger& & by perturbing &. 5 { + Åg. such that 5 could re-activate the original backdoor, i.e., Sop ( = L. In the following, we will present how to obtain a successful trigger perturbation Aę under white-box, black-box, and transfer attack scenarios, respectively.
White-box backdoor re-activation attack. In white-box scenario, the adversary has access to the parameters of ʃ but cannot manipulate them. In this case, we could obtain Dg by solving the constrained optimization problem min A A gi f), where
( Eisl cilastte)
Lior Dp:
E (c - Alog (ag.t)EDP
1 max Right
(4)
where ||. 11pp means lp norm, P is the perturbation bound, Lcu is cross-entropy loss, and >> is a hyper-parameter. This problem be easily solved using project gradient descent (PGD) [38].
Black-box backdoor re-activation attack. Although the re-activation attack under the white-box scenario easy implement, may be impractical. Thus, we also consider the practical black- box scenario, where the adversary lacks information to the defense model and can only query the model and obtain the predicted score. Consequently, the above problem (4) is longer directly optimized by the PGD algorithm. Inspired by existing black-box adversarial attacks [1, 8], we propose a novel random search based optimization algorithm. Specifically, we extend the query-based black-box box adversarial attack method Square Attack [1] that was designed for optimizing sample- specific perturbation, to solve problem (4), dubbed Universal Square Attack. Its overall procedure summarized in Alg. in Appendix.
Transfer- -baed backdoor re-activation attack. In addition to the query-based black-box attack, we also explore transfer-based attack scenario. In this scenario, the adversary trains backdoored model fe. and releases it to downstream users. The user receives model fe.- and obtains a defense model fe, based on fe. by some post-training defense. Thus, the adversary does not know the exact defense method, but has full information about the original trigger & and model fe, which has same model architecture as for- The adversary also has restricted query limits. Consequently, leveraging transfer attacks becomes a viable strategy for attacking. The main idea is that the adversary can imitate defense process get some defense models for, themselves, where i 1,..., M. Then these defense models can serve as surrogate models to generate perturbation De as follows:
A arg min ||
Σ LeoelAg; Dp. Soo.)-
Overall, we propose a universal backdoor re- re-activation attack that aims to enhance the performance of existing backdoor attack methods during inference. We have explored three scenarios -eie--hiteebree attack (WBA), query-based black-box attack (BBA), and transfer attack (TA). Besides, we would like to emphasize again that the proposed attack can be naturally extended multi-modal learning tasks, other than the classification task demonstrated above. The details are presented in Appendix A.
4 Experiments
4.1 Implementation details
Models and datasets. For image classification task, we evaluate all our attacks on three benchmark datasets CIFAR-10 [26]. Tiny ImageNet [28]. and GTSRB [49] over two network architectures, PreAct-ResNet18 [20] and VGGI9-BN [47]. We utilize the setup in BackdoorBench [59]. For MMCL task, we use the open-sourced CLIP model from OpenAl [44] as the pre-trained model. Following the setting of CleanCLIP [3], the model poisoned on the CC3M dataset [45] and subsequently tested through pero-shot evaluation on ImageNet-1K validation set [13].
Backdoor attacks. For image classification task, we adopt seven widely used backdoor attacks including: (1) five data poisoning attack: BadNets [18], Blended [10], LF [68], SSBA [31], and
5

---

Trojan [37]: and (2) two controllc attacks: Input-Aware [40] and WaNet [41]. We follow the default attack configuration as in BackdoorBench [59] and the 0t label is set to be the target label. For MMCL. task, we adopt four backdoor attacks including: BadNets, Blended, SIG [4], and VQA [52]. In data poisoning phase, 1500 samples out of 500K image-text pairs from CC3M dataset are poisoned and the target label is banana as in [3].
Backdoor defenses. For image classification task, we adopt six state-of-the the---o--hhh-ttt- art post-training defense methods: NC [53]. NAD [30], i-BAU [67]. FT-SAM [72]. SAU [58]. and FST [39]. For MMCL task, we consider two defense methods: (1) FT [3]: fine-tuning the model with multimodal contrastive loss using clean dataset; and (2) CleanCLIP [3]: fine-tuning defense method for CLIP models. All the detailed introduction about the above attack and defense methods can be found in Appendix D.
Implementation details. At backdoor injection phase, the poisoning ratio is set to 10%. following the configuration in BackdoorBench [59]. At defense phase, 5% clean samples are given to defend models. At backdoor re-activation phase, we consider defense models as our target model. The adversary is given 2% (i.e. 1000) poisoned samples to conduct attacks. We consider both foc and & norm attacks, and the perturbation bounds are set to 0.05 and 2, respectively. The loss hyper-paramere, A is I for all our experiments. For query-baset black-box attack, the maximum query limit is 10,000 for each image. For transfer attack, the adversary is given 10% poisoned samples to conduct backdoor re-activation attack. The & norm bound is set to I for transfer attack. We simply assume three surrogate models can be used and we just divided these defenses into two groups: (1) NC, NAD, i-BAU; and (2) FT-SAM, SAU, FST. Specifically, we generate perturbation in each group and test the ASRs in the other group. All the ASRs are tested on testing dataset. For MMCL tasks, we assume the adversary lacks knowledge of the downstream task. Therefore, attacks exccuted in the upstream task for both white-box and transfer attacks and subsequently tested in downstream zero-shot task. More details about the implementations can be found in Appendix D. We have provided the PyTorch² implementation of our method on Github.
4.2 Main results
Backdoor re-activation attack. Tab. 2 shows the performance of our backdoor re-activation attack under white-box attack (WBA) and query-based black-box attack (BBA) settings in comparison with ASRs of original backdoored models (No Defense) and defense models (Defense). By observing the table, the following profound insights emerge: (1) Compared to defense models, our attacks show a striking level of efficacy. Both our WBA and BBA have exhibited an impressively absolute improvement of 76.94% and 42.95% on average, respectively when compared against defense mechanisms, which shows the effectiveness of our re-activation attack method. (2) The close performance of our WBA compared to "No Defense" underscores the efficacy of our backdoor re-activation mechanism, affirming the recoverability of the backdoors in defense models. By setting WBA as an upper bound for backdoor recovery, the more realistic BBA reveals substantial attack performance. Despite a gap between the two approaches, we posit that this disparity can be lessened through sophisticated black-box attack strategy. (3) In terms of specific defenses, our attack against SAU and FST exhibits relatively poor ASRs. This suggests that SAU's backdoor removal efficiency is significant, which aligns with the subsequent analysis of Fig. 3. In contrast, FST's BBA seems comparatively subdued. It may be attributed to the reinitialized FC layers, effectively cutting backdoor activations. These insights serve as valuable pointers for crafting defense strategies in the future.
Backdoor re-activation attack via transfer attack. In this experiment, we group these defenses into two distinct groups: (1) weak group (NC, NAD, i-BAU) and (2) strong group (FT-SAM, SAU, FST) to better to observe the impact of defense methods on the performance of transfer-based acaiiiatttion attacks (TA). Two key findings emerged from results in Tab. 3: (1) Transfer attacks generally exhibit strong performance in comparison with results in Tab. 2. The ensemble attack strategies applied on the weak group demonstrate better attack effectiveness on strong defense models than that in BBAs. (2) Utilizing ensemble strategies on strong defense methods results in remarkably effective ASRs on weak defense models, surpassing even the efficacy of WBA Tab. This outcome raises concerns: if adversaries simulate stronger defenses to derive substitute models for launching transfer attacks, it could lead to serious security threats.
:/ http//gttttte wtt.:ub..oommm/iieaaaaaaaa htt.:p/r...attt..c..o/ooooooooar

---

Table 2: Performance (%) of backdoor re-activation attack on both white-box (WBA) and black- box (BBA) scenarios with -orm bound P = 0.05 against different defenses with CIFAR-10 on PreAct-ResNer18. The best results are highlighted in boldface.
Attacks
No Defense
NC [53]
NAD [20]
I-AU [67]
FE-SAM [721
SAU [St]
FST
Defense
BBA
Defense
WBA
BBA
Delense
WDA
DBA
Detense
BBA
Defense
BBA
Defense
BodNets [15]
93.79
2.01
96.78
27.91
1,96
94.78
49.56
4.45
97.42
$4.37
1.53
94.71
$1.25
1.30
93.10
37.91
1.45
97.93 42.69
Blended
99.76
99.76
99.13
47.64
99.82
14.14
26.83
99.63
12.07
99.56
57.29
98.37
99.62
Input Aware [40]
99.30
0.70
92.04
54.33
0.92
93.80
70,44
0.02
21.78
19.56
1,07
96.19
90.15
1.26
$5.39
22.25
0.00
90.72 44.65
LF1668
99.06
99,10
80.51
75.47
99.41
17.01
11.99
99.64
97.40
59.28
90.74
98.38
SSBA [31]
97.07
97.07
99.90
94.38
70.77
99.72
38.53
2.89
91.29
70.71
4,06
92.80
69.18
2.16
89.86
38.59
0.54
94.11 52.71
Trojan [57]
99.99
2.76
95.36
45.57
5.77
96.38
60.87
0.54
89.58
40.18
4.12
96.18
69.88
1.39
87.61
47.37
8.93
97.35 80.47
WiNer [41)
98.90
98.
100.00
90.64
0.73
96.21
77.65
0.85
94.67
75.01
0.96
94.95
78.66
95.33
60.
0.25
97.56
Asg 98.26
57.18
97.62
71.64
29.04
97.16
54,04
6.80
84.77
60.29
4.35
95.97
75.10
2.09
91.48
43.23
2.40
96,89 55.27
Table 3: Attack performance (%) on target models of transfer- re-activation attack (TA) with l--no g--norm bound p = against different defenses with CIFAR-10 on PreAct-Res
Attack
No Defense
[53]
NAD [30]
I-BUU [67]
FT-SAM [72]
SAU [58]
PST [19]
Defense
Defense
TA
Defense
TA
Defense
Definee
TA
Defense
RodNNet [15]
93.79
2.01
95:41
1.96
98.47
4.48
97.90
99.17
97:42
1.30
90.17
1,46
96.21
Blended [10]
99.76
99.76
100.00
47.64
99.98
26.83
99.83
98.64
99.63
5.20
93.36
0.20
24.07
Aupur-Aware [A0]
99.30
0.70
99.98
0.92
99.98
0.92
99.77
96.92
21.78
1.26
15.56
0.00
95.28
LF (68]
99.05
99.06
99.93
75.47
99.84
11.99
98.35
93.85
99.04
2.49
96.62
5.43
80.09
SSBA [31]
97.07
97.07
99:27
70.77
99.38
2.89
20.44
98.06
91.29
2.16
95.03
0.54
76.21
Trajas [S7]
99.99
2.76
99.76
5.77
99.09
0.54
96.18
96.57
59.58
1.39
83.67
8:93
21.79
WaNel [41]
0.88
08.99
0.82
0.26
Avg
98.26
57.18
99.16
29.04
99.51
6.80
85.18
97.44
$4.77
2.09
90.55
2.40
70.33
Effectiveness of attacks on CLIP models. Tab. 4 lists the performance of our backdoor re- activation attack under white-box attack (WBA) and transfer-based attack (TA) on the CLIP model. Our attacks yield significant improvements, with ASR enhancements of 34.87% and 43.35% on average, respectively, compared to defense models. The results for TA and WBA are very close. One possible reason is that the similarity between the FT and CleanCLIP methods leads to strong transfer performance. We advocate for the development of stronger defenses on CLIP to combat attacks. Due to space constraints, attack results and analysis on Tiny ImageNet (Tab. 12) and GTSRB (Tab. 13) datasets, and results on VGG19-BN models (Tab. 14) are provided in Appendix E.
4.3 Ablation study
Influence of norm bound and norm type. We studied the impact of norm type and norm bound on the attack performance. The results are shown in (a) and (b) of Fig. 2. It can be observed that it is difficult to achieve high success rates under smaller norm bounds. However, when the norm bound is sufficiently large, the attack effectiveness converges and approaching nearly 100% for both -nom and fn-mrrm types against all defense models.
Influence of the size of poisoned samples. We investigated the impact of the size of poisoned samples on attack performance for Blended attack. As shown in (c) and (d) of Fig. 2, increasing the number of training samples in WBA shows significant improvement in attack results. However, in
Table 4: Performance (%) of our attack on both white- Table 5: Our attacks (%) on defense
box (WBA) and transfer-based (TA) attacks with &x -orm bound = 0.05 against different defenses with ImageNet- with norm bound p 0.05 under IK on CLIP. Best results are highlighted in boldface.
models in comparison with clean ones
different model structures and datasets.
Attack
No Defense
Defense
FT [3]
WBA
TA
Defense
CleanCLIn IP [3]
WBA
TA
Semp
WBA
Clean Model
BBA
Detense Model
WBA
BBA
BadNets [18]
96.65
64.60
82.05
82.73
17.29
57.76
47.30
ResIB+CIFAR-ID 10
85.00
56.98
93,92
59.93
Alended [10]
97.71
49.77
96.57
98.64
18.57
89.61
72.65
ResI8-Tiny
39.76
71.04
40.81
SIG [4]
77.71
30.91
92.56
87.99
21.68
87.04
82.55
ResIS+GTSRB
53.33
50.87
67.14
61.81
Trojan [52]
98.21
82.07
97.14
97.46
49.82
87.43
78.25
VGG+CIFAR-10
68.80
43.60
85.15
51.04
Avg
92.57
56.84
92.08
91.71
26.84
80.46
70.19
Avg
61.72
41.37
79.31
53.40
7

---

I. Norm
Norm
White Box Attack
Black Box Attack
1.2
3.30
6.45
5.0
ASR.
0.6
NC
NC
NAD
NAD
NAD
LEAU
LEAU
H-AUU
SAU
FT-AMM
0.32
SAU
FT-SAM
121
FTM
54
FT:SMM
SAU
FST
FST
6.80
Dalasel
(o)
Figure 2: (a) and (b) show attack results under different norm types p and bounds for WBA. (c) and (d) show attack results under different number of poisoned samples for WBA and BBA.
Table 6: Detection performance (TPR %) on dif- Table 7: Performance (%) against test-time de- fenses.
ferent (model, poisoned samples) pairs.
SCALE-UP
Trrmsoa Peac
68.6
Josen Jasen Paos
Defense
SCALE-UP
SCALE-UP ACC
STRIP ASR
STRIP ACC
ZIP
ZIP
BatNien
Sentihie
52.7
3.8
2.2
622
0.9
Attack
ASR
ASR
ACC
STRIP
88.3
0.7
5.5
10.3
65
SUALE
731
354
BadNets
83.4
Teejan
Blended
34.1
46.2
49.2
92
48.1
81.5
STRIP
999
79
43
the BBA setting, the ASRs remains relatively stable and does not exhibit significant enhancements with the increase of training samples. This suggests that the difficulty in BBA lies in finding good universal perturbation, especially when dealing with large number of training samples. However, the successful attacks with minimal samples also highlight the significant potency of the attack method.
Attacks performance against clean models. To demonstrate the specific vulnerability defense models, we contrast the performance of our attacks the defense models in comparison with clean models. Tab. 5 provides a summary of our method's performance across all backdoor attacks and defense methods, in comparison of the ASRs clean models. It can be observed that, although some effectiveness is achieved on the clean models, the vulnerability of defense models is significantly higher than that of the clean model, with this gap being pronounced in particular defenses. This indicates that defense models are indeed more fragile in comparison with clean models.
4.4 Further analysis
Backdoor existence analysis. We provide more experimental demonstration the existence of backdoors in defense models. We employ our BEC metric to quantify the existence of backdoors in all defense models and visualize the relationship between BEC and backdoor activation rates, as depicted (a) of Fig. 3. We observe that backdoors persist across defense models, albeit with low backdoor activation rates. The BECs in SAU, SAM, and i-BAU are relatively low, while FST exhibits a notably high BEC. This contrast may stem from the former's optimization objectives resembling adversarial training, whereas the latter primarily disrupts activations through layers e-iiitialization.
Relationship between BECs and ASRs. We validate the relationship between the ASR of re- activation attack (WBA) and the residual of backdoors. We computed the Pearson Correlation Coefficients (PPC) between BECs of different defense models and their white-box ASRs among all attacks, as shown (b) of Fig. 3. is evident that most cases, there is strong correlation between the two. In other words, the more backdoors remain in models, the easier it is for attacks to succeed. Therefore, our metrics can serve as an indicator of backdoored model security.
Feature map visualization. Here we visualize the feature maps between different models to directly observe their similarities. Fig. 3 (c) displays the visualizations of activations from the final four convolutional layers of three models, sorted descending order according to backdoored model's TAC value, with each subplot arranged from top to bottom. It can be observed that the defense model and backdoored model exhibit similar patterns: highlighting activations in backdoor-related neurons. This directly indicates the persistence of backdoors within defense models.

---

Backelor Existence Coel feeieeeent
940
10
0.85
Layen Layen Layork lase
Clean
Cormaation
6d-
Attock
0.31
Defense
Bended
Eal Pestare 50th AMtaCks
Activation Rate
16)
Figure 3: (a). Visualization of the correlation between backdoor activation rate and BEC. (b). Pearson correlation coefficients of ASR and BEC under different attacks. (c). Visualization of feature maps.
Attack against test-time detection and defenses. Given that our attack is conducted during the test phase, it is essential to assess whether it can evade backdoor detection and defenses at test phase. To this end, we test three test-time backdoor detection methods: SCALE-UP [19], SentiNet [12]. and STRIP [17], well as three test-time defenses: STRIP [17]. ZIP [46], and SCALE-UP [19].
The detection task requires two input arguments, including the model and the query datasets. We evaluate five pairs, including (the original backdoored model fa. the original poisoned dataset Dp), (the defense model with FT-SAM fD.FTSAM. Dy), (fD.FT-AM: the re-activation dataset D Pp.Aç). (the defense model with SAU fD.SAU, Dp), (fD.SAU. the re-activation dataset Pp.De). The result in Tab. 6 shows that our attacks not markedly increase the TPR compared to the other two pairs. More detection performance our BBA and TA are shown in Tab. 18 in Appendix.
Tab. shows the defense results. It shows that attack maintains certain level of ASR against ZIP. However, for SCALE-UP and STRIP, there is a significant drop in ASR. Meanwhile, the model's ACC is also notably low. This experiment highlights the potential for future attack method designs aimed at evading test-time defenses. Possible strategies could include techniques to better align with feature distributions clean data and to avoid triggering excessively strong activations.
Attack against adaptive defense. Consider- ing defenders are aware of adversary' strate- gies, they can introduce random perturbations for queries so as to disrupt the adversary's 's ability. We assess both adversary's s ASR and the model accuracy clean samples under varying per- turbation bound. As depicted in Tab. 8, minor noise has slight impact on ASR. However, with larger noise amplitudes, despite failed attacks, the model's accuracy significantly affected.
Table 8: Results (%) against adaptive defense.
Defense
FST [39]
FT-SAM [72]
i-BAU [67]
SAU [58]
Noise
ACC
ASR
ACC
ASR
ACC
ASR
ACC
ASR
82.97
82.24
88.35
70.89
89,53
88.04
79.02
88.38
84.11
0.03
82.59
73.03
84,84
86.36
81,32
75.83
83,33
67.42
0.04
75.87
56.76
78.04
81.60
75.51
72.72
76.19
54.40
0.05
67.15
58.17
70.12
74.57
68.09
72.28
67.95
56.74
4.5 Comparison among OBA, RBA, and gUAA
To verify that our re- activation attack method finds a highly correlated backdoor with the original backdoor, and to distinguish it from general universal adversarial perturbation attack (gUAA), systematically compare the original backdoor attack (OBA), our re-activation attack (RBA), and gUAA from three key perspectives.
Table CKA scores between OBA, RBA, and gUAA.
To facilitate the understanding of our analysis, we firstly clarify the definitions and settings. OBA refers an existing backdoor attack following the standard backdoor injection and activation pro-
Defense
+-BAU
FTSAM
Attack L
Sesvcos
SytouoBA
SuncosA
Sypuuass
BodNess
8.507
0.192
0.170
0.599
0.194
0.109
Blended
0,712
0.196
092
0.712
0.197
0.193
cess; RBA means that, given the defense model, we aim to re-activate the injected backdoor of OBA by searching for a new trigger ç', starting from original trigger &, based on some original poisoned samples D,; gUAA refers to targeted universal adversarial perturbation attack (same class as OBA and RBA) where, given fog, we aim to find perturbation starting from clean samples D.- The searched UAP is denoted A. and the perturbed dataset Dc.. Our analyses follows:

---

Table 10: ASR (%) of RBA and gUAA with differ-Table 11: ASR (%) of OBA, RBA, and gUAA ent query numbers.
under different b0 noom of random noise.
Attack+Defense
Query number
1000
3000
5000
7000
Norm
0.03
Blendad+i-BUU
RBA
77.3
89.3
92.1
94.6
OBA
Blended-NAD
L.+NADD
99.1
99.8
59.8
98.9
98.4
99.6
97.3
98.6
gUAA
14.2
41.4
49.5
56.4
RBA
Bleaded+NAD
I.F-NAD
99.4
99.8
99.7
99.1
98.
98.7
84.0
96.6
Blembed-FT-SAM
gUAA
RBA
41.1
16.3
77.4
42.2
56.5
79.8
85.6
65.5
gUAA
Bleaded+NAD
L.+NAD
95.3
96.5
92.7
89.5
55.8
79.4
35.4
16.7
We analyze the backdoor activation mechanism in each attack. As demonstrated in Sec. 3.2, we adopt the CKA met- ric to measure backdoor effect similarity between models. Here calculate the (1) ('p..e), m (Dp)).
Activation mechanism of backdoor effect:
following three CKA scores: SRBA.OBA (1) (De.), m (1) (Dp)). (D))).
N E CKA(MB
S SQUAA.OBA
N E CKA(MD
SRBAQUAA
EE CKA(M ( (Dp.d.). RRBA across all attack- -effene pairs. This demonstrates that the backdoor activation mechanisms between RBA and OBA are highly similar, and both differ significantly from that of guAA.
mp (D..))) As shown in Tab. 9, SRBA.OBA >> SqUAA.OBA &
(1)
Starting from the original trigger &, it is easier and faster to find new trigger g that achieves a high attack success rate (ASR): As shown in Tab. 10, given the same query numbers, the ASR of RBA is much higher than that of gUAA, and RBA increases in speed faster than gUAA. This indicates that RBA is much closer to OBA than gUAA.
Compared to A, both the original trigger & and the new trigger g' are more robust to random noise: We discovered that the robustness to random noise can distinguish the trigger of an intended backdoor from the trigger of natural backdoor (i.e., gUAP). Specifically, we perturb É. ç,, and A with the same level of random noise and record the ASR of these attacks. As shown in Tab. 10, both OBA and RBA are more robust than gUAA. This confirms that RBA produces an intended backdoor trigger similar to OBA, rather than gUAP.
In conclusion, our analyses verify that our RBA method finds a backdoor highly correlated with the original buckdoor, rather than a less correlated one (new backdoor) general UAP (natural buckdoor). Thus, we assert that our RBA effectively re-activates the original backdoor.
Conclusion
This paper illuminates the false sense of security in backdoor defenses and proposes a new threat to enhance existing backdoor attacks in inference-time. Our pioneering introduction of the backdoor existence coefficient unveils the residual presence of backdoors within defense models. Moreover, propose novel optimization problem re- re-activate these dormant backdoors and craft distinct algorithms tailored specifically to white-box, black-box, and transfer attack scenarios. The proposed method can be integrated with existing backdoor attacks to boost their attack success rate during the inference stage. The efficacy of our method evidenced through exhaustive evaluation on both contrastive learning tasks. The threat revealed by this study
image classification and underscores the pressing need for designing advanced defense mechanisms in the future.
Limitations and future work. Despite the efficacy of our proposed method, its effectiveness is limited when confronted with defenders that inject noise into each query. Promising future work is to devise more sophisticated attacks that can bypass this defenses. Another limitation is that if defenders aim to decrease both ASR and BEC, our attacks will become challenging, even though directly optimizing the BEC is not feasible. This serves as another direction for our future work.
Broader Impacts. As deep neural networks sourced from untrusted origins face significant risks from backdoor attacks, this study provides a meaningful exploration into the false security in backdoor defense models. This could spark further advancements in backdoor defenses. Nonetheless, the potential misuse by ill-intended entities should be cautiously considered.
10

---

Acknowledgments
This work is supported by Guangdong Basic and Applied Basic Research Foundation (No. 2024B151502000 5555000955) National Natural Science Foundation of China (No. 62076213), Shenzhen Science and Technology Program under grants (No. JCYX2021000900005700000 Key Laboratory of Intelligent Digital Economy Security, the National Research Foundation, Singa- pore, and the CyberSG R&D Programme Office ("CRPO"), under the National Cybersecurity R&D Programme ("NCRP"), RIE2025 NCRP Funding Initiative (Award CRPO-GCI-NTU-002)
Longgang District

---

References
[1] Maksym Andriushchenko. Francesco Croce, Nicolas Flammarion, and Matthias Hein. Square attack: query-efficient black-box adversarial attack via random search. In European Conference on Computer Vision, 2020. 5, 17, 21
[2] Jiawang Bai, Kuofeng Gao, Shaobo Min, Shu-Tao Xia, Zhifeng Li, and Wei Liu. Badclip: Trigger-awar aware prompt learning for backdoor attacks on clip. In CVPR, 2024.
[3] Hritik Bansal, Nishad Singhi, Yu Yang, Fan Yin, Aditya Grover, and Kai-Wei Chang. Cleanclip: Mitigating data poisoning attacks in multimodal contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 112 123, 2023. 3, 3,5, 6, 7, 16, 19, 20, 21
[4] Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. In International Conference on Image Processing, 2019. 6, 7, 20
[5] Battista Biggio and Fabio Roli. Wild patterns: Ten years after the rise of adversarial machine learning. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, pages 2154 2156, 2018.
[6] Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. In International Conference on Learning Representations, 2022. 3
[7] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig. Benjamin Edwards, Taesung Lee. Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks deep neural networks by activation clustering. In Workshop on Artificial Intelligence Safery, 2019. 1,3
[8] Jinghui Chen and Quanquan Gu. Rays: A ray searching method for hard-label adversarial attack. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 1739- 1747, 2020. 5
[9] Weixin Chen, Baoyuan Wu, and Haoqian Wang. Effective backdoor defense by exploiting sensitivity of poisoned samples. In Advances in Neural Information Processing Systems, 2022. 3
[10] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv e-prints, pages arXiv-1712, 2017. 2, 5, 7, 19, 20, 22, 23
[11] Zhenzhu Chen, Shang Wang, Anmin Fu, Yansong Gao, Shui Yu, and Robert Deng. Linkbreaker: Breaking the backdoor-trigge link in dans via neurons consistency check. IEEE Transactions on Information Forensics and Security, 17:2000-200 2014, 2022. 3
[12] Edward Chou, Florian Tramer, and Giancarlo Pellegrino. Sentinet: Detecting localized universal attacks against deep lcarning systems. In 2020 JEEE Security and Privacy Workshops (SPW), pages 48-54, IEEE, 2020. 9
[13] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Conference on Computer Vision and Pattern Recognition, 2009. 5, 19
[14] Yinpeng Dong, Xiao Yang, Zhijie Deng, Tianyu Pang, Zihao Xiao, Hang Su, and Jun Zhu. Black-box detection of backdoor attacks with limited information and data. In International Conference on Computer Vision, 2021.
[15] Kuofeng Gao, Jiawang Bai, Bin Chen, Dongxian Wu, and Shu-Tao Xia. Backdoor attack on hash-based image retrieval via clean-label data poisoning. BMVC, 2023. 2
[16] Kuofeng Gao, Jiawang Bai, Baoyuan Wu, Mengxi Ya, and Tao Xia. Imperceptible and robust backdoor attack 3d point cloud. IEEE Transactions on Information Forensics and Security, 19:1267-1288 2023.
[17] Yansong Gao, Change Xu, Derui Wang. Shiping Chen, Damith Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th annual computer security applications conference, pages 113-125, 2019. 9
[18] Tianyu Gu, Kang Liu, Brendan Dolan-Gavite and Siddharth Garg. Badnets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230-47244 2019. 1, 2, 5, 7, 19, 20, 22, 23
[19] Junfeng Guo, Yiming Li, Xun Chen, Hanqing Guo, Lichao Sun, and Cong Liu. Scale-up: An efficient black-box input-level backdoor detection via analyzing scaled prediction consistency. In The Eleventh International Conference on Learning Representations. 9
12

---

[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European Conference Computer Vision, 2016. 5
[21] Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process, In International Conference on Learning Representations, 2022.
[22] Wenho Jiang, Hongwei Li, Guowen Xu, and Tianwei Zhang. Color backdoor: A robust poisoning attack in color space. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 8133 8142, 2023. 2
[23] Paramjit Kaur, Kewal Krishan, Suresh K Sharma, and Tanuj Kanchan. Facial-recognition algorithms: A literature review. Medicine, Science and the Law, 60(2):131 139, 2020.
[24] Alaa Khaddaj. Guillaume Leclere, Aleksandar Makelov, Kristian Georgiev, Hadi Salman, Andrew Ilyas. and Aleksander Madry. Rethinking backdoor attacks. In International Conference on Machine Learning, pages 12226-162366 PMLR, 2023. 3
[25] Simon Kornblith, Mohammad Norouzi, Honglak Lee, and Geoffrey Hinton. Similarity of neural network representations revisited. In International conference machine learning, pages 19-3999, PMLR, 2019. 18, 19
[26] Alex Krizhevsky et al. Learning multiple layers features from tiny images. 2009. 5, 19
[27] Ram Shankar Siva Kumar, Magnus Nyström, John Lambert, Andrew Marshall, Mario Goertzel, Andi Comissoneru, Matt Swann, and Sharon Xia. Adversarial machine learning-industry perspectives. In 2020 IEEE security and privacy workshops (SPW), pages 69-75. IEEE, 2020.
[28] Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 2015. 5, 19
[29] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learning: Training clean models on poisoned data. In Conference on Neural Information Processing Systems, 2021.
[30] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distilla- tion: Erasing backdoor triggers from deep neural networks. In International Conference on Learning Representations, 2021. 2, 6, 21, 22, 23
[31] Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. International Conference on Computer Vision, 2021. 1, 5, 7, 20, 22, 23
[32] Jiawei Liang, Siyuan Liang, Man Luo, Aishan Liu, Dongchen Han, Ee-Chien Chang, and Xiaochun Cao. Vl-trojan: Multimodal instruction backdoor attacks against autoregressive visual language models. arXiv preprint 2xXi::2402.13855 2024. 3
[33] Siyuan Liang, Jiawei Liang, Tianyu Pang, Chao Du, Aishan Liu, Be-Chien Chang, and Xiaochun Cao. Revisiting backdoor attacks against large vision-language models. arXiv preprint arXiv:2406.118 188444 2024.
[34] Siyuan Liang, Kuanrong Liu, Jiajun Gong, Jiawei Liang, Yuan Xun, Ee-Chien Chang, and Xiaochun Cao. Unlearning backdoor threats: Enhancing backdoor defense in multimodal contrastive learning via local token unlearning. arXiv preprint 2403 I 16257, 2024. 3
[35] Siyuan Liang, Mingli Zhu, Aishan Liu, Baoyuan Wu, Xiaochun Cao, and Ec-Chien Chang. Badelip: Dual- embedding guided backdoor attack on multimodal contrastive learning. arXiv preprint arXiv:2311.12075 2023. 3
[36] Liangkai Liu, Sidi Lu, Ren Zhong, Baofu Wu, Yongtao Yao, Qingyang Zhang, and Weisong Shi. Computing systems for autonomous driving: State of the art and challenges. IEFE Internet of Things Journal, (99):6499-686, 2020.
[37] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai Weihang Wang, and Xiangyu Zhang Trojaning attack on neural networks. Network and Distributed System Security Symposium, 2018. 2, 7, 20, 22, 23
[38] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. To- wards deep learning models resistant to adversarial attacks. In International Conference on Learning Representations, 2018. 5, 17
13

---

[39] Rui Min. Zeyu Qin, Li Shen, and Minhao Cheng. Towards stable backdoor purification through feature shift tuning. Advances in Neural Information Processing Systems, 36, 2024. 3. 6, 7, 9, 21, 22, 23
[40] Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. In Conference on Neural Information Processing Systems, 2020. 2, 6, 7, 20, 22, 23
[41] Tuan Anh Nguyen and Anh Tuan Tran. Wanet- imperceptible warping-based backdoor attack. In International Conference on Learning Representations, 2021. 2, 6, 7, 20, 22, 23
[42] Xiangyu Qi. Tinghao Xie, Yiming Li, Saeed Mahloujifar, and Prateek Mittal. Revisiting the assump- tion latent separability for backdoor defenses. In The eleventh international conference on learning representations, 2022. 3
[43] Xiangyu Qi. Tinghao Xie, Jiachen T Wang. Tong Wu, Saeed Mahloujifar, and Prateek Mittal. Towards a proactive } approach for detecting backdoor poison samples. In 32nd USENIX Security Symposium (USENIX Security 23). pages 1685 1702, 2023. 26
[44] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transfcrable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763, PMLR. 2021. 5
[45] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Papers), pages 2556-2565, 2018. 19
[46] Yucheng Shi, Mengnan Du, Xuansheng Wu, Zihan Guan, Jin Sun, and Ninghao Liu. Black-box backdoor defense via zero-shot image purification. Advances Neural Information Processing Systems, 36:57336- 57366, 2023. 9
[47] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recogni- tion. In International Conference on Learning Representations, 2015.
[48] Zhengyao Song, Yongqiang Li, Danni Yuan, Li Liu, Shaokui Wei, and Baoyuan Wu. Wpda: Frequency- based backdoor attack with wavelet packet decomposition. arXiv preprint arXiv:2401. 13578, 2024.
[49] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. The german traffic sign recognition benchmark: a multi-class classification competition. In international joint conference on neural networks,
2011. 19
[50] Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in hackdoor attacks. Advances in neural information processing systems, 31, 2018.
[51] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Muchine Learning Research, 9(1),, 2008. 25
[52] Matthew Walmer, Karan Sikka, Indranil Sur, Abhinav Shrivastava, and Susmit Jha. Dual-key multimodal backdoors for visual question answering. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pages 15375-15385, 2022. 3, 6, 7,
[53] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In Symposium on Security and Privacy, 2019. 1, 3, 6, 7, 21, 22, 23
[54] Hang Wang, Zhen Xiang, David Miller, and George Kesidis. Mm-bd: Post-training detection of backdoor attacks with arbitrary backdoor pattern types using a maximum margin statistic. In 2024 IEEE Symposium on Security and Privacy (SP), pages 15-15. IEEE Computer Society, 2023. 3
[55] Ruotong Wang, Hongrui Chen, Zihao Zhu, Li Liu, and Baoyuan Wu. Versatile backdoor attack with visible, semantic, speeifi,, and compatible triggers. arXiv preprint arXiv:2306. 00816, 2023. 2
[56] Zhenting Wang. Kai Mei, Juan Zhai, and Shiqing Ma. Unicorn: A unified backdoor trigger inversion framework. In International Conference on Learning Representations, 2023.
[57] Shaokui Wei, Hongyuan Zha, and Baoyuan Wu. Mitigating backdoor attack by injecting proactive defensive backdoor. arXiv preprint 245.:24055.61122 2024. 3
14

---

[58] Shaokui Wei, Mingda Zhang, Hongyuan Zha, and Baoyuan Wu. Shared adversarial unlearning: Backdoor mitigation by unlearning shared adversarial examples. Advances Neural Information Processing Systems, 36, 2024. 2, 6, 7, 9, 21, 22,
[59] Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, and Chao Shen. Backdoorbench: A comprehensive benchmark of backdoor learning. In Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. 1, 2, 5, 6, 19, 20,
[60] Baoyuan Wu, Hongrui Chen, Mingda Zhang, Zihao Zhu, Shaokui Wei, Danni Yuan, Mingli Zhu, Ruotong Wang, Li Liu, and Chao Shen. Backdoorbench: A comprehensive benchmark and analysis of backdoor learning. arXiv preprint arXiv:2401. 15002, 2024. 2
[61] Baoyuan Wu, Shaokui Wei, Mingli Zhu, Meixi Zheng. Zihao Zhu, Mingda Zhang. Hongrui Chen, Danni Yuan, Li Liu, and Qingshan Liu. Defenses in adversarial machine learning: A survey, arXiv preprint arXiv:331.088898 088900 2023. 3
[62] Baoyuan Wu, Zihao Zhu, Li Liu, Qingshan Liu, Zhaofeng He, and Siwei Lyu. Attacks in adversarial machine learning: systematic survey from the life-cycle perspective. arXiv preprint arXiv:2302.09455 2023. 2,
[63] Chuhan Wu, Fangzhao Wu, and Yongfeng Huang. Rethinking infonce: How many negative samples do you need? arXiv preprint arXiv:2 105. 1.303,, 2021. 16
[64] Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. In Conference on Neural Information Processing Systems, 2021. 1,3
[65] Yuan Xun, Siyuan Liang, Xiagjun Jia, Xinwei Liu, and Xiaochun Cao. Ta-cleaner: A fine-grained text alignment backdoor defense strategy for multimodal contrastive leaming. arXiv preprint arXiv:2409.17600 2024. 3
[66] Wenhan Yang, Jingdong Gao, and Baharan Mirzasoleiman. Robust contrastive language-image pretraining against data poisoning and backdoor attacks. Advances Neural Information Processing Systems, 36, 2024.
[67] Yi Zeng, Si Chen, Won Park, Zhuoqing Mao, Ming Jin, and Ruoxi Jia, Adversarial unlearning of backdoors via implicit hypergradient. In International Conference on Learning Representations, 2022. 1, 2, 6, 7, 9, 21, 22, 23
[68] Yi Zeng, Won Park, Z. Morley Mao, and Ruoxi Jia. Rethinking the backdoor attacks' triggers: A frequency perspective. In International Conference on Computer Vision, 2021. 2, 5, 20, 22, 23
[69] Xiaoyu Zhang, Yulin Jin, Tao Wang, Jian Lou, and Xiaofeng Chen. Purifier: Plug-and-play backdoor mitigation for pre-trained models via anomaly activation suppression. Proceedings of the 30th ACM International Conference on Multimedia, pages 4291-42994 -4299, 2022. 3
[70] Zhendong Zhao, Xiaojun Chen, Yuexin Xuan, Ye Dong, Dakui Wang, and Kaitai Liang. Defeat: Deep hidden feature backdoor allacks by imperceptible perturbation and latent representation constraints. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 15213- 15222, 2022. 2
[71] Runkai Zheng, Rongjun Tang, Jianze Li, and Li Liu. Data-free backdoor removal based channel lipschitzness. In European Conference on Computer Vision, 2022. 1,4, 18
[72] Mingli Zhu, Shaokui Wei, Li Shen, Yanbo Fan, and Baoyuan Wu. Enhancing fine-tuning based backdoor defense with sharpness-aware minimization. In International Conference on Computer Vision, 2023. 3, 6, 9, 21, 22, 23
[73] Mingli Zhu, Shaokui Wei, Hongyuan Zha, and Baoyuan Wu. Neural polarizer: A lightweight and effective backdoor defense via purifying poisoned features. Advances in Neural Information Processing Systems. 36, 2024.
[74] Rui Zhu, Di Tang, Siyuan Tang, XiaoFeng Wang, and Haixu Tang. Selective amnesia: On efficient, high-fidelity and blind suppression backdoor effects in trojaned machine learning models. In 2023 IEEE Symposium on Security and Privacy (SP), pages 1-19. IEBE, 2023. 26
[75] Zihao Zhu, Mingda Zhang, Shaokui Wei, Li Shen, Yanho Fan, and Baoyuan Wu. Boosting backdoor attack with learnable poisoning sample selection strategy, arXiv preprint 270.022307 2023. 2
[76] Zihao Zhu, Mingda Zhang, Shaokui Wei, Bingzhe Wu, and Baoyuan Wu. Vde: Versatile data cleanser for detecting dirty samples via linguissii inconsistency. In The Twelfth International Conference on Learning Representations, 2023. 3
15

---

Appendix
Structure of Appendix. We provide more analysis and experimental results in Appendix, which includes: (1) Formulation of multi-modal contrastive learning, backdoor attacks and our re-activation (2) More description and algorithms details in Appendix (3)
attack for MMCL in Appendix
Introduction of TAC and CKA in Appendix C. (4) Experimental implementation details in Appendix D. (5) More experimental results Appendix E. (6) Running time analysis in Appendix (7)
Visualization in Appendix G. (8) Additional experimental results in Appendix H.
A Backdoor multi-modal contrastive learning
A.1 Formulation of multi-modal contrastive learning task.
For multi-modal contrastive learning task, the training dataset is image-text pairs D {(v(), )/ S V x T. where V C Rd and T C Rd are image space and text space, re- spectively. For the network, we choose CLIP our primary MMCL model for the attack. CLIP is composed of a visual encoder fe. V + Rd and a textual encoder fe. T + Rd each with parameters O2 and O₂ representing their respective encoders. Denote the image embedding and text embedding as ve = Sa, (v()), ( = So. (t) respectively, for convenience. Given a batch of training pairs {(v(1, ) CLIP is optimized using the InfoNCE loss [63] as follows:
Ve .tee (T)
exp
min 0.)
log
Dj= exp u t
where T the temperature parameter. Given an input image (+), denote the output text of the model be (e( for convenience, where = (0.,, 0.).
A.2 Backdoor attacks for multi-modal contrastive learning.
For MMCL task, backdoor attacks could also be divided into data poisoning attack and training controllable attack. For data poisoning attack, an adversary creates poisoning pairs (v(1) + &, T) by patching a backdoor trigger & on the image ()) and revising the corresponding label into the target label 7 (for example, "a photo of banana" " in [3] and in our work). In training controllable backdoor attack, the adversary can control the training process to inject backdoors into the model. The goal of the adversary is to train poisoned model such that he, (0))) = {() and he, (v(+) + +) = T. And the goal of the defender is to purify the poisoned model such that the new model performs normally
as: hep (1))) t(() and hep (x(() + {) * T in inference time. Denote the encoder of poisoned
image and the target label as (ve (1)) and T., respectively for convenience. Our goal to search for perturbation A onto the original trigger & such that hep (+) + A T
Existing backdoor attacks primarily focus on achieving high attack success rates (ASR) in backdoor injection stages (I and II), with little consideration for the defensive impact in stage III. Given the failures of (()) + & in attacking he.. work focuses the re-activation attack in stage IV (please refer to See. 3.2 for formal definition of different stages of backdoors).
A.3
Re-activation attacks for multi-modal contrastive learning.
In this section we introduce our optimization formulation to learn the new trigger Ç' = E Ae. Since CLIP uses multi-modal contrastive learning instead of supervised learning to train the model, we also optimize the perturbation De by optimizing it with multi-modal contrastive learning loss. Given
number of no pairs (ag) (v(", T') from the poisoned dataset Dp and ne pairs (v()), t()) E D. from the clean dataset D., the optimization problem formulated as follows:
exp ((05)) ( )e Te/T)
A arg min |AellPSS
Σ
log
(7)
(i) )e Te/T) + E(v(), (000), ,9ED. exp (1) (1) + t(/i1T) exp (ug
(v
(, MT),,
16

---

Algorithm Black-box Backdoor Re-Activation Attack via Universal Square Attack (BBA) [1]
1: Input: Defense model f. training dataset D3, image shape c. h, w, norm p. perturbation bound
P. target label t E 1..... ....... number of iterations N, termination condition c.
2: Output: Perturbation A as in Eq. 4.
3: d x + init( A for Dp= + t Ag).
4: for = .......N N
do
5: if' ASR
then return De.
6: else
7:
(6) side length of the square modify (according to some schedule [1]):
8:
A new P (p, (6), w. G, Ag, (e,22 2) for æ E DD (see Appendix B for details);
9:
Enew + Project & + Ag new onto {= E IR || xllp - p} n [0, 1]d for x E Dp:
10:
new - (fnew 1) for x E DDp: pi
if new
1' then Ag t A
I'
hoow compute ASR;
12:
i it i i +1;
end if'
14: end for return A
where Ilp mcans l, norm, and i the perturbation bound. This problem can be solved using project gradient descent (PGD) [38] algorithm. Then the optimized A is attached to poisoned samples and the ASR is the probability of successful attacks out of the total number of new poisoned samples.
B Algorithms details
In this work, we provide some description and details of our attack algorithms.
White-box attack setting. shown Eq. 4, this is constrained optimization problem, which can be solved by using the classical project gradient descent (PGD) [38] algorithm to solve it. The main idea of PGD involves updating the perturbation using stochastic gradient descent in the initial step. Subsequently, in the following stage, the perturbation is constrained within the p-ball employing lp norm projection. We provide the algorithm description in Alg. 2. For additional insights, please refer [38] for more details.
Black-box attack setting. In this work, solve optimization problem in black-box setting, we utilize a randomized search strategy as emphasized in Square Attack [1]. Square Attack utilizes randomized search scheme where selects localized square-shaped updates random positions. This approach ensures that in each iteration, the perturbation is positioned near the boundary of the feasible set. A significant difference between our attack and Square Attack lies in their objective: Square Attack searches for perturbation for each image, terminating the query upon successful attack, while our objective is discover a highly generalizable universal perturbation restore the effectiveness of the backdoor utility. Therefore, we extend it to a universal Square Attack approach:
1. Firstly, initialize a universal perturbation.
2. In each iteration, randomly update our perturbation following the strategy in Square Attack. Apply the perturbation onto the image and then query the model with the new images.
3. Compare the loss: if the current loss is lower than the best loss, update the perturbation; otherwise, do not update and restart the search.
The above three steps represent the main concept of our algorithm. Details on the specific square update technique can be found in work [1].
Transfer attack setting. The main idea of the transfer attack to compute the averaged loss across models for each mini-batch The detailed algorithm is shown in Alg. 3.
17

---

Algorithm 2 White-box Backdoor Re-Activation Attack (WBA)
1: Input: Defense model f. training dataset Dp, norm p, perturbation bound target label E I,...,K, K, number of iterations N.
2: Output: Perturbation A as in Eq. 4.
3: initialize(Ae).
4: for io 0, O....N do
5: for mini-batch B { f)] C CPp do
6: Given f and input {(0) + Ag, (),1) compute the loss of Eq. 4;
7: Update Ae by minimizing via PGD algorithm:
8: end for
9: end for
10: return A4-
Algorithm 3 Re-Activation Attack via Transfer Attack (TA)
Input: Surrogate models fm: m 1,... M,, training dataset D,, norm p. perturbation bound D. target label / 1,..., K, number of iterations N.
2: Output: Perturbation á as in Eq. 5.
3: initialize(Ae))
4: for i = O.... N do
5: for imini-batche B = { t)}i C Dp do
6: I= 0:
7:
for = O,.., M- do
8:
Given fm and input {(r) + Dg. t) compute the total loss 4m Eq. 5; 1+1+
9:
10: end for
Update De by minimizing/ via PGD algorithm;
12: end for
13:
end for
14:
return A
C Introduction of TAC and CKA
We provide the detailed introduction of Trigger--cttvated Change (TAC) [71] and Centered Kernel Alignment (CKA) [25] in this section.
Trigger-activated Change. To measure the correlation of neurons with backdoors, Zheng al. [71] proposed the TAC metric to quantify the correlation between the impact of backdoors and neurons. Given the poisoned dataset Dp = {(=E (i) ())),, let the original clean dataset of DD to be Do, i.e, D. (1x(9) y ya)
(2) E eDp). Then the TAC can be computed as follows:
TAC (Dp, D.) = [Dpl (E),
Σ
(x) 110 (x
(8)
where & is the index of channel of the 1th layer. A higher TAC value assigned to neuron indicates stronger association with backdoors. In this work, with this metric, we first assign each neuron with a TAC value. In order to select neurons relevant backdoors and considering their sparsity nature, the top 10% of neurons based on their descending TAC values are chosen as the backdoor related neurons. Then the Backdoor Existence Coefficient be computed accordingly.
Centered Kernel Alignment. The Centered Kernel Alignment (CKA) [25] measures the similarity between representations, which utilizes HSIC to measure the independence between two distributions. It quantifies how well neural networks preserve similarity relations in the data across different layers. valuable tool in feature analysis and understanding DNNs especially for high-dimensional features. In this work we employ CKA to quantify the similarity between features in different
18

---

networks. As the work [25] shows, the Centered Kernel Alignment (CKA) is defined as follows: Let X =xd and Y RR be two representations from neural networks, where n represent number of samples and d is the feature dimension. The empirical estimator of HilberttSchhidt Independence Criterion (HSIC) is defined as:
LT (KHHH) HSIC L) (n 1)2
(9)
where H is the centering matrix Hn - k(x;,)) =fyy, Tag = (x),, x! as defined in [25]. Then the Centered Kernel Alignment is defined as:
_11T. The K and H are linear kernels: Kij
HSIC K, L) CKA(K, HHSICK, K) HSIC(L,
(10)
More details could be found in [25].
D Experimental implementation details
In this section, delve into the implementation details, covering the evaluation datasets, specifics of the attacks and defenses compared, and implementation of our proposed methods. All experiments are executed five times with varying random seeds and the averaged results are displayed in this work.
D.1 Datasets
For image classification task, we use three benchmark datasets: CIFAR-10 [26], Tiny ImageNet [28], and GTSRB [49] to assess the performance of our approach, following the benchmarks outlined in [59]. For MMCL task, subset of CC3M dataset [45] selected for backdoor injection and the poisoned models are tested through zero-shot evaluation on ImageNet-1K validation set [13]. All dataset splits are aligned in our experiments.
CIFAR-10: Total number of 60,000 images distributed among ten classes, with 5,000 images per class in the training set and 1,000 images per class in the testing set. Each image in CIFAR-10 is sized 32 x 32 pixels.
Tiny ImageNet: A subset of ImageNet [13] containing 200 classes, 500 training samples and 50 testing samples per class. Each image in Tiny ImageNet sized x 64 pixels.
GTSRB: A total of 39,209 training images and 12,630 testing images among 43 classes. Each image in GTSRB is sized 32 x 32 pixels.
CC3M: The CC3M dataset has about 3300K, 15K, 12K image-text pairs for the training, validation, and testing dataset, respectively. Each image in CC3M is sized 224 x 224 pixels. Following [3], 500K image-text pairs from the CC3M are selected in backdoor injection phase.
ImageNet-1KK KK: the ImageNet- IK dataset is subset of ImageNet dataset, which has a total 1000 classes. Each image in ImageNet- IK is sized 224 x 224 pixels.
D.2 Backdoor attack details.
We introduce the different backdoor attack methods first, followed by the experimental settings.
Fig. 4 and 5 show the visualization of poisoned samples in comparison with clean image for different backdoor attacks CIFAR-10 10 and ImageNet-IK IK dataset, respectively. The attack details for image classification task are as follows:
BadNets [18]: BadNets is trigger-additive attack which inserts patch of fixed pattern (a 3 x 3 white square patch in our work) to replace some pixels in the image. The patch size is 3 3 CIFAR-10 10 and GTSRB, and 6 x on Tiny ImageNet, following BackdoorBench.
Blended backdoor attack (Blended) [10]: Blended attack blends pre-defined image (Hello Kitty in our work) with the original image. The blend coefficient a is 0.2, following BackdoorBench.
19

---

Clean
Badnet
Blended
InputAware
LF
SSBA
Trojan
WaNet
Figure 4: Visualization of poisoned samples for different backdoor attacks on CIFAR- 10 dataset.
Clean
Badnet
Blended
SIG
TrojanVQA
Figure 5: Visualization of poisoned samples for different backdoor attacks on ImageNet- 1K dataset.
imput-aware dynamic backdoor attack (Input-Aware) [40]: Input-Aware is training-
controllable attack that first learns trigger generator by adversarial training. Then the generator is used to produce sample-specific triggers during model training.
Low frequency attack (L.F) [68]: LF first learns universal adversarial perturbation (UAP) and filters the high-frequency artifacts. Then the filtered UAP is the trigger and patched onto the clean samples to generate poisoned samples.
Sample-specifie backdoor attack (SSBA) [31]: SSBA first trains autoencoder. Then the autoencoder is used to fuse triggers with clean samples to generate poisoned samples.
Trojan backdoor attack (Trojan) [37]: Trojan first learns universal adversarial perturbation (UAP), and then patches it onto the clean samples to generate poisoned samples.
Warping- poisoned networks (WaNet) [41]: WaNet first defines a warping function to perturb the clean samples generate poisoned samples. Then the adversary controls the training process to make sure the model learns the specific warping.
More details can be found in BackdoorBench [59].
For MMCL. task, we follow CleanCLIP'ss settings[3]. The attack details for MMCL. task are as follows:
BadNets [18]: BadNets is trigger-additive attack which inserts patch of fixed pattern (a 16 x 16 random noise patch in our work) to replace some pixels in the image.
Blended backdoor attack (Blended) [10]: Blended attack blends a pre-defined image (a global random noise patch in work) with the original image. The blend coefficient is 0.2.
SIG [4]: SIG attack designs a sine wave pattern noise as trigger, which has a same size with the image. The blend coefficient is 0.2.
20

---

Trojan [52]: Trojan VQA is a training-controllable controllable attack in which the adversary utilizes both modalities to generate triggers, which has a size 16 16.
To poison image classification task, we use a poisoned dataset with 10% poisoning ratio to train the poisoned model. To poison the MMCL model, start with the pre-train pre-trained CLIP model which is trained on 400M image-text pairs. After that, a total of 500K image-text pairs within which 1500 samples poisoned pairs is used for backdoor injection. The model is trained for 10 epochs with learning rate of le-6, and a batch size of 128.
D.3 Backdoor defense details.
We introduce the different backdoor defense methods in this section.
Neural cleanse (NC) [53]: NC first searches for minimal UAP to detect backdoors. If the model is detected as a backdoor model, purifies the model by unlearning the optimized UAP.
Neural attention distillation (NAD) [30]: NAD use knowledge distillation strategy which distills the attention across the model to acquire new clean model.
Implicit backdoor Adversarial unlearning (i-BAU) [67]: I-BAU designs a implicit hyper- gradient method to solve the adversarial training optimization.
FT-SAM [72]: It utilizes sharpness-awan minimization to fine-tune the poisoned model.
Shared adversarial unlearning (SAU) [58]: SAU first generates shared adversarial examples and then unlearns these adversarial examples to purify the model.
Feature shift tuning (FST) [39]: FST encourages feature shifts by re-iniltialization the linear classifier and fine-tuning the model.
CleanCLIP [3] CleanCLIP use both multi-modal contrastive loss and in-modal self- supervised loss to fine-tune the model.
FT [3]: FT uses multi-modas contrastive loss to fine-tune the model.
More details about the implementation of defenses can be found in BackdoorBench [59]. For CleanCLIP, we follow the work's setting [3] that the CLIP model is trained for 10 epochs with 50 steps of warm-up using learning rate of 4.5e-6, and a batch size of 64. A total of 10K training pairs are selected from CC3M to train.
D.4 Backdoor re-activation attack details.
During the inference phase, we implement our re-activation attack by searching for a global universal perturbation (same size as images) without altering model parameters. For image classification tasks, we employ the same optimized hyperparameters to learn the perturbation across various models and datasets. Specifically, the details are as follows:
For white-box attack, we use the SGD optimizer with learning rate of 0.05, update the adversarial perturbation within the inner loops for 5 steps, and train for a total of 50 epochs. The hyperparameter for the loss is fixed at 1. The training dataset 1000 poisoned samples that are randomly selected from the original poisoned samples. The batch size is
set to 256.
For black-box attack, we follow the original hyperparameters in [1]. The updated criterion is based on the decrease in loss rather than the improvement in ASR, as we have found that this approach yields better results.
For transfer attack, we maintain same hyperparameters to those used in white-box attacks except for training epochs, which is set to 100. We also smaller bound 1. The training samples are set to 5000. For ensembling these surrogate models, we average their losses in each mini-batch
For MMCL task, we use SGD optimizer with a learning rate of 0.01, update the adversarial perturbation within the inner loop for one steps, and train for a total of 40 epochs. We use the -oomnorm with a 0.05 norm bound. We use 1500 poisoned image-text pairs and some clean reference data for optimization.
21

---

E Experimental results on different datasets and models
In this section, we showcase the performance of our attacks under various settings to demonstrate the superior performance of our method.
Table 12: Performance (%) of backdoor re-activation attack on both white-box (WBA) and query- based black-box (BBA) attacks with le -norm bound p 0.05 against different defenses with Tiny The best results are highlighted in boldface.
ImageNet on
Attacks
No Defense
NC [53]
NAD [10]
BAU [67]
STSAM [72)
SAU [58]
FST [39]
Defense
Deferse
Defense
BBA
BadNers [15]
99
59:90
99.01 01 09.97
0.27
55.58
26.21
3,61
99.40
08.
0.21
45.96
23.73
0.28
57.88
50.87
0.02
49.96
17.15
Bleadea [10)
96.32
99.87
75.36
0.01
76.38
51.18
1.32
91.75
0.48
lapun- [40]
99.04)
0.30
99.90
9.38
11.15
36.48
25.03
52.13
24.99
10.64
0.29
37.37
0.81
0.58
19:04
7.32
0.02
6.36
2.20
LP [68]
98.51
58.63
96.76
97.29
56.00
97.90
51.05
3.77
97.18
55.02
5.14
97.30
30.59
2.47
78.96
50.42
2.66
78.37
1.39
SSBA [3]
97.69
0.05
98.17
14:75
69.47
97.05
69.07
9.14
96.26
80.53
0.38
34.04
0.05
0.00
4.19
0.30
0.73
66.89
51.11
Trejan [57]
99.97
0.27
99.99
13.75
1.01
61.31
26.77
0.80
99:44
99.67
0.21
50.08
25.00
0.01
18,00
5.56
0.58
68.92
0.31
WaNee [41]
96.50
96.50
66.36
69.59
0.87
51.89
18.57
63.68
47.65
0.79
73.61
53.77
41.83
80.59
25.35
0.15
77.89
27.74
98.83
54.43
94.73
57.77
32.08
75.36
47.23
26.25
$2.85
74.04
6.35
62.49
29.90
6.39
47.98
21.57
0.78
62.84 14.34
Attack performance on Tiny ImageNet dataset. Tab. 12 presents the performance of our re- activation attack with both WBA and BBA applied on the Tiny ImageNet dataset with PreAct- ResNet18, compared with ASRs of original attack models (No Defense) and defense models (Defense). Careful observation and analysis of this table furnishes some important insights:
While not achieving the same high level efficacy as in previous experiments, our attacks still show reasonable effectiveness against defense models. On average, our WBA and BBA improve ASRs by 50.00% and 19.77% respectively when compared against defense mechanisms, which is actually high than that on CIFAR-10 dataset. This highlights some potential security vulnerabilities in these defense models, although the final ASRs is less severe than those exposed in the previous dataset.
2.
The performance of our WBA establishes the viability of our backdoor recovery mechanism in a more challenging setting. further verifies the latent recoverability of backdoors in defense models. Despite the existing gap between our WBA and the more realistic BBA, we suggest that this gap can be reduced with further optimization of our black-box attack strategy.
3. When comes to defense mechanisms, similar to previous observations, attacks on three defenses FT-SAM, SAU and FST show less impressive ASRs. This can be seen an indication of the significant efficiency of their backdoor removal mechanism. While these mechanisms are more effective in this dataset, these insights are still crucial for developing future defense strategy development.
4. It's worth mentioning some failed cases in our experiment on the Tiny ImageNet dataset. Although attack methods generally show promising results, the performance in some particular instances falls short of expectations. Future work can gain valuable insights from scrutinizing these instances more closely. Such failures in specific settings serve stepping stone toward the development of more effective and robust attack strategies, such increasing the diversity of random searches.
Attack performance on GTSRB dataset. Tab. 13 presents the performance of our re-activation attack with both WBA and BBA applied on the GTSRB dataset with PreAAct-ResNetl compared with ASRs of original attack models (No Defense) and defense models (Defense). A meticulous examination of the results in Tab. 13 provides pivotal insights:
1. Consistent with earlier experiments our attacks display impressive effectiveness against these defense models. In these tests, our WBA and BBA average ASRs show improvement of 50.58% and 45.25% respectively compared to the defense mechanisms. This achievement exposes vulnerabilities in the current defense models that were previously unnoticed and highlights the robustness of our attacks.
2. The effective performance of our WBA affirms the potency of our backdoor recovery mechanism. The backdoors' resilience and latent recoverability in defense models are
22

---

Table 13: Performance (%) of backdoor re-activation attack on both white-box (WBA) and query- based black-box (BBA) attacks with fo -orm bound P 0.05 against different defenses with GTSRB on PreAct-RRsNett1. The best results are highlighted in boldface.
Attacks
No Defense
NCJS1
NAD [30]
I-AU [67]
PESAM [22]
SAU [58]
EST [29]
Defense
BBA
Defense
Defense
WDA
Defesse
BDA
Defence
BBA
Defense
WBA
BBA
BoiNers [18]
95.02
01022
62.43
55.59
70:94
94.66
92.57
0.00
48.04
-0.93
0.17
99,43
55.15
0.01
36.11
30.01
0.02
57.49 48.30
Blended [10]
100.00
99.18
76.50
100,00
100,00
99.10
59.56
58.45
29.99
87.75
84.56
77.26
99.94
97.89
Input Aware [10)
95.85
0.03
53.76
45.01
55.55
96.23
94.51
0.00
50.29
06.42
0.02
64,04
59,21
0.00
29.74
19.78
0.00
37.15
33.06
LF LFF[6S]
78.06
67.30
79.76
99.28
77.56
2:55
69.34
61.99
0.04
44.43
27.14
0.82
71.95
67.21
SSBA [31]
99.77
2,43
66.97
63.22
96.95
99
97.37
0.18
36.30
30.70
0.70
$7.14
53.65
1.95
37.13
31.36
32.55
92.99
91.29
Trujan [?]]
100,00
0.36
64.14
60.00
0.10
55.46
68.33
0.00
33.51
18.68
0.11
62,82
61.14
0.06
55.44
49.81
1.95
79.16
74:30
WaNee (41]
89.03
77.65
0.04
82.73
80.42
56.68
44
0.00
63.94
56.61
0.08
41.61
10.97
0.00
65.74
57.96
Avg
98.35
1.69
69.51
64.47
50.25
89.72
87.24
14.90
57.48
50.36
1.36
66.61
62.32
4.59
47.46
39.09
16.08
72,06
67.14
further substantiated. Additionally, the performance gap between our WBA and the more realistic BBA suggests room for further enhancement of our black-box attack strategy.
3. Concerning the defense mechanisms, the ASRs against i-BAU and SAU continue to be relatively less impressive. The indication of these defense mechanisms' efficiency in backdoor removal remains constant. Despite the improved effectiveness witnessed in the GTSRB dataset, these results works reference for the design of more robust defense strategies in the future.
Table 14: Performance (%) of backdoor re-activation attack on both white-box (WBA) and query- based black-box (BBA) attacks with fo- norm bound م 0.05 against different defenses with CIFAR-10 on VGGI9-BN BN. The best results are highlighted in boldface.
Attacks
No Defense
NC [S3]
NAD [30]
I-BU [67]
FESAM [72]
SAU [SS]
Defanse
WRA
Defense
WBA
Defense
WRA
Defense
WRA
BRA
Defense
WBA
BadNets [18]
94.43
5.08
95.31
34.57
5.77
94,80
64.86
3.13
92.69
39.68
1.29
68.77
24.54
4.28
85.84
66.59
Blended [10]
99.50
99.50
99.72
98.52
86.98
99.78
78.47
51.67
99.94
88.
8:23
98.56
75.13
7.81
96.81
62.33
Input Aware
97.02
97.02
99.34
97.37
14.04
83.38
19.13
78.93
99.38
97.41
3.41
79.14
27.47
1.19
60.07
27.07
LF LF|
13.85
1.26
91.66
15.38
3.07
88.88
43.07
6.66
69.79
7.27
2.17
0.73
0.76
1.56
79.07
25.54
[3]]
95.10
98.23
87.60
98.79
12:37
72.07
1.84
59.81
25.17
67.79
Trgjnn [37]
100.00
100.00
100.00
100.00
5.18
95.00
$8.59
2.69
85.93
38.76
5.13
78.03
2.34
0.19
41.67
21.95
Waleet [41]
96.49
96.49
99.97
97.27
2,40
1.10
92.21
22
93.48
Ang
85,20
70.64
97.75
71.83
25.35
93.99
51.38
22.55
90.89
58.05
3.31
68.18
31.99
2.83
74.96
41.96
Attack performance on VGG19-BN network. Tab. 14 presents the performance of our re- activation attack with both WBA and BBA applied on the CIFAR-10 dataset with VGG19-BN architecture, compared with ASRs of original attack models (No Defense) and defense models (Defense). Detailed analysis of the results provides the following key takeaways:
1. Our attacks display remarkable potency against the VGG19-BN network, with both our WBA and BBA demonstrating impressive average ASRs. Specifically, our WBA achieves a significantly high ASR, further emphasizing the backdoor's 's recoverability, even in this more complex network architecture. As for BBA, although its ASR doesn't 't reach the same level as WBA, it presents commendable rate, denoting a successful real-world adversarial scenario.
2. The superior performance of our WBA attests the robust and tenacious nature of our backdoor recovery mechanism, showcasing our approach's 's adaptability and effectiveness across different network structures. The observable gap in ASR between WBA and BBA can be an impetus for refining the black-box attack strategy.
Attack performance with different norm types under different backdoor attacks. In (a) and (b) of Fig. 3 in the main script, we display the ASR results under different norm types and bounds using Blended attack model. Here, we more experiments on different attacks and the results are shown in Fig. 6. As can be seen from the figure, our re-activation attack demonstrates a consistent trend across various attack models, showing stable high ASRs when the bound approaches 2 and 0.05 for &g formm and lo norm attacks, respectively.
23

---

I. Norm
f Narm
f. Norm
1 Norm
2.5
3.8
ASR
NC
NC
NC
MC
IBAU
IBAU
IBAU
IBAU
0.2
SAU
SAU
SAU
SAU
FST
FST
PST
rST
0.2
0.05
0.0r
200
0.05
6.03
0.07
(a) BadNets
(b) BadNets
[c] LF
(d) LF
Norm
Norm
1.0
1.0
0.8
0.8
0.0
0.5
NAD
NAD
1-AU
0.4
BAU
FT-SAM
FT-SAM
SAU
0.2
SAU
0.0
0.0
SSBA
SSBA
Figure 6: (a) and (b) show the ASR results under different norm type p and bound for BadNets. (c) and (d) show the ASR results under different norm type and bound for LF. (e) and (f) show the ASR results under different norm type p and bound p for SSBA.
F Running time analysis
Table 15: Running time analysis.
up (min.)
Res1&+CIFARIO Res18+Tiny Res1&+GTSRB VGG+CIFARIO CLIP+CC3M
WBA
2.5
4.4
1.7
2.3
32.5
BBA
20.1
36.8
10.4
30.0
N/A
TA
11.3
23.5
8.4
14.8
32.5
In this section, we conduct an analysis of algorithm complexity based on running time statistics. Except for the initial training phase for attack and defense, all our re-activation attacks are trained on a single 3090Ti GPU. We provide a comparative view the running times. Since the running time of attack is only related to trainning dataset and network, while independent with specific backdoor attack or defense methods, we didn't 't specify the particular method, as the running time is consistent methods. As displayed in Tab. 15, our attack achieves impressive speed. This be attributed primarily to our attack requiring a smaller number of training samples, and our approach's efficiency in computing adversarial samples, needing only few inner-loop iterations achieve satisfactory performance. As a result, the training speed is expedited. Unlike in traditional adversarial attacks, our attack will require no further training once the optimal UAP solution is found. This condition poses a considerable threat in reality. It is noted that "N/A" appears for BBA on the CLIP models and CC3M dataset in the table, as did not conduct black-box attacks for the CLIP models. For query-based black-box attack, the attacker cannot directly access the target model (such as weights or gradients) and CLIP models only return the final matching score or ranking results. This limits the ability of query-based black-box attacks. Moreover, there are no relevant studies for reference. Thus, we marked related result as "N/A". Additionally, we want to emphasize that BBA requires a large number of queries to the target model to achieve satisfactory attack performance, whereas TA only needs a single query to launch an attack. Clearly, TA is both more efficient and practical compared to BBA.
24

---

G
Visualization
In this section, we provide two visualization techniques to showcase the existence of backdoors: feature map visualization and t-SNE visualization.
Feature maps visualization. We visualize the feature maps, i.e., the features after all convolutional layers, of LF attack and different defense models. We rank these features in descending order of the TAC value of the attack model, meaning that, in each image's s subplot from top to bottom, illustrates a sort from high to low of backdoor effect. From Figures7 to 12, we can make the following observations:
In the attack model, the highlighted part of the feature maps (the upper sections of each subplot) indicates the existence of backdoor in the model.
The highlighted corresponding section the defense model suggests that the defense model is still sensitive backdoor samples. This sensitivity manifests activate neurons related to the backdoor, even when presented with such samples.
ability to primarily
We have also visualized the feature maps of the clean model and have found that no such phenomenon exists in the clean model. This comparison indicates stark difference in behavior between the defense model and the clean model.
1ycer10.coov1
1 1.ccov0c
layer1.1.com/
1ayer1.1.comv2
layor2.0.com 1
layer2.0.co 0.con22
10ye2.1.convv
layer2.1.comv
1ayer3.0.comm
layer3. 0.conu2
1ayer3, convv
layer3.1.conv2
layerd.0.convl
layer4.0.com/ cono22
layer4,1 convil
layeré.1.conv2
Sorted feature map visualization for all convolutional layers PreAcct-ResNet18 with the features in descending order of TAC values for LF backdoor attack.
Figure 7:
T-SNE visualization. We attempt to observe the backdoor effect in defense models by visualizing features of poisoned and clean samples via t-SNE visualization [51]. illustrated Figures 13 to 15, black dots denote poisoned samples while different colors signify various classes of clean samples. Several observations can be drawn from these figures:
Across these attacks, the backdoor samples clustered in the feature space.
In the defense models, numerous backdoor samples still cluster together, indicating that the backdoor traits of these samples continue to dominate the network's recognition of backdoor images, even though these no longer classified into the target class.
25

---

layer1.0.comg
2ycov10.conv2
leyer1.1.conv
leyer1.1.com2
layer2 0.covv
layer2 0.corv2
Mayer.1..orvv.
layer2.1.corvg
layer3.0 3onvvl1
1ayer3.0. 3.c.v
1yee3..1.onvv.
layer3.1.conv/
layor4, 0.conv1
layer4, D. conv2
layer4.1.ccm 1oyor4.1.covv1
layer4.1.comg
Figure 8: Sorted feature map visualization for convolutional layers on
with the features in descending order of TAC values for NAD defense against LF backdoor attack.
Thus, viewing from the t-SNE visualization, may also infer that backdoors still exist within these defense models.
H Additional experimental results
H.1 Attack performance against recent defenses
To test the attack performance against recent defenses, we evaluate the performance against two defense methods: SEAM [74] and CT [43], respectively. The evaluations are conducted on CIFAR-10 dataset with PreActtResNet1$ network, and the results are shown Table 16. It found that both SEAM and CT vulnerable to the proposed re-activation attack. We would like to emphasize that we have not claimed all post-training defenses are vulnerable to re-activation attacks. The primary objectives of our work are: (1) to reveal this new threat, which has been validated against several classic post-training defenses, and (2) to provide effective tools for evaluating the vulnerability of both existing and future post-training defenses. Therefore, future post-training defense strategies should take this threat into account and aim to mitigate the proposed re- re-activation attack.
Table 16: ASR (%) of our attack against SEAM and CT.
Post-training defense
SEAM
CT
Original attack J. Re--activation attack
No re-activation
WBA
BBA
No re-activation
WBA
BBA
BadNets
5.33
97.53
33.51
0.00
99.58
92.21
Blended
6.79
98.40
69.79
1.34
100.00
99.00
Input-Awar Aware
1.27
92.10
48.59
70.95
99.96
85.80
LF
13.22
97.61
65.63
3.28
99.63
99.23
26

---

leyer1.0.conv
layer1.0.comv
layer1.1 convl
layer1.1.conv2
layer2.0.come
layer2 0.conu2
1ayer2.1.co 1oon2
1ayer2.1.conw2
1eyer3.0.oomvl
layer c.0.con22
layer3. convl
1ayer3.1.conv2
leyer4.0.conv.
0yyer4.0coom/
layer4,1 conv1
layer4,1.conv2
Figure 9: Sorted feature map visualization for convolutional layers on PreAct-ResNeds with the features in descending order of TAC values for NC defense against LF backdoor attack.
H.2 Transfer attack across model architectures
In the previously discussed transfer-basee re-activation attack, we considered scenario where the attacker is the publisher of the backdoored model. Here, we explore more strict scenario, where the attacker is only the publisher of the poisoned dataset and has no knowledge of the defender's model architecture or training process. Therefore, when carrying out transfer-based re-activation attack, the adversary can perform a transfer attack across different model architectures. We study this problem as follows.
Threat model: Here we present more strict setting where the adversary can only manip- ulate the training dataset, while having no access the training and post-training stages. Thus, the adversary only knows the original trigger &. but has no knowledge of fe, fe, Compared the previous threat model, one major challenge is the unknown architecture of the target model fD.
Main attack steps: Compared to the steps in the previous setting, there is one additional step where the adversary must first train backdoored model So, based D,- which has different architecture than fo. All remaining steps are the same as those in the previous setting.
As shown in Table 17, although the transfer attack across model architectures does not achieve as high ASR as the transfer attack with the same architecture (i.e., the results in Table of the main manuscript), it still demonstrates certain degree of backdoor transferability. This is an intriguing phenomenon worthy of further exploration.
Experimental results:
H.3 More experimental results on test-time detection
In Table of the main manuscript, we analyzed the performance of our WBA attack against test- cest-time detection. Here, we present additional experimental results, focusing on more backdoor attack methods, and evaluating the performance of our WBA, BBA, and TA attacks. As shown in Tab. 18, our three kinds of attacks do not markedly increase the TPR compared the defense models. These findings provide insights to develop more stealthy re-activation backdoor attacks in the future.
27

---

layer1.0.conv
layer1.0.comv
layer1.1convv
layer1.1.com2
1.0
c.covv
0.co22
byye2.1.convi
layer2.1.conv/
0.5
layer3.0.conv Ion.vvl
1ayer3.0, conv2
Myer3.1.convl
Inyer3.1.onv2
.0.0
hyer4.0.convl
layer4, 0.cov2
hyer4.1.comg
hayer411.cov2
15
Figure 10:
Sorted feature map visualization for all convolutional layers on PreAct-ResNet18 with the features in descending order of TAC values for FST defense against LF backdoor attack.
Table 17:
Transfer re-activation attack preformance (ASR %) against the target model PreAct- ResNet18, using different architectures of source models.
Source Model
WideResNee22-22 FT-SAM
ResNet18 FT-SAM
VGG19-BN
Attack L Defense
i-BAU
SAU
i-BAU
SAU
i-BAU
FT-SAM
SAU
BadNets
95.6
84.1
60.0
53.4
30.9
26.5
89.2
71.7
48.6
Blended
98.5
98.5
79.1
64.1
92.8
90.1
Table 18: ASR of our three attack methods against three test-time backdoor detection methods.
Attack]
Detection]
[Backdoored]
FT-SAM (Defense) WBA (Ours) BBA (Ouns) TA (Ours)
SAU (Defense)
WBA (Ours) BBA (Ours) TA (Ours)
SCALE-UP
0.3955
0.7959
0.6858
0.6225
0.8222
0.7949
0.4954
0.4648
0.5924
BadNets
SemiiNet
STRIP
0.8834
0.0073
0.0553
0.0027
0.2576
0.1033
0.0647
0.0144
0.0760
SCALEUP F-UP
0.6077
0.5420
0.6577
0.5358
0.7635
0.7500
0.7138
0.6398
0.6942
Blended
SemiNet
0,0292
0.0000
0.0000
0.0000
0.0000
00000
0,0000
0.0000
0.0000
STRIP
0.5946
0.0076
0.5007
0.0007
0.7727
0.0086
0.0742
0.0002
0.1087
SCALE-UP
0.9323
0.6387
Input-aware
NemtnNet
0-4030
0.0000
0.0000
0.0000
0,0000
0.0000
0,0000
0.0000
0.0000
STRIP
0.0085
0.0369
0,2380
0.0199
0.6853
0.0368
0.0536
0.0108
0.0682
SCALE-UP
0.8663
0.8490
0.8355
0.7652
0.8913
0.4286
0.2748
0.5714
0.3308
Senti
0.000
0.0000
0.0000
0.9880
0.9322
STRIP
0.3438
0.0141
0.2911
0.0127
0.2872
0.2484
0,2011
0.1743
0.6934
SCALE-UP
0.4744
0.8082
0.7217
0.6903
0.8555
0.8505
0.7188
0.7187
0.7646
SSBA
SemiNet
0.0886
0.0000
0.0000
0.0000
0.0000
0.2827
0.1029
0.2611
0.2022
STRIP
0.6191
0.2044
SCALE-UP
Trojan
SemiNet
0.0292
0.0110
0.0105
0,0000
0,0000
0.0211
0.0147
0.0089
0.0044
STRIP
0.9999
0.0194
0.2984
0.0143
0.7993
0.0419
0.0116
0.0150
0.0009
SCAL E-UP
0,5012
0.9070
D.8207
0.8137
0.8888
0.7568
0.6980
0.6601
0.7635
Semineet
STRIP
0.0086
0.04/
0.2231
0.0112
0.6535
0.1651
0.1580
0.0277
0.2799
28

---

inyorl...oovv
layer1.0.conv2
layer1.1.conv
layer1.1.conv2
04
1yyer2.0.comm
#yyer2.0.conv2
1yer2.1.corvv/
1ayer2.1.comm
layer3.0.com/
layer3.0.com/
layer3.1.convl
layer3.1.conv2
...
0.convl
layer4
layer4.
-044
Figure 11: Sorted feature map visualization for all convolutional layers PreAct-ResNett1 8 with the features descending order of TAC values for FT-SAM defense against LF backdoor attack.
layer1.0.onvv
byer1.0.comw2
1.yer
layer1 1.conv2
layer2.D.co vvl
layer2.0. c.co2.0
layer2.1.cam//
layer2.1. 2.1..cmru
layor3.0.cowl
keyor3.0.com2
leyer3.1.comvl
layer3.1.comv2
-0.2
ayer4.D.com/
byya440.comv2
bayar4.1.comv1
layer4.1.comv2
0-44
0.8
Figure 12: Sorted feature map visualization for all convolutional layers PreAct-ResNetll with the features descending order of TAC values for clean model.
29

---

Abplox
Figure 13: Comprison of T-SNE visualization between Blended attack model and different defense models on CIFAR-10.
LF
NC
NAD
IBAU
FT-SAM
SAU
FST
Astrnaiia
Prioneid
Figure 14: Comprison of T-SNE visualization between LF attack model and different defense models on CIFAR-10.
Trojan
Aavicaille
Honw
Poionead
Figure 15: Comprison of T-SNE visualization between Trojan attack model and different defense models on CIFAR-10.
30

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]-
Justification: The main claims presented in the abstract and introduction offer a precise overview of what accomplished and investigated in the research, providing accurate reflection of the paper's scope and contributions.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes].
Justification: The paper discusses the limitations of work. See the "Limitations and future work" section for details.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
31

---

Answer:
[NA]
Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: The paper provides detailed information on the experimental setup for the reproduction of experimental results.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
32

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
We provided the code via anonymous website.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines Inttps://nige cc/ publ c gui des SoodeSubm s.s ssionn onnooll cy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: /nips cc/ cttppcccpccccccccccc
iguudeess......... CodeS Suddi ss oioooool icy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data,
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer:[Yes:
Justification:
We outline all details related to the training and test processes, including data splits, hyperparameters, their selection process, and the types of optimizers used, thereby providing a comprehensive understanding of the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have considered the randomness and run the experiments multi times for main results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
33

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have clearly reported the resources used for experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https::///// /neuriurrppp iss cc h/Et csGu de nes?
Answer: [Yes]
Justification: The research conducted in the paper conforms, in every respect, with the NeurIPS Code of Ethics
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We have discussed it in "Broader Impacts" section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
34

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: Our work has no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: Properly credited.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
35

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papersw www..ww. thoode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors encouraged to reach out the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector:
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
36

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "VyTtdpqReV", "reviewer": "Reviewer_AhAB", "summary": "The paper investigates the idea that an attacker could modify the trigger to make it effective against models modified by post-training defenses (to _re-activate_ the backdoor in defended models).\n\nThe paper presents a measure of how much the effect of the backdoor shows persists in model activations and shows empirically that post-training-defended models have intermediate activations more similar to backdoored models than clean models when given backdoored inputs.\n\nThe paper defines 3 $\\ell_p$-bounded universal adversarial attacks for backdoor re-activation: a white-box (access to parameters), a black-box (query access) and a transfer attack (access to model architecture).\n\nThe attacks are very successful in most cases, but has a limited effect in cases of some attack-defense combinations. Validation experiments show that the universal adversarial attacks are more effective on defended models that on clean models.", "questions": "**Questions:**\n- Why is $\\lambda$ set to $1$ rather than $0$? What happens when $\\lambda=0$?\n\n**Suggestions:**\n- It might be good to reference the work that introduced universal (image-agnostic) adversarial perturbations: [^2].  \n- L100: Mention how [34] attempted to enhance the backdoor signal at inference and clarify what this means.\n- L133: introduce the meaning of the $\\boldsymbol x_{\\boldsymbol\\xi}$ notation (there is a triggering function $(\\boldsymbol x, \\boldsymbol\\xi) \\mapsto \\boldsymbol x_{\\boldsymbol\\xi}$).\n- L230: remove the part \"we just divided these defenses into two groups: (1) NC, NAD, i-BAU; and (2) FT-SAM, SAU, FST\" because it makes the reader ask why, but it is explained later and not so important at this point.\n- Fig. (3) c: Mention the attack.\n- Increase the size of Table 3.\n\n[^2]: [Universal adversarial perturbations](https://arxiv.org/abs/1610.08401)", "limitations": "The paper provides a short (but valuable) discussion on limitations. It might be improved by noting (again) that the proposed attack applies to post-training defenses and that other defenses (including test-time defenses) are not considered in the paper. \n\nAdversarial training and randomized smoothing were not considered as stronger defenses.", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "- The paper is very well written and structured. It is clear an easy to read.\n- The \"backdoor re-activation\" investigation is interesting and the proposed method is novel.\n- The experimental evaluation, including validation studies, is very well done and provides valuable insights.\n- The source code is provided and will presumably be published.", "weaknesses": "I do not see any very important weaknesses.\n\nSome phrases are or might be unclear:\n- L9: \"lie dormant\" (the state of the defended model instance does not change).\n- L163: What is is BEC of $f_{\\boldsymbol\\theta_\\text{A}}$? Eq. (3) defines it as a function of multiple model instances.\n- L168: \"greater existence\"\n- L242: What do the percentages represent in particular. Are they percentages (multiplicative) or percentage points (additive)?\n- The claim (2) in L243 is not completely accurate because an important part of the effect is due to the pure effect of the adversarial attack (table 5).\n\nJustification of the definitions in Eq. (2) and (3):\n- Why is CKA a good choice? What principles or assumptions is it based on?\n- Why does BEC include intermediate model activations rather than the final activations?\n\nErrors:\n- Fig 5: 1. The SIG example looks like a product of the image and the sinusoid, but it should be a sum according to the paper [^1]. What caused this? 2. The clean image is not the same crop as the triggered images.\n\nWriting mistakes:\n- L40: missing comma before \"backdoor\",\n- L137: \"remove backdoor\" -> \"remove the backdoor\", defined -> denoted,\n- L173: The equation $\\boldsymbol\\xi'=\\boldsymbol\\xi + \\Delta_{\\boldsymbol\\xi}$ is not consistent with the definition of $\\boldsymbol x_{\\boldsymbol\\xi}$ if the function $(\\boldsymbol x, \\boldsymbol\\xi) \\mapsto \\boldsymbol x_{\\boldsymbol\\xi}$ is not the addition operator.\n- There is no $\\min$ on the right side of Eq. (4).\n- Eq. (5) the model instance uses the same notation as logit functions $f_i$.\n- L200: \"learning task\" -> \"learning tasks\"\n- L204: CIFAR-10c\\[20\\]\n- L253: groupe\n- L258: \"on weak\" -> \"on the weak\"\n- L261: adversarys\n\nI apologize for any errors on my part.\n\n[^1]: [A new Backdoor Attack in CNNs by training set corruption without label poisoning](https://arxiv.org/abs/1902.11237)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Firstly, we would like to show our sincere appreciation to the reviewer for dedicating the valuable time, offering gracious affirmation, and providing constructive.\n\n**Q1. Why is CKA a good choice in Eq. (2) and (3)?**\n\n**R1:** Thanks for this insightful comment. We would like to explain from two aspects:\n* **Our task and challenge:** In this work, we need to measure the quantity of backdoor existence within a model by measuring the backdoor effect similarity between models. We calculate the similarity between feature maps in backdoor related neurons, which are structured and high-dimensioned feature representations. For such representations, traditional metrics such as $L_k$ norm distance or cosine distance could disrupt the structure information of these representations.\n* **CKA is highly suitable for our task:** CKA [1] measures the similarity between two representations, which utilizes HSIC to measure the independence between two distributions. CKA is widely used for measuring similarity of high dimensional representations between networks and thus is highly suitable for our task. Its effectiveness is verified in our work by comparing the relationship between BEC and our re-activation attack ASRs (Fig.3(b)).\n\n**Q2. Why does BEC include intermediate model activations in Eq. (3)?**\n\n**R2:** We compute BEC across the entire network because backdoor attack has a cumulative effect, and each layer contributes to a successful backdoor attack. We have empirically demonstrated that our BEC metric and the re-activation ASR have a high positive correlation in Fig. 3(b) in the main manuscript and the details of ASR with BEC are shown in **Table 1**. The positive correlation shows that BEC is effective in measuring backdoor existence.\n\n**Table 1: Comparison between ASR and BEC.**\n|Attack$\\downarrow$|Defense $\\rightarrow$|NAD|i-BAU|FT-SAM|SAU|FST|\n-|:-:|:-:|:-:|:-:|:-:|:-:\nSSBA|ASR|0.997|0.913|0.928|0.899|0.941\nSSBA|BEC|0.771|0.500|0.502|0.313|0.765\nWanet|ASR|0.962|0.947|0.949|0.953|0.976\nWanet|BEC|0.656|0.733|0.607|0.619|0.758\n\n**Q3. Why is $\\lambda$ set to 1 rather than 0? What if $\\lambda=0$?**\n\n**R3:** \n* **The function:** The second term in Eq.(4) serves as a regularization item, aiming to encourage the model to minimize the highest probability of non-target classes, thereby enhancing the potency of the attack. If $\\lambda=0$, it might lead to the model maintain high output probabilities for non-target classes, thus potentially diminishing the attack performance. \n* **Results: Table 2** illustrates the results of WBA when $\\lambda=0$. It shows a slight decrease in the overall attack effectiveness with $\\lambda=0$.\n\n**Table 2: Re-activation attack performance (%) under different values of $\\lambda$.**\n|Defense$\\rightarrow$|FT-SAM|FT-SAM|SAU|SAU|FST|FST|\n:-:|:-:|:-:|:-:|:-:|:-:|:-:\n$\\lambda\\rightarrow$|0|1|0|1|0|1\nBadNets|93.2|94.7|92.8|93.1|96.1|97.9\nInput-Aware|95.6|96.2|84.2|85.4|89.4|90.7\n\n**Q4. Attack against test-time defenses.**\n\n**R4:** Thanks. Here we test our attack against test-time defenses. \n* **Settings:** With the poisoned samples optimized by our re-activation attack on defense models with FT-SAM, we test the effectiveness against test-time defenses  STRIP[2], ZIP[3], and SCALE-UP[4]. \n* **Results:** **Table 3** shows that our attack maintains a high ASR against ZIP. For the SCALE-UP and STRIP, there is a significant decrease in ASR. However, the ACC of the model is low, meanwhile.\n* **Analysis:** This experiment inspires future design of attack methods capable of evading test-time defense. Strategies might include methods to closely match the feature distributions of clean data, and preventing overly strong activations.\n\n**Table 3: Performance (%) against test-time defenses.**\nDefense$\\rightarrow$|SCALE-UP|SCALE-UP|STRIP|STRIP|ZIP|ZIP\n:-:|:-:|:-:|:-:|:-:|:-:|:-:\nAttack$\\downarrow$|ASR|ACC|ASR|ACC|ASR|ACC\nBadNets|29.8|53.7|83.4|9.3|23.6|80.7\nBlended|34.1|46.2|49.2|9.2|48.1|81.5\n\n**Q5. The SIG example in Fig. 5.**\n\n**R5:** Thanks. It’s somewhat an illusion. The mechanism in use is actually an additive trigger. This aligns accurately with the approach described in the original paper. For the crop, we will modify it in the revision.\n\n**Q6. Some unclear phrases.** \n\n**R6:** Thanks for your careful checking of our manuscript. Our clarification of the indicated phrases is as follows: \n* **L9: \"lie dormant\".** It means that the backdoor cannot be activated by the original trigger, but it still exists in the model. \nL163: What is BEC of $f_{\\theta_A}$?It is 1,because $\\rho_{\\text{BEC}}$  ($f_{\\theta_{\\text{A}}}$, $f_{\\theta_{\\text{A}}}$,$f_{\\theta_{\\text{C}}}$;$D_p$) = $\\frac{1}{N}$ $\\sum_{l=1}^{N}$ $\\frac{S_{\\text{A},\\text{A}}^{(l)}(D_{p})-S_{\\text{C},\\text{A}}^{(l)}(D_{p})}{S_{\\text{A},\\text{A}}^{(l)}(D_{p})-S_{\\text{C},\\text{A}}^{(l)}(D_{p})}=1$. \n\nNote that the second and third arguments in $\\rho_{\\text{BEC}}$ serves as two reference models to measure the backdoor existence of the model corresponding to the first argument. \n* **L168: \"greater existence\".** We will modify it to *stronger existence*.\n* **L242: What do the percentages represent in particular?** They represent absolue improvements on ASR values.\n* **L243: The claim (2) is not completely accurate.** Thanks for this careful comment. Following your suggestion, we will revise it to \"which shows the effectiveness of our re-activation attack method\" in the manuscript. \n\n**Q7. Writing mistakes and suggestions.**\n\n**R7:** We appreciate your attentiveness to the errors in our manuscript. We will revise the paper based on your suggestions.\n\n[1] Similarity of Neural Network Representations Revisited, ICML2019.\n\n[2] STRIP: a defence against trojan attacks on deep neural networks, ACSAC2019.\n\n[3] Black-box backdoor defense via zero-shot image purification, NeurIPS2023.\n\n[4] SCALE-UP: an efficient black-box input-level backdoor detection via analyzing scaled prediction consistency, ICLR2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "C2iAeepfHT", "reviewer": "Reviewer_ULSi", "summary": "Deep neural networks have been demonstrated to be vulnerable to backdoor attacks. Various existing defense strategies have been proposed to remove backdoors. However, this paper observes that backdoors still remain in the models. It introduces a new metric called the backdoor existence coefficient that measures the existence of backdoors in deep neural networks. These dormant backdoors can be easily reactivated during inference with minor perturbations, even under black-box conditions where only model queries are available. Specifically, the paper adds an additional perturbation to the original backdoor trigger and aims to optimize this perturbation to recover the attack effect. The evaluation is conducted on three datasets, two models, and a few multimodal models. The experiments show that the proposed approach can successfully recover the original attack performance.", "questions": "See above.", "limitations": "The authors have addressed the limitations.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper observes that backdoors are simply dormant after applying defenses. They can be re-activated, which is an interesting observation.\n\n2. The proposed metric can measure whether backdoors are indeed removed from the model.", "weaknesses": "1. While the observation is interesting, the method used in this paper to reactivate the backdoors is questionable. What the paper does is simply add an $L_\\infty$ perturbation to the original trigger and then optimize the perturbation. There are two problems with this approach. Firstly, existing work [1] has already shown that it is very easy to generate a universal adversarial perturbation to cause misclassification. How does the paper guarantee that the final optimized trigger is not simply a universal adversarial perturbation? Secondly, natural backdoors can be easily identified in deep neural networks. Existing work [2] has conducted a comprehensive study showing that various injected backdoors can manifest in clean models. How does the paper ensure that the generated triggers are not simply natural backdoors? Actually, the results reported in Table 5 confirm this concern. For example, on a clean model Res18+CIFAR-10, the attack success rate is 85%, which is only 8% lower on a defense model. This shows that the optimization is not specific to \"re-activate\" injected backdoors but simply crafts a natural backdoor.\n\n2. Following the last point, there is no metric used in the paper to measure the faithfulness of the recovered backdoor. Without a comparison between the injected trigger and the recovered trigger, it is hard to distinguish whether the proposed approach actually reactivates the injected backdoor or simply generates a universal adversarial perturbation or a natural backdoor. Without a rigorous evaluation of the generated trigger, the claim of \"re-activation\" should be avoided.\n\n3. There are many recent defenses, such as SEAM [3] and CT [4], that were not evaluated in the paper. It is important to test as many defenses as possible to show the generalizability of the observations. Otherwise, it gives readers a false sense that all the defenses have the same problem.\n\n4. The re-activation is based on the original trigger and an optimized perturbation. What if the original trigger is not used and the perturbation is simply optimized? Will this also achieve the attack performance? If this is the case, it means the generated perturbation is simply a universal adversarial perturbation or a natural backdoor.\n\n\n[1] Moosavi-Dezfooli, Seyed-Mohsen, et al. \"Universal adversarial perturbations.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2017.\\\n[2] Tao, Guanhong, et al. \"Backdoor vulnerabilities in normally trained deep learning models.\" arXiv preprint arXiv:2211.15929 (2022).\\\n[3] Zhu, Rui, et al. \"Selective amnesia: On efficient, high-fidelity and blind suppression of backdoor effects in trojaned machine learning models.\" 2023 IEEE Symposium on Security and Privacy (SP). IEEE, 2023.\\\n[4] Qi, Xiangyu, et al. \"Towards a proactive {ML} approach for detecting backdoor poison samples.\" 32nd USENIX Security Symposium (USENIX Security 23). 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the positive review on the interesting observation. Your insightful questions and concerns are greatly appreciated. Please inform us if these responses effectively address all your inquiries.\n\n**Q1. The differences among our re-activation attack (RBA), the original backdoor attack (OBA), and general universal adversarial perturbation attack (gUAA, i.e., natural backdoor).**\n\n**R1:** Thanks for your insightful concerns.\nIn the **Common Response**, we conduct a comprehensive comparison among our RBA, the original backdoor attack (OBA), and gUAA. This analysis is undertaken through three distinct aspects: the activation mechanism of the backdoor effect, the search rate, and the robustness against random noise. The three-perspective analyses verify that our RBA method finds a highly correlated backdoor with the original backdoor, rather than a less correlated backdoor (i.e., *new backdoor*), or a general universal adversarial perturbation (i.e., *natural backdoor*). Thus, we can claim that our RBA actually re-activates the orginal backdoor.\n\nPlease refer to **Common Response** for our analysis. Hope above points could address your concern, and we will clearly clarify the differences in the revised manuscript.\n\n**Q2. Suggestion of adding evaluations of some recent defenses, such as SEAM and CT.**\n\n**R2:** Thanks for this constructive suggestion. \n- **Additional evaluations:** Following your suggestion, we evaluate the performance of our re-activation attack against SEAM and CT, respectively. The evaluations are conducted on CIFAR-10 dataset with PreAct-ResNet18 network, and the results are shown in **Table 1**. It is found that both SEAM and CT are vulnerable to the proposed re-activation attack. \n- **Clarification:** We would like to emphasize that we have never claimed that all post-training defenses are vulnerable to re-activation attack, which is not rigorous. The main aim and contribution of our work are (1) **revealing this new treat**, which has been verified on several classic post-training defenses, and (2) **providing effective tools** to evaluate the vulnerability of any old or new post-training defenses. Consequently, the future post-training defenses should consider this threat and bypass the proposed re-activation attack. \n\n**Table 1: ASR (%) of our RBA attack against SEAM and CT.**\n|Post-training defense $\\rightarrow$|SEAM|SEAM|SEAM|CT|CT|CT|\n:-:|:-:|:-:|:-:|:-:|:-:|:-:\nRe-activation attack $\\rightarrow$, Original attack $\\downarrow$\\|No re-activation|No re-activation|WBA|BBA|No re-activation|WBA|BBA\nBadNets|5.33|97.53|33.51|0.00|99.58|92.21\nBlended|6.79|98.40|69.79|1.34|100.00|99.00\nInput-aware|1.27|92.10|48.59|70.95|99.96|85.80\nLF|13.22|97.61|65.63|3.28|99.63|99.23\n\n**Q3. What if the original trigger is not used and the perturbation is simply optimized?**\n\n**R3:** Thanks. Our response is expanded from two aspects: \n- **If the original trigger is not used, it devolves into a general universal adversarial perturbation attack (gUAA).** Actually, the distinctions between our RBA attack (starting from the original trigger) and gUAA has been carefully analyzed from three perspectives in the **Common Response**. It is clear that our re-activation attack re-activates the orginal backdoor attack, rather than finds a general universal adversarial perturbation or a natural backdoor. \n- **Additional comparions under the white-box attack setting and analysis.** In addition to the black-box comparions shown in Table 2 of the *Common Response*, here we also supplement the white-box comparisons, as shown in **Table 2** below. Although the ASR values of gUAA is also high, its activation mechanism is intrinsically different with our re-activation attack (refer to the analysis in *Common Response*). **gUAA just verifies a well-known threat once again** that DNN models are likely to be vulnerable to (universal) adversarial attacks. In contrast, **our re-activation reveals a new threat** to existing post-training backdoor defenses. \n\n**Table 2: Comprisons between our RBA and gUAA under the white-box setting, measured by ASR (\\%).**\n|Defense$\\rightarrow$|SAM|SAM|SAU|SAU|\n:-:|:-:|:-:|:-:|:-:\nOriginal backdoor $\\downarrow$, Attack $\\rightarrow$|gUAA|RBA|gUAA|RBA\nInput-Aware|83.08|96.19|78.72|85.39\nLF|87.29|97.40|83.78|90.74\nSSBA|84.52|92.80|83.50|89.86\nTrojan|88.34|96.18|79.04|87.61\nWanet|82.07|94.95|80.82|95.33", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jdLYszUWN7", "reviewer": "Reviewer_G6gk", "summary": "This paper present a novel insight in adversarial robustness, which is posed by the effective removal of backdoor attacks once defenses have been applied. Interestingly, the authors claim that these are still embedded inside compromised machine learning models.\nHence,  the authors propose a novel metric, the Backdoor Existence Coefficient (BEC) to detect this issue.\nThe authors show that both models with/without backdoor defenses correlates positively with this metric, while clean models are not.", "questions": "**Are the authors sure that their re-activation is not creating novel backdoors?** The authors must clarify this crucial aspect. If so, the paper is really bringing plenty of contributions to the community. Otherwise, it is just another backdoor attack that is not adding much to the discussion around backdoors. I will be willing to increase my score if this point is clarified.", "limitations": "The authors address the limitations of their work.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Very interesting insights on backdoor defenses.** BEC is timely and interesting, since it addresses a relevant problem. Also, this metric can be used to understand wheather a defense is correctly working or not, opening the road to novel techniques that strip backdoors away.\n\n**Clear description of the internals of the technique.** Authors clearly explain how they retrieve neurons, and how to use CKA to compare them with back-doored models.", "weaknesses": "**Backdoor reactivation sounds like new backdoors** My only concern is posed by the fact the reconstruction of backdoor is similar to injecting one. Thus, it is not clear whether the attack is creating a novel backdoor or it is really manipulating weights to make that backdoor active again. \n\n**Unclear transfer evaluations.** The authors state that transfer attacks exhibit strong performance, but the authors do not provide a clarification on how. In general, this is counterintuitive, since black-box evaluations should be less effective than the ones that exploit the perfect knowledge on the model. The authors should better clarify this point.\n\n**(Minor) Paper structure can be improved.** Both abstract and introduction focus on the metric BEC, but then it is neglected to a brief discussion in section 4 (figure 3).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank the reviewer for the valuable time in reading our work and constructive concerns. We are encouraged by the positive comments of very interesting insights and clear description.\n\n**Q1. Whether backdoor re-activation is creating a novel backdoor or not.** \n\n**R1:** Thank you for your insightful question. We would like to refer you to the **common response** for a comprehensive analysis on the relationship between the original backdoor attack (OBA) and our re-activation attack (RBA) in three perspective. \n\nMoreover, we want to clarify that the attacks proposed in this work involve three scenarios, namely white-box scenario, black-box scenario, and transfer-based scenario, where the latter two representing more realistic situations. However, none of the three attacks involve revising model parameters.\nParticularly, black-box attacks and transfer attacks operate under the assumption that the attacker lacks access to the target model's parameters. Thus, we are not engaged in any form of weight manipulation.\n\nOverall, our re-activation backdoor attack shows a strong correlation with the original backdoor and our attacks involve no modification of network weights. Hope the above points could address your questions.\n\n**Q2. Clarification of the setting of transfer attack (TA).**\n\n**R2:** Thanks for this concern. We would like to clarify the setting of transfer-based re-activation attack (TA), and present some analyses about its effectiveness.\n* **Setting of TA in our main manuscript.**\n     - **Threat model:** As demonstrated in Line 132-135 of the main manuscript, the adversary trains a backdoored model $f_\\text{A}$ and releases it to users. The user receives model $f_\\text{A}$, and obtains a defense model $f_\\text{D}$ based on $f_\\text{A}$ by some post-training defense. Thus, *the adversary does not know the exact defense method, but has full information about the original trigger* $\\boldsymbol{\\xi}$ *and model* $f_\\text{A}$ *which has same model architecture as* $f_\\text{D}$. \n     - **Main steps in TA**: There are three main steps in TA. **Step 1:** Based on $f_\\text{A}$, the adversary can obtain a surrogate model (or a collection of surrogate models) by utilizing some existing post-training backdoor defenses, denoted as $f_{\\text{D}'}$. **Step 2:** The adversary conduct a white-box re-activation atack against $f_{\\text{D}'}$, based on the original trigger $\\xi$, to obtain a new trigger $\\xi_{D'}'$. **Step 3:** The adversary adopts $\\xi_{D'}'$ to attack the target defense model $f_{\\text{D}}$ during inference.\n    - **Why TA is effective for re-activating backdoor:** As analyzed in *Q1 of Common Response* and Sec. 4.4 in the main manuscript, the original backdoor still exists in defense models. In other words, the backdoor effects of both the surrogate model $\\xi_{D'}'$ and the target model $\\xi_{D}'$ are highly similar with that of the original backdoored model $f_\\text{A}$. Thus, it is natural to deduce that $\\xi_{D'}'$  and $\\xi_{D}'$ also have similar backdoor effects. It explains the high ASR of our TA. \n    \n* **Further evaluation with a more strict setting: transfer attack across model architectures.**\n    - **Threat model:** Here we present a more strict setting that the adversary can only manipulate training dataset, while has no access to the training and post-training stage. Thus, the adversary only knows the original trigger $\\boldsymbol{\\xi}$, but has no knowledge of $f_\\text{A}$ or $f_\\text{D}$. Compared to above threat model, **one major challenge is the unknown architecture of the target model $f_\\text{D}$**.\n    - **Main attack steps:** Compared to the steps in the above setting, there is one additional step that the adversary should firstly train a backdoored model $f_\\text{A}'$ based on $D_p$, which has different architectures with $f_{\\text{D}}$. All remaining steps are same with those in the above setting.\n    - **Experimental results:** As shown in **Table 1**, although transfer attack across model architectures doesn't achieve as high ASR as the transfer attack with the same architecture (*i.e.*, results in Table 3 of the main manuscript), it still has a certain degree of transferability of backdoors, which is an intriguing phenomenon worthy of further exploration.\n \n**In summary**, the effectiveness of the transfer re-activation is mainly due to the high backdoor effect similarity between surrogate and target defense models, even in different architectures. We think its performance could be further improved by borrowing ideas from the field of transfer adversarial attacks. Its practical threat should be seriously considered by future defenses. \n\n**Table 1: Transfer re-activation attack ( ASR %) against the target model PreAct-ResNet18, using different source models (WideResNet28-2, ResNet18, VGG19-BN).**\n|Source Model|WRN28|WRN28|WRN28|ResNet18|ResNet18|ResNet18|VGG|VGG|VGG|\n:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:|:-:\nDefense$\\rightarrow$ Attack$\\downarrow$|i-BAU|FT-SAM|SAU|i-BAU|FT-SAM|SAU|i-BAU|FT-SAM|SAU\nBadNets|95.6|84.1|60.0|53.4|30.9|26.5|89.2|71.7|48.6\nBlended|98.5|98.5|83.2|79.1|75.5|64.1|97.9|92.8|90.1\n\n**Q3. Paper structure can be improved.**\n**R3:** Thanks for this constructive suggestion. We would like to clarify that since the backdoor existence phenomenon and its metric BEC are the most critical contributions of this work, we emphasize them in both Abstract and Introduction. Following your suggestion, we will expand Sec. 3.2 by adding some analysis about the characteristics of the proposed BEC score in the revised manuscript. \n\n**Finally, we sincerely hope that above responses could well address your concerns. We will be more than glad to have further discussions with you if there is any remaining concerns. Once again, we express our heartfelt gratitude for your valuable and constructive comments.**", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yGQ2sbRBBV", "reviewer": "Reviewer_ccvM", "summary": "The paper examines whether backdoors in deep learning models processed by defense algorithms can be reactivated and how to achieve this reactivation. First, it proposes a metric to measure the presence of backdoors, called the backdoor existence coefficient, which ranges from 0 (backdoors are nonexistent) to 1 (backdoors are still present). This coefficient is computed in two steps: (i) identifying the backdoor-related neurons in the attacked (not defended) model, and (ii) computing the similarity between the feature maps of a subset of backdoor-related neurons in the clean model, the attacked model, and the defended model.\n\nSecond, the paper proposes three attacks: one for the white-box setting, one for the black-box setting, and one that exploits transferability to perform a reactivation attack against the defended models. The fundamental building block consists of finding a universal adversarial perturbation that can be applied to all instances with the trigger that previously worked, making it effective again.\n\nThe experimental evaluation considers six defenses, seven attacks, and three image datasets along several axes. It shows that the proposed attack is effective in finding the universal adversarial perturbation, i.e., in modifying the poisoned instances in a way that reactivates the backdoor.", "questions": "- Could you please provide more details about the transfer-based attack? (See Weaknesses section for more details)\n- How can adding the perturbation to already perturbed instances impact their detectability? (See Weaknesses section for more details)\n\n**Updates after authors' response**: the authors provided insightful responses to the questions and provided data and arguments for successfully solving the weaknesses listed above. I strongly invite the authors to add the content of their response to the next version of their paper.", "limitations": "The authors have summarized the limitations of their proposals in a specific section of the paper, but they do not address the problem of the detectability of the generated attacks. They also discuss the potential negative societal impacts, asserting that their work may lead to the development of new defenses. Finally, the details of the experimental evaluation and implementations are extensively documented.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "I think this is an interesting paper of moderate-impact that sheds light on the fact that backdoor defenses may not effectively remove the presence of backdoors in the attacked model. In my opinion, the paper shows the following strengths:\n\n- **Novel metric to evaluate the presence of backdoors in a model after defense**: the paper proposes a new metric that demonstrates the backdoor is still implanted in the model even after being processed by a defense procedure. Moreover, it shows that the backdoor existence score is strongly positively correlated with the attack success rate of the three proposed attacks for the majority of the considered defenses. This indicates that the score is reliable in showing the existence of the backdoor in the model and can be adopted by practitioners to assess the presence of backdoors after applying a defense algorithm.\n\n- **The three proposed attacks show reasonable effectiveness**: the three proposed attacks are able to generate new poisoned instances that exhibit a high success rate compared to the success rate of the original trigger instances on the defended model.\n\n- **Comprehensive experimental evaluation**: the experimental evaluation is thorough and detailed, considering several useful perspectives, such as the influence of the size of poisoned samples, and various settings.", "weaknesses": "Even though I appreciate the conceptual contribution of the paper, I think that it needs to be improved in some aspects:\n\n- **Presentation of the surrogate attack is confusing and lacks of details**: I consider the transfer-based attack to be the most interesting and realistic one since it is deployable even if the attacker has no access to information about the target defense. However, it is not clearly explained. The authors should clearly state the assumptions behind this attack, such as whether the attacker has access to a surrogate model imitating the target model in its clean, attacked, or defended state, and how the attacker can obtain that surrogate model. Additionally, the authors should clearly highlight the differences between the knowledge the attacker has in the black-box and transfer-based scenarios. For example, the authors state that \"the adversary lacks prior knowledge of the defense model\" in Section 3.3 which seems to be the same assumption as the black-box setting. Therefore, it is unclear what the pros and cons of the surrogate-based attack are compared to the black-box attack and vice versa.\n\n- **Detectability of the attacked poisoned instances is not discussed**: the proposed attacks involve finding a universal adversarial perturbation to add to test instances that already contain the trigger. An important aspect that should be evaluated is the detectability of the perturbed instances since they may be easily detected after perturbation by defenses like [a] (but many other defenses at inference time exist, see [b]). The authors should show the appearance of some examples generated by the three proposed attacks and evaluate the detectability of the perturbed images against some defenses to make their proposal more convincing.\n\n- **Only mean results are shown**: the authors states that ```all experiments are executed 5 times with varying random seeds``` in Section D of the Appendix. Therefore, the authors should also show the standard deviations of the success rates of the attacks presented in the tables of the paper, as this would help evaluate the variability of the success rate of the proposed attacks over different trials.\n\nFinally, I report a minor weakness from a section of the Appendix:\n- **Description of the running times is confusing**: I appreciate that the authors reported the running times of their proposed attacks on different architectures in Section F of the Appendix. However, they should clearly describe the settings in which these running times are obtained, such as the considered attack and defense, to better support their analysis. Moreover, they should explain why some results are labeled as ```N/A```.\n\n[a] Tramer et. al., SentiNet: Detecting localized universal attacks against deep learning systems, IEEE Security and Privacy Workshops, 2020\n\n[b] Cinà et. al., Wild Patterns Reloaded: A Survey of Machine Learning Security against Training Data Poisoning, in ACM Computing Surveys, 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We deeply appreciate the reviewer's valuable time and thoughtful comments.\n\n**Q1. Details of the surrogate-based attack (i.e., the transfer attack), and the pros and cons of the surrogate-based attack in comparison with the black-box attack (BBA).**\n\n**R1:** Thanks for this constructive comment. We would like to refer you to our response to **Q2 of Reviewer G6gk**, where we clarify the threat model, main attack steps, as well as additional evaluations and analyses. \n\nWe compare our transfer attack (TA) and BBA in two aspects: \n* **Attack effectiveness:** According to reported results in Table 2 and Table 3 of the main manuscript, TA achieves higher ASR than BBA in most cases because:\n    * **For BBA**: The effectiveness of BBA is limited by the specific black-box attack method (Square attack in this work). More advanced method might improve the performance. It is also limited by query numbers, where we show that ASR can be improved with increasing query numbers in **Table 2 of Common Responce**.\n    * **For TA**: As analyzed in the **Common Responce**, there is a backdoor effect similarity between the original backdoored model and the defense model. Thus it is easier to optimize a highly transferable trigger.\n* **Attack efficiency:** BBA requires lots of queries to the target model to achieve satisifying attack performance, while TA just needs one query to attack the target model. Obviously, TA is more efficient and more practical than BBA. \n\n**In summary**, TA has better performance on both effectiveness and efficiency than BBA. However, we would like to emphasize that our main goal of designing re-activation attack algorithms under three scenarios is to verify the backdoor existence in defense models.  All three attacks could be further improved by developing more advanced algorithms. \n\n**Q2. Detectability of the attacked poisoned instances (*i.e., new poisoned samples obtained by our re-activation attack*) is not discussed.**\n\n**R2:** Thanks for this constructive comment. It is valuable to study whether the proposed attack will increase the detectability during inference. Our study is as follows:\n* **Settings:** We adopt three representative inference-time poisoned sample detection methods including SCALE-UP [1], SentiNet [2], and STRIP [3]. The detection task requires two input arguments, including the model and the query datasets. We evaluate on five pairs, including <the orginal backdoored model $f_{\\text{A}}$, the original poisoned dataset $\\mathcal{D}_p$>, <the defense model with FT-SAM defense $f_{\\text{D,FT-SAM}}$, dataset $\\mathcal{D}_p$>, <model $f_{\\text{D,FT-SAM}}$, the re-activation dataset $\\mathcal{D}_{p,\\Delta{\\xi}}$>, <the defense model with SAU defense $f_{\\text{D,FT-SAM}}$, dataset $\\mathcal{D}_p$>, <model $f_{\\text{D,SAU}}$, the re-activation dataset $\\mathcal{D}_{p,\\Delta{\\xi}}$>.\n* **Results and analyses:** \n    * For **test-time detection**, the result in **Table 1** shows our attacks do not markedly increase the TPR compared the the other two pairs. More detection performance on our BBA and TA are shown in **Table 6 of PDF**. The appearance of generated poisoned samples are also provided in **Figure 2 of PDF**.\n    * For **test-time defense**, our attack under test-time defenses has been shown in the response to **Q4 of Reviewer AhAB**, where our attack maintains a certain ASR against these defenses. \n\n**In conclusion**, these findings offer valuable insights, paving the way for the future design of more stealthy re-activation attacks. \n\n**Table 1: Detection performance (TPR %) of different poisoned samples under three detection methods.**\nAttack|Detection$\\downarrow$|$f_{\\text{A}},\\mathcal{D}_p$|$f_{\\text{D,FT-SAM}},\\mathcal{D}_p$|$f_{\\text{D,FT-SAM}},\\mathcal{D}_{p,\\Delta{\\xi}}$|$f_{\\text{D,SAU}},\\mathcal{D}_p$|$f_{\\text{D,SAU}},\\mathcal{D}_{p,\\Delta{\\xi}}$\n:-:|:-:|:-:|:-:|:-:|:-:|:-:\nBadNets|SCALE-UP|39.6|79.6|68.6|79.5|49.5\nBadNets|SentiNet|37.7|3.6|2.2|0.2|0.9\nBadNets|STRIP|88.3|0.7|5.5|10.3|6.5\nTrojan|SCALE-UP|92.6|84.9|73.1|81.6|55.4\nTrojan|SentiNet|2.9|1.1|1.05|2.1|1.5\nTrojan|STRIP|99.9|1.9|29.8|4.2|1.2\n\n**Q3. Only mean results are shown.**\n\n**R3:** Thanks. \n* **Results**: Here we provide the mean results with standard deviations (std.) under five different random seeds among several attacks and defenses in **Table 2** for our black-box attack. More results of our WBA and TA is presented in **Table 4 and 5** in **PDF**.\n* **Analysis**: It can be observed that the std. is small among these attack and defense method across three of our attacks. It shows our attack methods are stable. \n\n**Table 2: Mean result with std. for our BBA.**\n||NAD|i-BAU|FT-SAM|SAU\n:-:|:-:|:-:|:-:|:-:\nBadNets|49.4$\\pm$ 2.6|57.7$\\pm$ 4.6|44.1$\\pm$ 1.2|37.5$\\pm$ 2.8\nBlended|14.6$\\pm$ 0.3|94.4$\\pm$ 2.9|90.1$\\pm$ 4.0|83.2$\\pm$ 5.1\n\n**Q4. Description of the running time.**\n\n**R4:** Thanks.\n* We would like to clarify that the running time of our attack is only related to trainning dataset and network, while independent with specific backdoor attack or defense methods. Thus, we didn't specify the particular method, as the running time is consistent across methods.\n* Regarding the \"N/A\" in Table 10, we executed WBA and TA on the CLIP models. For query-based black-box attack, the attacker cannot directly access the target model (such as weights or gradients) and CLIP models only return the final matching score or ranking results. This limits the ability of query-based black-box attacks. Moreover, there are no relevant studies for reference. Thus, we marked related result as \"N/A\".\n\nWe hope this explanation resolves your queries. We welcome your feedback or further inquiries. Thanks again.\n\n[1] SCALE-UP: An efficient black-box input-level backdoor detection via analyzing scaled prediction consistency, ICLR2023.\n\n[2] Sentinet: Detecting localized universal attacks against deep learning systems, SPW2020.\n\n[3] STRIP: A defence against trojan attacks on deep neural networks, ACSAC2019.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank all reviewers for their valuable time and constructive comments, and are encouraged by the positive comments of **very interesting insights** (AhAB,ULSi,G6gk,ccvM), **novel ideas and good contribution** (AhAB,G6gk,ccvM), **moderate-impact** (ccvM), **extensive experimental evaluation** (AhAB,ccvM), and **good writing** (AhAB,G6gk).

We would like to present a common response to a very critical comment, mentioned by **Reviewer ULSi** and **Reviewer G6gk**. 

**Q1. Systematic comparison among original backdoor attack (OBA), re-activation attack (RBA), and general universal adversarial perturbation attack (gUAA).**

**R1:** We highly appreciate this insightful comment. We analyze the relationships among RBA, OBA and gUAA from three perspectives:
* **Definitions and settings:** To facilitate the understanding of our analysis, we firstly clarify the definitions and settings: 
    - **OBA** means one existing backdoor attack following the standard backdoor injection and activation process. The trigger and backdoored model are denoted as $\boldsymbol{\xi}$ and $f_\text{A}$, respectively; 
    - **RBA** means that given the defense model $f_\text{D}$  (*i.e.*, conducting one post-training defense on $f_\text{A}$), we aim to re-activate the injected backdoor of OBA via searching for a new trigger $\boldsymbol{\xi}'$, starting from $\boldsymbol{\xi}$, *i.e.*, based on some original poisoned samples $D_p$. The searched new trigger is formulated as $\boldsymbol{\xi}' = \boldsymbol{\xi} + \Delta_{\boldsymbol{\xi}}$, and the new poisoned dataset is denoted as $D_{p,\Delta_{\boldsymbol{\xi}}}$; 
    - **gUAA** means that given $f_\text{D}$, we aim to search for a targeted universal adversarial perturbation (the same class with OBA and RBA) via adversarial attack, starting from clean samples $D_c$. The searched UAP is denoted as $\Delta$, and the perturbed dataset is denoted as $D_{c, \Delta}$. *Note that it is also called natural backdoor by Reviewer ULSi.* 

* **Analyses:** Our analyses are expanded as below: 
    - **Activation mechanism of backdoor effect:** We analyze the backdoor activation mechanism in each attack. As demonstrated in Line 151-161 of Sec. 3.2, we could adopt the Centered Kernel Alignment (CKA) metric to measure backdoor effect similarity between models, by comparing the backoor-related neurons' activation maps (*i.e.*, $\tilde{m}$ in Line 156). Specifically, we calculate the following three CKA scores: 

       $S_{\text{RBA,OBA}}=\frac{1}{N}\sum_{l=1}^{N}\text{CKA}(m_{\text{D}}^{(l)}(D_{p,\Delta_{\xi}}), m_{\text{A}}^{(l)}(D_p))$, 

       $S_{\text{gUAA,OBA}}=\frac{1}{N}\sum_{l=1}^{N}\text{CKA}(m_{\text{D}}^{(l)}(D_{c,\Delta}), m_{\text{A}}^{(l)}(D_p))$, 

      $S_{\text{RBA,gUAA}}=\frac{1}{N}\sum_{l=1}^{N}\text{CKA}(m_{\text{D}}^{(l)}(D_{p,\Delta_{\xi}}),m_{\text{D}}^{(l)}(D_{c,\Delta}))$. 

       As shown in **Table 1**, $S_{\text{RBA,OBA}} \gg S_{\text{gUAA,OBA}} \approx S_{\text{RBA,gUAA}}$ at all pairs of attack-defense. It demonstrates that **the backdoor activation mechanisms between RBA and OBA are highly similar, and both of them differ significantly with that of gUAA**. 
    - **Starting from the original trigger $\boldsymbol{\xi}$, it is easier and faster to find a new trigger $\boldsymbol{\xi}'$ that achieves high attack success rate (ASR):** As shown in **Table 2**, given same query numbers, the ASR of RBA is much higher than that of gUAA, and the increasing speed of the former is faster than the latter. It demonstrates that **RBA is much closer to OBA than gUAA**. 
    - **Compared to gUAP $\Delta$, both the orginal trigger $\boldsymbol{\xi}$ and the new trigger $\boldsymbol{\xi}'$ are much more robust to random noise.** We have found a characteristic, *i.e.*, the robustness to random noise, to distinguish the trigger of intended backdoor from the trigger of natural backdoor (*i.e.*, the gUAP). Specifically, we perturb $\boldsymbol{\xi}, \boldsymbol{\xi}', \Delta$ with same level of random noises, and record the ASR of these attacks. As shown in **Table 3**, both OBA and RBA are more robust than gUAA. It verifies that RBA returns an intended backdoor trigger similar to OBA, rather than a gUAP.
* **In conclusion**, we believe above analyses verify that *our RBA method finds a highly correlated backdoor with the original backdoor, rather than a less correlated one (*i.e.*, new backdoor), or a general UAP (*i.e.*, natural backdoor)*. Thus, we claim that **our RBA actually re-activates the orginal backdoor**. 

The full versions of Tables 1-3 are shown in **PDF** for reference. We also provide a schematic diagram in **Fig. 1 of PDF** illustrating the relationship among RBA, OBA and gUAA.
We will add above analyses into the revised version to better demonstrate the mechanism of our method. 

**Table 1: CKA scores between OBA, RBA, and gUAA.**
|Defense$\Rightarrow$|i-BAU|i-BAU|i-BAU|FT-SAM|FT-SAM|FT-SAM
:-:|:-:|:-:|:-:|:-:|:-:|:-:
Attack$\downarrow$|$S_{\text{RBA,OBA}}$|$S_{\text{gUAA,OBA}}$|$S_{\text{RBA,gUAA}}$|$S_{\text{RBA,OBA}}$|$S_{\text{gUAA,OBA}}$|$S_{\text{RBA,gUAA}}$
BadNets|0.607|0.192|0.170|0.599|0.194|0.169
Blended|0.712|0.196|0.192|0.712|0.197|0.193

**Table 2: ASR (%) of RBA and gUAA with different query numbers.**
|Attack+Defense|Query number$\Rightarrow$|1000|3000|5000|7000
:-:|:-:|:-:|:-:|:-:|:-:
Blended+i-BAU|RBA|77.3|89.3|92.1|94.6
Blended+i-BAU|gUAA|14.2|41.4|49.5|56.4
Blended+FT-SAM|RBA|41.1|77.4|79.8|85.6
Blended+FT-SAM|gUAA|16.3|42.2|56.5|65.5

**Table 3: ASR (%) of OBA, RBA and gUAA under different $l_{\infty}$-norm of random noise.**
||Norm$\Rightarrow$|0|0.03|0.06|0.09
:-:|:-:|:-:|:-:|:-:|:-:
OBA|Blended+NAD|99.8|99.8|99.6|97.3
OBA|LF+NAD|99.1|98.9|98.4|98.6
RBA|Blended+NAD|99.8|99.7|98.7|84.0
RBA|LF+NAD|99.4|99.1|98.1|96.6
gUAA|Blended+NAD|95.5|92.7|79.4|35.4
gUAA|LF+NAD|96.5|89.5|55.8|16.7

Meta-review: The paper under review presents a novel investigation into the resilience of deep neural networks to backdoor attacks, even after the application of post-training defense mechanisms. The authors propose a new metric, the Backdoor Existence Coefficient (BEC), to measure the persistence of backdoor vulnerabilities in defended models. The primary contribution of the paper is the introduction of the Re-Activation Backdoor Attack (RBA), which demonstrates that backdoors can be reactivated in defended models by slightly perturbing the original backdoor trigger. Based on the reviewers’ detailed assessments and the authors’ comprehensive rebuttal, I recommend accepting the paper with a suggestion for minor revisions. The paper’s contribution to understanding and quantifying the persistence of backdoors in defended models is valuable and timely, given the growing focus on the security of AI systems. However, the authors should take the reviewers’ feedback into account in the final revision to improve the clarity and rigor of the presentation. Specifically, clearer articulation of the reactivation attack’s nature, further details on surrogate model usage, and a thorough discussion on the detectability of the reactivated backdoors will strengthen the overall impact of the work.
Decision: Accept (poster)