Revisiting the Assumption of Latent Separability for Backdoor Defenses
Authors: Xiangyu Qi
Tinghao Xie
Yiming Li
Saeed Mahloujifar
Prateek Mittal
Venue: ICLR 2023 poster (2023)
Keywords: Backdoor Attacks

Abstract: Recent studies revealed that deep learning is susceptible to backdoor poisoning attacks. An adversary can embed a hidden backdoor into a model to manipulate its predictions by only modifying a few training data, without controlling the training process. Currently, a tangible signature has been widely observed across a diverse set of backdoor poisoning attacks --- models trained on a poisoned dataset tend to learn separable latent representations for poison and clean samples. This latent separation is so pervasive that a family of backdoor defenses directly take it as a default assumption (dubbed latent separability assumption), based on which to identify poison samples via cluster analysis in the latent space. An intriguing question consequently follows: is the latent separation unavoidable for backdoor poisoning attacks? This question is central to understanding whether the assumption of latent separability provides a reliable foundation for defending against backdoor poisoning attacks. In this paper, we design adaptive backdoor poisoning attacks to present counter-examples against this assumption. Our methods include two key components: (1) a set of trigger-planted samples correctly labeled to their semantic classes (other than the target class) that can regularize backdoor learning; (2) asymmetric trigger planting strategies that help to boost attack success rate (ASR) as well as to diversify latent representations of poison samples. Extensive experiments on benchmark datasets verify the effectiveness of our adaptive attacks in bypassing existing latent separation based backdoor defenses. Moreover, our attacks still maintain a high attack success rate with negligible clean accuracy drop. Our studies call for defense designers to take caution when leveraging latent separation as an assumption in their defenses. Our codes are available at https://github.com/Unispac/Circumventing-Backdoor-Defenses.

Published as a conference paper at ICLR 2023
REVISITING THE ASSUMPTION OF LATENT SEPARA- BILITY FOR BACKDOOR DEFENSES
Xiangyu Qi'; Tinghao ; Yiming Li Saeed Mahloujifar', Prateek Mittal
Princeton University Tsinghua Shenzhen International Graduate School, Tsinghua University (xiangyugi, thx, s sfr, pwitttaly@priinetoo... edu; 1i Li-ymll-aai-m pym18mmails wwwgguut.iinnnunnneeo
ABSTRACT
Recent studies revealed that deep learning is susceptible to backdoor poisoning attacks. An adversary can embed a hidden backdoor into model to manipu- late its predictions by only modifying a few training data, without controlling the training process. Currently, tangible signature has been widely observed across a diverse set of backdoor poisoning attacks - models trained on a poi- soned dataset tend learn separable latent representations for poison and clean samples. This latent separation is so pervasive that a family of backdoor de- fenses directly take it as a default assumption (dubbed latent separability as- sumption), based on which to identify poison samples via cluster analysis in the latent space. An intriguing question consequently follows: is the latent sepa- ration unavoidable for backdoor poisoning allucks? This question is central to understanding whether the assumption of latent separability provides a reliable foundation for defending against backdoor poisoning attacks. In this paper, we design adaptive backdoor poisoning attacks to present counter-examples against this assumption. Our methods include two key components: (1) set of trigger- planted samples correctly labeled to their semantic classes (other than the tar- get class) that can regularize backdoor learning: (2) asymmetric trigger plant- ing strategies that help to boost attack success rate (ASR) as well as to diver- sify latent representations of poison samples. Extensive experiments on bench- mark datasets verify the effectiveness of our adaptive attacks in bypassing ex- isting latent separation based defenses. Our codes are available hhtpp: t/ghthub. ctt. in spac/ Ci reumvent Backono Deffesses
INTRODUCTION
Overparameterized deep neural network (DNN) models can fit complex datasets perfectly and gen- eralize well on i.i.d. data distributions. However, the strong capacity of these models also render them susceptible to backdoor poisoning attacks (Gu et al.] 2017; Chen et al.] 2017] Turner et al.] 2019 Li et al. 2022). In backdoor poisoning attack, an adversary only manipulates small por- tion of the victim's training dataset. The victims will train their own model on the manipulated dataset and consequently get a backdoored model. Typically, the adversary will poison the victim's dataset by injecting a small amount of backdoor poison samples, each of which contains a backdoor trigger (e.g. specific pixel patch) and is labeled to specific target class. A DNN model trained on this poisoned dataset will be backdoored in that they tend to learn an artificial correlation between the backdoor trigger and the target class. These attacks are stealthy since backdoored models behave normally natural samples and therefore users can hardly identify them.
Despite the stealthiness in terms of model performance on natural samples, has been commonly observed (Tran et 2018 Chen et al.) 2019 Huang et al., (022) that backdoor poisoning attacks tend to leave tangible signatures in the latent space of backdoored models. As visualized in Fig [b Fig Ig] poison and clean samples from the target class consistently form two separate clusters in the latent space, across diverse set of backdoor poisoning attacks. The pervasiveness of the latent separation renders itself oftentimes assumption in this work. A family of defenses (i.e., latent separation based backdoor defenses) explicitly base their designs this assumption. These defenses first train base classifier on the
default assumption, which call latent separability

---

Published as a conference paper at ICLR 2023
(a) No Poison
(b) BadNet Cu et al 2017
(c) Blend hen et al. 2017
(d) CL Turner al. 2019
( TaCT
Lang et
( ISSBA Li 2021d
Dynamic N guyyn & Tran 2020
(h) Hap-Blend (Ours)
Adap-Putch (Ours)
Figure 1: T-SNE visualization of latent separability characteristic on CIFAR-10. Each point in the plots corresponds to a training sample from the target class. Caption of each subplot specifies its corresponding poison strategy. To highlight the separation, all poison samples are denoted by red points, while clean samples correspond to blue points.
poisoned dataset, and expect the base model will naturally learn separable latent representations for poison and clean samples respectively. After that, they perform cluster analysis the latent space of the base model. If the latent separation characteristics reliably arise, these defenses will be able to identify the outlier cluster formed by poison samples, and thus accurately filter out these poison samples from the training set. We note that this family of defenses are particularly important and successful in the backdoor defense literature. Popular proposals in this family like Spectral Signature (Tran et al. (0188) and Activation Clustering (Chen et al. 2019) have already become indispensable baselines. and recent
of-the-rrt proposals including SCAn (Tang al.] (02T)) and SPECTRE (Hayase et al. (02T) in this family even claim to achieve nearly perfect recall with negligible false positive rate against a diverse set of attacks. Given the pervasiveness of the latent separation and its profound success in the application of backdoor defenses, natural question arises: Is the latent separation unavoidable for backdoor poisoning attacks?
In this work, we revisit the assumption of latent separability and expose failure regions of defenses based on it. Specifically, we design adaptive backdoor poisoning attacks (without control of the model training process). which actively suppress the latent separation while maintaining high attack success rate (ASR) with negligible clean accuracy drop. Two critical components are un- derlying the design of our adaptive attacks (see Fig 2 for overview): (1) Data poisoning based regularization. After planting the backdoor trigger to a set of samples, we do not mislabel all of them to the target class. Instead, we randomly keep a fraction of them (namely regularization samples) still correctly labeled to their real semantic classes. Intuitively, these additional regularization sam- ples penalize the backdoor correlation between the trigger and the target class. (2) Trigger planting strategies that promote asymmetry and diversity. One may notice that penalization on the backdoor correlation induced by regularization samples can also greatly hurt the attack success rate (ASR). We alleviate this problem via asymmetric trigger planting strategies. As illustrated in Fig 2 we apply weakened triggers when we construct regularization and payload samples for data poisoning, while the original standard trigger would only be used during test time to activate the backdoor. Con- ceptually, in this way, since test-time backdoor samples (with the standard trigger) contain stronger backdoor features than those of regularization samples (with weakened triggers), the test-time attack can well mitigate the counter force from regularization samples and still maintain a high ASR. Be- sides asymmetry, our design also promotes diversity of triggers during data poisoning poison samples could be stamped with different partial triggers, selected from a diverse set of trigger partitions. Intuitively, this diversity allows backdoor poison samples to scatter more diversely in the latent representation space, and can thus avoid being aggregated into an -asy-to-- cluster.
different
In conclusion, the main contributions of this paper are four-fold. (1) We confirm that the latent separability assumption holds across diverse set of backdoor poisoning attacks in the existing lit- erature. (2) We reveal that this assumption could fail, leading to poor performance of defenses that explicitly base their designs on it. (3) We design some simple yet effective adaptive backdoor poi-
'The first two authors contributed equally to this paper. Correspondence to: Xiangyu Qi, Tinghao Xie, Yiming Li, and Prateck Mittal.
2

---

Published as a conference paper at ICLR 2023
Attacker's
Payload
Backdoored Model
Test-time Backkdoor Samples with The Standard Trigger
Samples
(Misjlabeled Target Class
Samples:
Induce High ASR
Random
Correctly with High Acouracy
Clean Tast
Plant Weakened
Spiit
Regularizatin
Samples
Diversified Triagers
Samples
Denpp Lounming
Random Partition
Trigger-planted Samples
Correctlly Labeled
Separate Clean and Poison
PiC
Standard
Trigger
Trigger Planting Strutegies That
Clean
Cluster Analysis
Latent Spare
Promote Asymmetry & Diversity
Samples
Polsoned Dataset
An overview of adaptive backdoor poisoning attacks (here we take Adaptive-Blend introduced in Section 5.2 as an example for illustration). Two key components render our attacks adaptive: (1) Poisoning-based regularization, which penalizes the backdoor correlation and helps to suppress the latent separation; (2) Trigger planting strategies that promote asymmetry and diversity, which help to maintain high attack success rate as well as to improve latent space stealthiness. Please refer to our Section 5.1 for more technical details.
Figure 2:
soning attacks to present pounter-examples against this assumption with two key novel components. (4) We conduct extensive experiments on benchmark datasets, verifying the effectiveness and the stealthiness in countering detection methods of our adaptive attacks.
RELATED WORK
Backdoor Poisoning Attacks. Backdoor poisoning attacks (Gu et al. 2017 Chen et al. 2017;
Tumner et al. 2019 Li et al. (021d) are also frequently referred to poison-only backdoor attacks. This category of attacks only assume control over a small portion of the victim's training data, while the vietim will train her own models on the poisoned dataset from scratch. Other backdoor attacks
that assume additional control over the training process (Shokri et al. 2020) or even weights of deployed models (Liu et al. 2017a Qi et al. |021 2022a) do not fall in this category and are not considered in this work. We refer readers to Li et al. (2022 for a more comprehensive review.
Latent Separation for Backdoor Defenses. It has been commonly observed (Tran et al., 2018) that models trained on a poisoned dataset tend to learn very different latent representations for back- door and clean samples in the target class, which form two separate clusters (see Fig This phenomenons is so pervasive that a family of defenses directly take the latent separation as a default assumption and propose to identify poison samples via performing cluster analysis the latent space. This family includes Spectral Signature (Tran et al.) 2018) and Activation Clustering (Chen) al. 2019), which are most commonly evaluated baselines. More recent proposals (Tang et al.) 2021 Hayase et al., [021] in this family further claim to achieve nearly perfect recall with negligi- ble talse positive rate against& a diverse of attacks even in very low poison rate cases.
Adaptive Backdoor Attacks Against Latent Separation Based Defenses. A family of adaptive backdoor attacks (Tan & Shokri, 2020 Xia et al., 2022 Doan al. 2 Ren et al.) |021 Cheng jet al. 202T; Zhong et al.] (022) explicitly aim to reduce the latent separation between poison and clean samples. However, they do not fit into the paradigm of backdoor poisoning attacks they assume additional control over the whole training process and thus directly encode the latent insep- arability into the training objectives of attacked models. A more relevant work is Tang et al. (221)) which points out that their source-specific poison-only attack can reduce latent separation. However, as shown in Fig Ic, when the base model trained along with standard data augmentation, there still a notable separation between the clean and poison populations, and actually Tang et al. (2021) themselves also show that an improved latent space cluster analysis suffices to perfectly separate poison and clean samples of this attack. Thus, it is still unclear whether a poison-only backdoor attack can overcome the latent separation evade backdoor defenses built on it. In this work, we fill the gap and design adaptive backdoor poisoning attacks that can actively suppress the latent separation (and thus circumvent existing latent separation based defenses).
Other Backdoor Defenses. There are other defenses that are not built on latent separation. These include trigger synthesis (Wang al. 2019 Guo al.) 2022). model diagnosis (Xu et al. 2021 Kolouri et al. 2020). sample diagnosis (Gao et al. 2019 Guo et al.) 2023), finettuning (Liu et al. 2017b Li et 2021b), poison suppression (Li al. 202 Huang et al. (022) and proactive training (Qi et al. 2022b), etc. Many of these proposals also have their own limitations revealed by
existing literature (refer] Li et al. (2022) ) but they are not our focus in this work.
3

---

Published as a conference paper at ICLR 2023
3 NOTATIONS AND THREAT MODEL
Notations. We study image classification with DNN models. We denote model by Fo X' [C], where 0 are trainable parameters, X is the input space, C' is the number of classes, and [C] := (1.2...., C}. We decompose Fo as Fe = lg o So. where la is the last linear prediction layer that transforms a latent representation into the final prediction label, and fo is the feature extractor. Given input X, fo(x)) E H. is the latent representation of x w.r.t .t model Fe. H denotes the latent representation space, and (x) = Lo a fo()) is the predicted label. For backdoor poisoning attacks, we denote the clean training set by D {( y:) i= 1, ,....,,..,, .,).. We denote the backdoor trigger planting strategy by a transformation T : X' + X, and the adversary's 's poison label flipping strategy is denoted by L: X' [C] + [C]. We use J == (ii,..., to denote indices of the p
+
data points that are controlled by the adversary. The resulting poisoned training set is denoted as D Ppassnn
{ 01) ....., n1.. where
(Tx), i E
[Li, E
x =
ú. =
(1)
T2+
otherwise
Vb:
otherwise
Threat Model. We consider the standard threat model of backdoor poisoning attacks (poison-
only backdoor attacks), where the adversary only control small portion of the victim's training data and the vietim will train her own models from scratch on the poisoned dataset manipulated by the adversary. Specifically, the adversary will design a trigger planting strategy T and label flipping strategy L to manipulate the controlled training samples (as formulated in Eqn D. A victim model trained on the poisoned dataset D Doossn will be backdoored that is, during test time, the model will (mis)classify a trigger- paagerr-patteddddee input to target class t with high probability, while keeping approximately the same performance to that of a benign model on genuine inputs.
4 PROBLEM FORMULATION: TOWARDS POISON-ONLY BACKDOOR ATTACKS THAT CAN ACTIVELY SUPPRESS THE LATENT SEPARATION
Latent Separability Assumption for Backdoor Defense. Given a poisoned dataset Ppoisons one can train a backdoored model Fg ;= lg o fe via running a standard empirical risk minimization procedure on 2 poisamn i.e., 0 p.pppis Latent separability assumption indicates that, in the latent representation space generated by the backdoored model Fo, poison and clean samples from the target class t will form separate clusters, while samples from a non-target class only form a single homogeneous cluster (see Fig D.. Latent separation based backdoor defenses (Tran et al.) 2018 Chen et al. 2019 Hayase et al. 2021 Tang et al.) 2021) propose to run cluster analysis on H [Z+ 81 cf for each class Typically, the defender will design heterogeneous criterion I(.) that takes H as input and judges whether this set is heterogeneous (i.e., contains separate clusters). On the heterogeneous H' identified by the criterion I, the cluster analysis will divide H' into two empirical clusters H3 and H4, where H1, is the suspected cluster formed by poison samples. The dataset will be cleansed by simply removing those training samples that generate H-
Our Goals. This work revisits the assumption of latent separability for backdoor defenses against poison-only backdoor attacks. We investigate adaptive backdoor poisoning attacks that can actively suppress the latent separation between poison and clean samples. Ideally, against such adaptive attacks, the criterion I used by a defense should fail to detect the heterogeneity in //f and the cluster analysis would neither accurately separate poison and clean samples.
Perspectives that Motivate Our Design. Two heuristic and mutually complementary perspectives on the latent separation phenomenon have inspired our design in this work. The first perspective at- tributes the latent separation to the dominant impact of backdoor triggers (Tran et al. (2018) during the inference of backdoored models. The intuition is - in order to "push" a (trigger-planted) back- door poison sample from its semantic class the target class, a backdoored model tends learn an excessively strong signal for the backdoor trigger pattern in latent representation space such that the signal can overwhelmingly beat other semantic features to make its dictatorial decision. The strong backdoor signal that exclusively appears in backdoor poison samples thus leads to the sepa- ration. The second perspective is that, backdoored models learn separate representations for poison and clean samples simply because they tend to learn a separate shortcut rule (Geirhos et al. 2020) (solely based on the trigger pattern) to fit those poison samples without using any semantic fea- tures. The sense is - backdoor learning is often independent of (or only weakly correlated to) the semantic features used by the main task, thus the backdoored model that fits the poisoned dataset essentially just learns two unrelated (or weakly related) tasks. From this aspect, there is not even an
4

---

Published as a conference paper at ICLR 2023
appealing reason for backdoor models to learn homogeneous latent representations for samples from the two heterogeneous tasks. Motivated by these perspectives, we conceive that a desirable adaptive backdoor poisoning attack (that can mitigate the latent separation) might need to encode some form of regularization, so as to (1) penalize the backdoored model for learning abnormally strong signals for the backdoor trigger; (2) encourage interconnection between backdoor learning and learning of the main task. These intuitions finally lead to our design in Section 5
5
OUR METHODS
We design adaptive backdoor poisoning attacks following the insights we introduce in Section 4 In we first present the generic framework underlying the design of our attacks. Then, in claborate concrete attacks that we implement in this work.
Section 5.
Section 5.2 2
5.1 A GENERIC FRAMEWORK FOR ADAPTIVE BACKDOOR POISONING ATTACKS
Overview. We present an overview of our design in Fig 2 Unlike typical backdoor poisoning attacks, in our framework, we do not label all trgger-plaaned samples to the target class. As shown, after planting the backdoor trigger them into two disjoint groups. For one group, we still label them to the target class (we call this group payload samples) to establish the backdoor correlation between the trigger pattern and the target label; while the other group (namely regularization samples) will instead be correctly labeled to their real semantic classes (that can be diffident from the target class) to regularize the backdoor correlation. Formally, following our notations in Section 3 the adversary will specify a conservatism ratio 1) E [0, 1), with which our label flipping strategy formulates as:
set of samples (sampled from all classes), we randomly split
C(z,, v1) = (1, with probability 1 -1
{ y/i with probability 1
Moreover, we introduce ideas of asymmetry and diversity into our trigger design we apply diverse set of weakened triggers to construct regularization and payload samples for data poisoning, while the original standard trigger is used during test time to activate the backdoor.
Regularization Samples. We note that, the introduction of regularization samples well incorpo- rates our two insights from Section f First, with regularization samples, the backdoored model can no longer learn a dominantly strong signal for the backdoor trigger that dictatorially votes for the target class, otherwise, it can not fit regularization samples that are correctly labeled to other classes. This explains the naming of regularization samples intuitively, they serve as regularizers that help to penalize the backdoor signal in the learned latent representations. Second, the model can neither fit all trigger- planted samples via a simple shorteut rule. Instead, now has to fit a much more complicated boundary that should decide when to classify a trigger-planted input to the target class and when to classify it to its real semantic label, where the boundary is randomly generated. To successfully fit this boundary, the model must rely on both the trigger pattern and artifacts from the semantic features that coexist with the trigger, thus the Icarned latent representations for backdoor samples should be more balanced fusion of both the trigger pattern and semantic features.
Asymmetric Triggers. The introduction of asymmetric triggers is critical for our attacks to still maintain a high attack success rate (ASR). As one may easily notice, since regularization samples penalize the backdoor correlation, side-effect could be the drop of attack success rate (ASR). To mitigate this problem, rather than using the same trigger for both data poisoning and test-time attack, in our design, we apply weakened triggers for data poisoning and use the (stronger) original standard trigger only for the test time. The intuition is: During test time, the backdoor samples (with the standard trigger) contain stronger backdoor features than those of regularization samples (with weakened triggers). This then enables test-time backdoor samples to have sufficient "power" to mitigate the counter force from regularization samples and thus still achieve a high ASR. We note that the idea of asymmetric triggers traces earliest back to Chen et al. (2017). however the context is different. In order to evade human inspection on the poisoned dataset, et al. (2017) propose to use weakened triggers that are visually less evident for data poisoning, and point out that a high ASR can still be maintained if the original standard trigger is used in test time. In our context, we use weakened triggers mainly to undermine the negative impact induced by regularization samples.
Trigger Diversification. We also highlight that the trigger diversification in our design can also help our attacks to mitigate the latent separation. Intuitively, since different poison samples could be planted with different triggers, these poison samples may scatter more diversely in the latent
5

---

Published as a conference paper at ICLR 2023
representation space. We expect such more diverse scattering can prevent these poison samples from aggregating into an easy-to-identify cluster.
5.2 INSTANTIATIONS OF OUR ATTACKS
Note that, our framework presented in Fig 2 is generic and can be creatively combined with existing techniques instantiate powerful adaptive attacks. Following this framework, instantiate two concrete attacks via directly adapting commonly used image blending based and patch based poison strategies, namely Adaptive-Blas Blenn and Adaptive-Part respectively.
Adaptive-Blend. An interesting point revealed by Fig that the simple Blend attack (Chen et al.] (017) turns out to induce the least latent separation, better than many attacks that usually deemed more advanced and stealthy, This suggests image blending based triggers as good candidates for design- ing attacks with weak latent separation. For this reason, we design Adaptive-Bles attack according to framework. Specifically, Adaptive- Blend introduces a conservatism ratio of 0.5 to balance the number of payload and regularization samples, and still adopts the process ō = (1 - a) I + a ' from Chen et al. 2017) to blend the trigger pattern T into genuine image x' to construct the trigger-plantie sign, we still take the standard a 0.2 for test-time attacking, but use weaker asymmetric opacity of = 0.15 for poison samples. Moreover, for stronger and more diverse asymme- try, we propose to partition the standard full trigger (Fig 3 left) into 4 x 4 = 16 pieces - the full trigger would still be used for test-time attacking, while randomly apply only 50% of the partitioned trigger pieces (e.g., Fig right) to each poison sample during data poisoning. This additional partition boosts both the ASR and the latent space stealthiness (see Section 6.3.3).
via directly adapting the naive Blend
Figure 3: In Adaptive-Blen Blend, we partition the full trigger image into 4 x 4 = 16 pieces (left), and ran- domly apply only 50% of these trigger pieces (right) to each poi- son sample, during data poisoning- Red lines demonstrate the grids by which we randomly mask the orig- inal trigger.
sample ã. As for asymmetric de-
Adaptive-Patch. Although the triggers could be more visually detectable, for comprehensiveness, we also instantiate our adaptive attack with patch based triggers, namely Adaptive-Patch. Empir- ically, since patch based triggers usually induce stronger latent separation (e.g., see Fig [b). we correspondingly turn to a larger conservatism ratio of 1 2/3. For trigger planting, rather than sticking to a single patch pattern, we prepare a more diverse set of 4 patch triggers (Fig 6f for data poisoning. Specifically, each poison sample is randomly attached to only one of the four trig- gers with a low opacity (e.g., 50%) (Fig 6m). At test time, we asymmetrically apply two (of the four) fully opaque triggers (e.g., Fig 6g sinultaneously to achieve high ASR.
EXPERIMENTS
6.1 MAIN SETTINGS
Datasets and Model Architectures. evaluate our adaptive attacks on three benchmark datasets that are commonly used in backdoor learning literature: CIFAR-10 (Krizhessky) 2012). GT-
SRB (Stallkamp et al., 2012) and 10-classes subset of Imagenet (Russakovsky et al. 2015). ). For building base models, we also consider three different architectures including ResNet-20 (He et al. 2016). VGG-16 (Simonyan Zisserman, 2014) and Mobilenet- V2 (Sandler al. 2018). Due the space limit, in this section, we only present our results on CIFAR-10 with ResNet-20 20. refer inter- ested readers to Appendix B for results on other datasets and architectures. Detailed configurations on dataset split and training details of base models are deferred to Appendix
Attacks. We evaluate our Adap-Blend and Adap-Patch attacks presented in Section 5.2
We compare our adaptive attacks with six representative attacks in the literature. These attacks correspond to diverse set of poisoning strategies including both classical and advanced ones. BadNet (Gu et al. 2017) and Blend (Chen et al. 2017) correspond to typical dirty-label attacks with patch-like triggers and blending based triggers respectively. Dynamic (Nguyen & Tran 2020
and ISSBA (Li et al. 2021d) correspond to input-aware backdoor attacks. CL (Turner et al. 2019)
clean label attack. TaCT (Tang et al. (022) is source-specifie attack. Unless explicitly speci- fied, for every attack, by default, we use 150 (payload) poison samples for data poisoning. Detailed attack configurations are described in Appendix A.2

---

Published as a conference paper at ICLR 2023
Table I: Latent separability based defenses against our adaptive attacks on CIFAR-10.
No Poison
Blend
BadNet
ISSBA
Dynamic
CL
TaCT
Adap-BBlend (Ours)
Adap-Patch (Ours)
Without Delonse
Clean Accuracy
ASR
920
91.7
89.0
99.9
91.5
91.6
95.3
91.8
97.5
%2.1
90.6
91.8
96.5
91.6
76.5
91.5
97.5
Spectral
Eliminatoon Rate
53.8
98.0
63.5
87.8
94.4
62.9
13.3
10,0
Signtture
[0018]
ASR
Clean Accuracy
Sacrifice Rate
90.9
150
91.5
58.6
4,4
91.4
4.2
1.3
91.5
4.3
1.1
86.1
72.4
4.3
91.7
40.8
4.2
91.5
96-4
4.3
91.5
62.0
4.5
91.5
93.1
4.5
Activation
Clostering
Eliminzioo Rate
Sacrifice Rate
0.0
6.0
100,0
30.6
33.5
33,1
ASR
Accuracy
91.7
91.4
91.6
91.5
92.1
91.6
91.6
Eliminztion
62.9
100,3
SCAn Lang 2021
Sacrifice Rate
0.0
3.5
0.0
4.9
1.2
ASR
87.8
1,0
0.9
16.3
32.9
0.5
76.2
97.5
Clean Accuracy
92.0
91.7
91.1
91.6
91.7
91.8
90.8
91.6
91.5
Elimination Rate
96.4
100,0
100.0
99.8
100,0
100.0
6.9
000
SPECTRR Hayase et 20AT
Sacrifice Rate
1.5
0.2
0.2
0.2
0.2
0.2
0.2
0.5
0.5
ASR
5.7
0288
1.0
7.7
1.6
1.7
69.0
94.8
Clean Accuncy
91.6
91.7
91.7
91.6
91.6
91.6
91.6
91.4
91.5
Silhouette Score
0.2608
0.4744
0.3933
0,4358
0.3964
0.2866
0,1065
0.0856
Defenses. To validate the "adaptiveness" of our attacks against latent separation based backdoor defenses, we evaluate the four state-of-the-art defenses from this family: Spectral Signature (Tran) et al. 2018), Activation Clustering (Chen al.) 2019). SCAn (Tang et al. 2021) and SPEC- TRE Hayase et al. 2021). All of these defenses are designed to detect and eliminate backdoor poison samples from the poisoned dataset, based on the assumed latent separation characteristics.
Metrics. For backdoor defenses that we evaluate, we measure their: 1) Elimination Rate, ratio of (payload) poison samples that they successfully detect; 2) Sacrifice Rate, ratio of clean samples falsely eliminated; 3) Attack Success Rate (ASR) of models retrained on the cleansed set; 4) Clean Accuracy of models retrained on the cleansed set. Note that, ASR is defined as the ratio of trigger- planted samples that are mispredicted to the target class, while clean accuracy the accuracy on genuine test samples. Moreover, to quantify the latent separation between clean and poison samples, we report the Silhouette Score (Rousseeuw) 1987) of latent representations in the target class. A silhouette score is in the range from ( to T. A lower silhouette score indicates weaker separation. All the numbers that we report are average results across three independent repeated experiments.
6.2 MAIN RESULTS
Visualization. Fig] 1 plots latent representations of poison and clean samples for different attacks, visualized by T-SNE (Van der Maaten & Hinton 2008). As shown, notable latent separations are consistently observed on all the baseline attacks that we consider, while the poison and clean samples of our attacks mix with each other (Fig Th li). To further reveal the extent of latent inseparability of our adaptive attacks, we also use Support Vector Machine (SVM (Cortes Vapnik 1995)) to find the (linear) boundary that best separates poison and clean samples in the latent representation space. Fig 4 visualizes the histogram of distances between each data point and the SVM hyperplane. As shown, compared to non-adaptive attacks (Fig 4aland our adaptive attacks (Fig 4b and 4d bring the poison and clean samples much closer.
Against Latent Separation Based Defenses. We present our main results in Table T As shown, against SPECTRE (Hayase al. 2021) defense, the strongest latent separation based defense in the literature, none of the six baseline attacks survive - SPECTRE can always eliminate almost all poison samples with negligible sacrifice of clean samples. This is consistent with our (qualitative) visualization in Fig 1 where notable latent separations are observed for all these attacks. It is also consistent with our quantitative measure of the latent separation - all the six baseline attacks induce high Silhouette scores (> 0.25). In comparison, our two adaptive attacks exhibit evident stcalthiness in the latent representation space both the visualization results (Fig [i and Silhouette scores indicate much weaker latent separation, and all the four latent separation based defenses are consis- tently defeated (ASR is still larger than 20% after defense). Besides, our adaptive attacks always achieve high ASR with negligible clean accuracy drop in all the cases. When no defense is applied, both Adap-Blend (>75%) and Adap-Pate achieve high ASR (>95%). While none of the other six baseline attacks makes thorough all these defenses after cleansing and retraining, both our Adap- Blend and Adap-Patch consistently retain considerable ASR (Adap-Blec Blend > 60% and Adap- -atph 90%), surviving each of them. We point out that, our results serve counter examples against
7

---

Published as a conference paper at ICLR 2023
(a) Blend
(b) Adap-Blend
(c) BadNet
(d) Adap-Patch
Figure 4: Visualization of latent representation spaces fitted by SVM. We use SVM to find the opti- mal (linear) boundary that separates poison and clean samples, and plot the histograms of (signed) distances between each point and the SVM hyperplane.
A01
ASR
25.21
25
60
SCAn
SCAn
SChn
SCAs Sox
40
40
20
20
50
100
150
200
250
500
1000
50
100
150
200
250
500
1000
Paisan
Paison
(a) Adap-daBBlonn Blend
(b) Adap-Patch
Figure 5: Defense results w.r.t. different (payload) poison samples. For Adap-Blend, we as
many regularization samples as payload samples; for Adap-Patch, we use twice the regularization sample number payload samples. The black dotted lines show the ASR. We use different colors to represent the results different defenses, where the solid lines correspond to Elimination ("Eli") and the dotted lines correspond to Sacrifice ("Sac").
Table 2: Adap-Blan Blend with different regularization sample numbers, with fixed 150 payload samples.
Regularization Samples
150
ASR
89.0
86.5
83.9
76.5
74.1
70.4
63.6
60.9
58.0
56.5
Elimination
45.3
0.0
SPECTRE
Sacrifice Rate
0.2
0.3
0.3
0.5
0.5
0.5
0.5
0.5
0.5
0.5
ASR
9.1
29.2
37.9
69.0
78.2
68.5
57.9
68.7
64.0
62.8
the assumption of latent separability and the "adaptiveness" of our attacks are also validated. We also supplement results on other datasets and model architectures (see Appendix B)
6.3
ABLATION STUDIES
6.3.1 POISON RATE
As mentioned Section our main experiments consistently use 150 (payload) poison samples for poisoning attack. In Fig we supplement additional results of our adaptive attacks with different number of poison samples. Specifically, we increase the number of payload poison samples from 50 to 1,000, and the number of regularization samples also proportionally vary according to the fixed conservatism ratio y) that we specify in Section 5.2 As shown, one key takeaway is when the poison sample number grows too large (e.g., 1,000), the stealthiness of our adaptive attacks start to significantly degrade. This is not surprising rigid trigger pattern, the trigger pattern would become increasingly statistical significant, and models will unavoidably learn strong signal for this pattern in spite of the regularization. This indicates that moderate poison rate is also necessary condition for the success of adaptive attacks.
with more and more poison samples containing the
6.3.2 STRENGTH OF REGULARIZATIN
Now, fix the number of payload poison samples (150 samples), and investigate varying number of regularization samples - this reflects different strength regularization. Specifically, we evaluate Adap-Blem Blend against SPECTRE, and present the results in Tab We can generally tell that: 1) when
8

---

Published as a conference paper at ICLR 2023
Table 3: Ablation study see every part of our adaptive strategy is necessary.
(a) Blending attack with lower poison rates.
(b) Adap-Blend with partial components.
(%)
ASR
SPECTRE
SPECTRE
Poison Samples
Elimination
Retrained ASR
Elimination
Retrained ASR
50
69.7
84.0
8.7
No Diversity Asymmetry
52.1
28.0
32.5
100
8144
97.0
5,3
No Regulatization Samples
89.0
67.3
9.1
5.7
With Both
76.5
69.0
the regularization is weak (e.g., O, 50, 100 regularization samples), our adaptive attacks could still be detected; 2) when the regularization is becoming stronger, our adaptive attacks start to mitigate the defense, though the ASR suffers from more sacrifice.
6.3.3 IS EVERY PART OF OUR ADAPTIVE STRATEGY NECESSARY?
Simply reducing poison rate is not enough. In Section 6.3.1 we reveal that a low poison rate is necessary for the success of our attacks. Nonetheless, as shown in Tab when we lower the poison rate (to as few 50 poison samples) of blending attack, will still be cleansed by SPECTRE. Thus, simply reducing poison rate is not sufficient for mitigating the latent separation.
Simply relying on regularization samples is not enough. Regularization samples are important our design, is the trigger planting strategy we adopted (See Section [5. for discussion). If we use the standard symmetric trigger for both the data poisoning and test-time attack, both the ASR and latent space stealthiness would degrade- - the first row of Tab 33b shows Adap-Blend without asymmetric trigger partitioning, where it has a lower ASR (= 50%) and could be further suppressed (retrained ASR = 30%) by SPECTRE.
Simply relying on asymmetric and diversified triggers is not enough. Reversely, we study how our adaptive strategy behaves when we don't t use regularization sample and rely solely on our trigger planting strategy. As shown in the second row of Tab 3b) though the original ASR gets higher, as a trade off. much larger fraction (> 67%) of the poison samples can now be recognized and re- moved by SPECTRE and the retrained ASR drops severely. This further confirms that regularization samples are vital for our attacks. See Fig 9 for further w.r.l. the necessity of regularization.
7 DISCUSSIONS
We expect an ideal adaptive attack can make the poison and clean samples completely indistinguish- able in the latent space. This has been achieved under stronger threat model where the the training process is controlled (Shokri et al.; 2020; Xia et al. 2022 Doan et al. 2021 Ren al., 2021 Cheng et al. 2021 Zhong et al. 2002). In this paper, we take a step to this goal under poisoning-one only threat model. We propose adaptive backdoor poisoning attacks that can suppress the latent separation and circumvent existing defenses based on it. However, as shown in Fig 4 under oracle visualization, there is still a difference between poison and clean distributions, though the difference is greatly reduced. A remaining question is - is it possible to achieve the ideal indistinguishable goals under the poison-only setting? We encourage future work look into this question.
CONCLUSION
In this work, we revisited the assumption of latent separability for backdoor defenses. We revealed that this assumption could fail, leading to the failure of backdoor defenses built on this assumption. Specifically, we provided our insights on the phenomenon of latent separation, and designed adaptive attacks that can mitigate this separation. Empirical study and evaluation of various latent separation- based defenses showed that our adaptive poisoning attacks indeed suppress the latent separation and render them ineffective. We call for defense designers to take caution when leveraging latent separability as an assumption in their defenses. We also encourage further defenses to take our attacks into consideration for more comprehensive evaluation.

---

Published as a conference paper at ICLR 2023
ETHICS STATEMENT
During our study, we restricted all of our adversarial experiments within the laboratory environment, and did not induce any negative impact in the real world. Though our attacks may be mitigated by many existing defenses, the illustration of our attacks is only conceptual. We encourage future work to design stronger defenses that resist our attacks.
ACKNOWLEDGEMENTS
This work was supported in part by the National Science Foundation under grants CNS-1553437 and CNS-1704105, the ARL's Army Artificial Intelligence Innovation Institute (A212), the Office of Naval Research Young Investigator Award, the Army Research Office Young Investigator Prize, Schmidt DataX award, Princeton E-ffiliates Award, and Princeton's 's Gordon Y. Wu Fellowship. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the funding agencies.
REFERENCES
Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in cnns by training set corruption without label poisoning. ICIP. pp. 101-105, 2019.
Eitan Borgnia, Valeriia Cherepanova, Liam Fowl, Amin Ghiasi, Jonas Geiping, Micah Goldblum, Tom Goldstein, and Arjun Gupta. Strong data augmentation sanitizes poisoning and backdoor attacks without an accuracy tradeoff. In ICASSP, pp. (855-3859, 2021a.
Eitan Borgnia, Jonas Geiping, Valeriia Cherepanova, Liam Fowl, Arjun Gupta, Amin Ghiasi, Furong Huang. Micah Goldblum, and Tom Goldstein. Dp-instahide: Provably defusing poisoning and backdoor attacks with differentially private data augmentations. In ICLR Workshop. 202
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering, In AAAI Workshop, 2019,
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526, 2017.
Siyuan Cheng, Yingqi Liu, Shiqing Ma, and Xiangyu Zhang. Deep feature space trojan attack of neural networks by controlled detoxification. In AAAI, 2021.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20(3):273-297, 1995.
Khoa Doan, Yingjie Lao, and Ping Li. Backdoor attack with imperceptible input and latent modifi- cation. In NeurlPS, 2021.
Fastai. Fastai/imagenette:: A smaller subset of 10 easily classified classes from imagenet, and a little more french, 2019. URL
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. ACSAC, 2019.
Robert Geirhos, Jömn-Henrik Jacobsen, Claudio Michaelis, Richard Zemel, Wieland Brendel, Matthias Bethge, and Felix A Wichmann. Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(1 11:65-66777 2020.
Tianyu Gu, Brendan Davan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv: 2017.
Junfeng Guo, Ang Li, and Cong Liu. AEVA: Black-box backdoor detection using adversarial ex- treme value analysis. In ICLR, 2022.
10

---

Published as a conference paper at ICLR 2023
Junfeng Guo, Yiming Li, Xun Chen, Hanqing Guo, Lichao Sun, and Cong Liu. Scale-up: An efficient black-bor box input-level backdoor detection via analyzing scaled prediction consistency. In ICLR, 2023.
Jonathan Hayase, Weihao Kong, Raghav Somani, and Sewoong Oh. Spectre: defending against backdoor attacks using robust statistics. In ICML, pp. 4129-413333 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, pp. 770-778, 2016.
Kunzhe Huang, Yiming Li, Baoyuan Wu, Zhan Qin, and Kui Ren. Backdoor defense via decoupling the training process. In ICLR, 2022.
Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash, and Heiko Hoffmann. Universal litmus patterns: Revealing backdoor attacks in cnns. In CVPR, pp. 301-310, 2020.
Alex Krizhevsky. Learning multiple layers of features from tiny images. University of Toronto, 05 2012.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Anti-backdoor learn- ing: Training clean models on poisoned data. In NeurlPS, 2021a.
Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In ICLR, 2021b.
Yiming Li, Tongqing Zhai, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor attack in the
physical world. In ICLR Workshop, 2021c
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey. IEEE Transac- tions on Neural Networks and Learning Systems, 2022.
Yuezun Li, Yiming Li, Baoyuan Wu, Longkang Li, Ran He, and Siwei Lyu. Invisible backdoor attack with sample-specific triggers. In ICCV, pp. 16463-16472, 2021d.
Kang Liu, Brendan Dolan-Gavite and Siddharth Garg. Fine-pruning: Defending against backdoor- ing attacks on deep neural networks. In RAID, pp. 273-294, 2018.
Yannan Liu, Lingxiao Wei, Bo Luo, and Qiang Xu. Fault injection attack deep neural network. In ICCAD, pp. 131-138, 2017a.
Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. In ECCV, pp. 182-199, 2020.
Yuntao Liu, Yang Xie, and Ankur Srivastava. Neural trojans. In ICCD, pp. 45 48, 2017b.
Anh Nguyen and Anh Tran. impee-impereeptibbe warping-based backdoor attack. In ICLR, 2021.
Tuan Anh Nguyen and Anh Tran, Input-aware dynamic backdoor attack. In NeurlPS, pp. 3454- 3464, 2020.
Minlong Peng, Zidi Xiong, Mingming Sun, and Ping Li. Label-smoothed backdoor attack. arXiv preprint arXiv:2202.1 /203, 2022.
Xiangyu Qi. Jifeng Zhu, Chulin Xie, and Yong Yang. Subnet replacement: Deployment-stage back- door attack against deep neural networks in gray-box setting. arXiv preprint a7X.v2200..07240, 2021.
Xiangyu Qi. Tinghao Xie, Ruizhe Pan, Jifeng Zhu, Yong Yang, and Kai Bu. Towards practical deployment-stage backdoor attack on deep neural networks. In CVPR, pp. 13347-13357, 2022a.
Xiangyu Qi, Tinghao Xie, Jiachen T. Wang, Tong Wu, Saeed Mahloujifar, and Prateck Mit- tal. Towards a proactive ml approach for detecting backdoor poison samples, 2022b. URL https://arre otg/abs /20/22205.22 13616

---

Published as a conference paper at ICLR 2023
Yankun Ren, Longfei Li, and Jun Zhou. Simtrojan: Stealthy backdoor attack. In ICIP, pp. 819- 119-823, 2021.
Peter Rousseeuw. Silhouettes: a graphical aid to the interpretation and validation of cluster analy- sis. Journal of computational and applied mathematics, 20:53-65, 1987.
Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision, 115())211-22) 2015.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chia Chich Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. In CVPR, pp. 4510-4520, 2018.
Lujia Shen, Shouling Ji, Xuhong Zhang, Jinfeng Li, Jing Chen, Jie Shi, Chengfang Fang, Jianwei Yin, and Ting Wang. Backdoor pre-trained models can transfer to all. In CCS, 2021.
Reza Shokri et al. Bypassing backdoor detection algorithms in deep learning. In Euro&&P, pp- 175-183, 2020.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409. 1556, 2014.
Johannes Stallkamp, Marc Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Bench- marking machine learning algorithms for traffic sign recognition. Neural networks, 32:323-332, 2012.
Te Juin Lester Tan and Reza Shokri. Bypassing backdoor detection algorithms in deep learning. In Eu 2022..
Di Tang, XiaoFeng Wang, Haixu Tang, and Kehuan Zhang. Demon in the variant: Statistical analysis of duns for robust backdoor contamination detection. USENIX Security, 2021.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. arXiv preprini 181 1.06366 2018
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. 2019.
Label-consistent backdoor attacks.
arXiv preprint
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), 2008
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In IEEE S&P, pp. 707-723, 2019.
Pengfei Xia, Hongjin Niu, Ziqiang Li, and Bin Li. Enhancing backdoor attacks with multi-level mmd regularization. IEEE Transactions on Dependable and Secure Computing. 2022.
Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. DBA: Distributed backdoor attacks against federated learning. In ICLR, 2019.
Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. Detecting trojans using meta neural analysis. In IEEE S&P, 2021.
Nan Zhong, Zhenxing Qian, and Xinpeng Zhang. Imperceptible backdoor attack: From input space to feature representation. In IJCAI, 2022.
12

---

Published as a conference paper at ICLR 2023
(b)
(d)
(h) Adap-Blend
(i) Adap-Blend
(i) 1a Trigger
(k) ndd Trigger
(1) 3 Trigger
(m) 4th Trigger
(n)
Adap-K
(Test) Blend
(Test)
Figure 6: The example of poisoned samples.
A EXPERIMENT CONFIGURATIONS
A.1 COMPUTATIONAL ENVIRONMENTS
All of our experiments are conducted on a workstation with Intel Xeon Silver 4214 1 CPU cores, 384 GB RAM, and GeForce RTX 2080 Ti GPUs.
A.2 ATTACK CONFIGURATIONS
For each attack, we reuse the same triggers adopted in the paper. "Blend" and "Adap-Blend" use the blending-trigger (Fig 6a) with 20% opacity. "TaCT" uses the trojan square trigger (Figóm) with 100% opacity. Eventually, "Adap-Patch" use the four triggers and 30% opacities, respectively. Specifically, each of their poison training sample randomly selects one of the four triggers (see Fig 6j 6m), while each test sample uses the enhanced two triggers demonstrated in Fig 6n
Fig [6 with 50%, 20%, 50%
The target class is set to class 0. 150 poison samples are used for "Blend", "BadNet", "ISSBA", "Dynamic" and "CL". For "TaCT", Blend", we use 150 payload samples and 150 regularization samples. For our "Adap-Pater Patch", we use 150 payload samples and 300 regularization samples.
150 poison samples and 150 cover samples. For "Adap-
For all backdoor models, we adopt the standard training pipeline. SGD with a momentum of 0.9, a weight decay of 10-4, and batch size of 128, is used for optimization. Initially, set the learning rate to 0.1. On CIFAR-10, we follow the standard 200 epochs stochastic gradient descent procedure, and the learning rate will be multiplied by factor of 0.1 the epochs of 100 and 150. On GTSRB we use 100 epochs of training, and the learning rate is multiplied by 0.1 at the epochs of 40 and 80.
We consider widely adopted data augmentations, i.c. RandomlorizontallLL. RandomCrop for CiFAR- 10 and RandomRotat ion for GTSRB. We notice that data augmenta- tion may affect defense results significantly, while defenders do not know whether to use augmen- tation or not. Therefore, we report the better defense result of the two backdoor models with and without augmentation. In another sentence, we report upper-bound results the defenses which might be affected by the incorporation of data augmentation during backdoor training. Also, to rule out the effect randomness, we train three models on three seeds for each configuration and report their average results.
and
A.2.1 FAIRNESS CONSIDERATIONS
Trigger Selections. For other non-adaptive attacks, we follow the trigger selections adopted in their original works in order to objectively evaluate how they perform against the defenses. For our attack, we argue that our trigger design itself vital part of our design our triggers should not necessarily be the same to triggers of other attacks. Still, when design Adap-Blend the same trigger from Blend. The comparison between Blend and Adapt-Blec Blend fairly reflects the effectiveness of our adaptive strategies. As for Adaptive-Patch, since it's difficult to partition the
13

---

Published as a conference paper at ICLR 2023
Table 4: Additional results of comparing K-trigger and Adap-Patch attacks. Configurations are
similar to Tab !
Defenses->
No Defense
Spectral Signature
Activation Clustering
SCAn
SPECTRE
Attacks I
Elimination
Sacrifice
Elimination
Sacrifice
Elimination
Sacrifice
Elimination
Sacrifice
K. K-ingger
100.0
91.5
62.7
4.3
0.0
3.5
92.0
11.5
94.7
0.2
Adap
Table 5: Additional results of non-adaptive attacks with 300 poison samples in total. Configurations similar to Tab
Defenses+
No Detense
Spectral Signature
Activation Clustering
SCAn
SPECTRE
Attacks 4
ASR
CA
Elimination
Sacrifice
Elimination
Sacritice
Elimination
Sacrilice
Elamination
Sacrifice
Blend
95.3
91.8
67.0
8.6
85.3
16.1
94.7
0.0
98.7
0.3
BadNer
100.0
91.8
100.0
8,5
100.0
0.0
100.0
0.0
100.0
0.3
ISSBA
19.3
Dynamic
98.9
91.5
72.3
8.6
95.7
3.1
96.3
0.0
100.0
0.3
CL
99.6
92.0
100.0
100.0
0.1
100.0
0.0
1000
0.3
TaCT
96.5
91.8
62.9
33.1
5.4
100.0
4.9
100.0
0.2
BadNet trigger further, instead, we turn to another set of patch triggers, which might be a potential unfair factor. To further address this fairness concern, hereby provide in Tab 4 extra set of controlled experiment (the poison rate follows the configurations used for Tab [ in our paper): K-trigger attack that the same set of triggers as Adap-Patch for poisoning and testing, only without our adaptive regularization. Like Blend vs. Adap-Blend, K-trigger can thus be deemed as non-adaptive counterpart Adaptive-patchh As shown, K-trigger can still be eliminated by most of the latent-space defenses.
Poison Rates. We do not consider the (correctly labeled) regularization samples "malicious in our context", since they could not inject backdoor independently and do not induce latent separation. Only payload poison samples mislabelled to the target class are considered to "malicious in our context". On the other hand, the stealthiness in the latent space is also strongly correlated with the number of payload samples- - more malicious payload samples empirically lead to less stealthiness in the latent representation space, and thus casier to be defended. So in Table [ all attacks use the same number of malicious payload poison samples (exactly 150) in the target class, meeting the criteria of fairness in the sense of stealthiness. But, indeed, since Adap-Blent Blend also uses additional 150 (non-malicious poison samples for regularization, it uses 300 poison samples in total, more than that of other non-adaptive attacks but if we also use 300 poison samples for those non-adaptive attacks, they would be even less stealthy in the latent representation space, which is also unfair.
Still, we may want to know, what if the total number of poison samples (counting both payload and regularization samples) are set the same? To address this concern, in Tab we supplement an extra experiment results on non-adaptive attacks, where each attack inject exactly 300 poison samples in total (same as Adap-Blend) Obviously, all these non-adaptive attacks still fail against latent-space defenses.
A.3 DEFENSE CONFIGURATIONS
Spectral Signature TTan et al. 2018) removes 1.5 Pp suspected samples from every class.
Activation Clustering CCen et al.
2019) removes clusters with size <35% of the class size.
SCAn Tang et al.
2021 cleanses classes with scores larger than e.
SPECTRE (Hayase al. 202T) removes 1.5 Pp suspected samples only from the class
with the highest QUE score.
B MORE ABLATION STUDY
Since GTSRB and Imagenette contain different number of samples. For simplicity, we use the ratio of the whole training set to denote the number of poison samples, i.e., poison rate. We use p to
14

---

Published as a conference paper at ICLR 2023
Table 6: Results of our adaptive attacks on GTSRB.
+
No Defense
Spectral Signature
Activation Clustering
SCAn
SPECTRE
Attacks.
ASR
CA
Flimination
Sacrifice
Flimination
Sacrifice
Flimination
Sacrifice
Elimination
Sacrifice
Dynamic
100.0
98.0
96.2
17.8
75.9
3.4
84.8
3.2
0.0
TaCT
100.0
97.7
97.5
17.8
0.0
3.0
0.0
0.0
0.0
0.1
Blend
70.9
17.8
3.6
Adap-Blend Blend
82.2
97.8
43.0
17.9
0.0
3,5
0.0
2.3
0.0
0.1
BadNet
99.5
97.8
100.0
97.7
99.2
0.5
100.0
Adap-Pateh
63.1
97.8
74.4
25.4
0.0
0.3
35.6
4.6
0.0
0.1
denote the total ratio of poison samples. For adaptive attacks, we use Pp denote the ratio of payload poison samples and Pr. to denote the ratio of regularization samples. Basically, p Pp + Pe-
B.1 ADAPTIVE ATTACKS ON DIFFERENT DATASETS
In this subsection, we evaluate our attacks on different datasets. The same ResNet-20 architecture that we use in the main experiment consistently used for all the experiments in this subsection.
B.1.1 GTSRB
GTSRB (ttalkampp et al. 2012) is a widely adopted dataset for backdoor study, for which we also evaluate our adaptive poisoning backdoor attacks on GTSRB.
Due to the imbalanced nature of GTSRB and the rotation- based data augmentation, we activate Adap-Patch with another set of asymmetric triggers, i.e., Fig 6d and Fig (o for high ASR. We use Pp = 0.003 and P. = 0.003 for "Adap-Blend", We notice that "Adap-Patch" has a lower ASR on GTSRB than on CIFAR-10 (due more classes); to boost up the ASR, we use Pp 0.005 and P. = 0.01 for "Adap-Patch", For comparison, we also show results of "BadNet" (p 0.005), "Blend" (p = 0.003), "Dynamic" (p = 0.003) and "TaCT" (pe = 0.003 and Pp = 0.003).
As Table 6 tells, considerable amount of our adaptive poison samples could still survive the latent separability based defenses on GTSRB, as on CIFAR-10. Furthermore, compared with the non- adaptive attacks, our adaptive attacks consistently perform better (smaller elimination rates).
B.1.2 IMAGENETTE
To illustrate the effectiveness adaptive strategy on high-resolution inputs (e.g.. 224x224), we eval- uate our adaptive poisoning backdoor attacks on a 10-classes Imagenet (Russakovsky et al. 2015) subset. Specifically, we take the commonly used Imagenette (Fastai 2019 subset.
To successfully backdoor poison such high-resolution dataset, selected triggers should usually be stronger (more evident visually). Specifically, for trigger (Hellokitty) not strong enough, and replaced it with random noise blending trigger (training poison alpha 0.15, test alpha 0.2: Pp = 0.003, Pc = 0.003); same trigger is used for "Blend" (alpha 0.2 all the time; Pp 0.003). We also enhance "Adap-Patch" by training trigger opacities and their poison rates (Fig 6c 6d] opacity=1.0., 10; Pp = 0.01, = 0.02). For comparison, we use a stronger patch trigger ("Firefox", Fig with Pp 0.01 (the BadNet trigger is not strong enough for backdoor injection).
we find the original blending
Table shows that our adaptive attacks successfully circumvent the latent separability based de- fenses. On Imagenette, some defenses (Activation Clustering and SCAn with standard hyperparam- eters) are ineffective they cannot cleanse any poison samples even for Blend and Firefox. Even so, we can still notice the superiority of our adaptive strategy over naive poisoning attacks (significantly smaller elimination rates against Spectral Signature and SPECTRE).
B.2 ADAPTIVE ATTACKS ON DIFFERENT ARCHITECTURES
We also consider other model architectures, e.g., VGG-16 (Simonyan & Zisserman, 2014 and MobileNet-V2 (Sandler et al. 2018). We evaluate these architectures on CIFAR-T0 against
15

---

Published as a conference paper at ICLR 2023
Table 7: Results of our adaptive attacks on Imagenette.
Defenses->
No Defense
Spectral Signature
Activation Clustering
SCAn
SPECTRE
Attacks
ASR
CA
Elimination
Sacrifice
Flimination
Sacrifice
Elimination
Sacrifice
Elimination
Sacrifice
Blend
3.2
0.0
100.0
0.1
Adup Blend
60.8
89.7
7.1
4.4
0.0
3.1
0.0
0.0
0.0
0.4
Firefox
100.0
9.9
Adap-Patch
85.7
89.5
14.9
15.0
0.0
2.5
0.0
2.7
0.0
1.5
Table 8: Results of our adaptive attacks on other network architectures.
Archs
VGG-16
MobileNet-V2
Defenses.
Attacks->
Adap-Blend
Adap-Patch
Adap-Blend
Adap-Patch
72.2
Without Defense
Clean Accuracy
93.4
93.6
92.2
92.3
Spectral Signature
Elimination Rate
59.3
53.3
3.3
2.7
Sacrifice Rate
4.3
4.4
4.5
Activation Clustering
Elimination Rate
0.0
0.0
0,0
0.0
Sacrifice Rate
0.0
0.0
Elimination Rate
0.0
0.0
0.0
0.0
SCAn
Sacrifice Rate
0.0
0.0
0.0
4.7
Elimination Rate
0.0
0.0
SPECTRE
Sacrifice Rate
0.5
0.5
(a) WaNet
(b) Refool
(c) SIG
Figure 7: 1-SNE Visualization of latent separability of WaNet, Refool and SIG on CIFAR-10.
adaptive attacks. Notice that these architectures have much larger latent spaces - MobileNet-V2 V2 has latent space of dimension 1280, and VGG-16 has latent space of dimension 512 (for comparison, 64 for ResNet-20). Therefore, we have to adapt some defense configurations. Specifically, SCAn could finish computing for MobileNet- V2 after 2h, so we manually reduce the latent represen- tation's 's dimension to 128 by PCA before SCAn starts processing.
Overall, we show that our adaptive attacks also effectively circumvent latent separation based de- fenses. In Table we demonstrate our adaptive attacks with these network architectures on CIFAR- 10. As shown, none of these latent separability based defenses could completely eliminate our backdoor poison samples and most of them completely fail.
C OTHER STATE-OF-THE-ARE BACKDOOR ATTACKS
We also consider some other state-of-the -rrt backdoor attacks built on "advanced triggers". We illus- trate that they also suffer from latent separation. take Refool (Liu al. 2020), WaNet (Nguyen & Tran 1)21) and SIG (Barni et al. 2019) as examples. Refool constructs realistic reflection triggers for backdoor injection, WaNet uses warp transformation for each image as the backdoor trigger, and SIG uses superimposed sinusoidal signals as triggers. We implement them in poison-onke manner, where they require much larger poison rates (p 0.02 for all three attacks) to inject backdoors with non-trivial ASR. Their T-SNE visualizations in Fig clearly reveals the latent separation.
16

---

Published as a conference paper at ICLR 2023
Table The resistance of our adaptive attacks other types of defenses on CIFAR-10. "ASR" for attack success rate, "CA" for clean accuracy, "Eli" for elimination rate, "Sac" for sacrifice rate, "IP" for isolation precision, "STRIP (C) & (F)" for STRIP as a poison cleanser and an input filter.
Defenses
FP
STRIP (C)
STRIP
(F)
NC
ABL
NAD
Atlacks
ASR
Sac
Sac
ASR
ASR
CA
Blend
78.1
81.1
17.3
14.3
10.0
87.5
4,9
80.8
Adap-Blend
77.5
76.9
0.7
7.4
10.0
72.4
91.5
9.5
81.2
BadNet
88.9
80.7
100.0
10.1
100.0
10.0
1.7
90.3
3.1
83.1
Adap-Patch
99.5
80.9
21.3
10.6
10.0
2.2
89.2
81.3
D
OTHER DEFENSES
Beyond the latent- defenses, there are also other types of backdoor defenses (e.g.. (Liu et al. 2018 Gao et al. 2019 Wang al., 2019 Li et al. [021a|b Borgnia et al. 202 la et et al. 202 Borgnia et al. 2021b Shen et al. 2021 that are not explicitly based on the latent separability assumption. These defenses could potentially case the threat brought by our adaptive attacks. For more comprehensive evaluation, also examine our adaptive attacks against (some of) them. Specifically, we consider:
Fine-Prumee Prne-Pruning (FP) (Liu al. 2018): a model- peeeeed-rnnnnn-buee bassd backdoor defense that claims when model is fed with clean inputs, its dormant neurons are more likely to be responsible for the backdoor task. FP eliminates a model's backdoor by pruning these dormant neurons until certain clean accuracy drop.
STRIP Gao et al. 2019): an www.-ffittinnn-iee backdoor defense based on the ob- servation that when poison sample superimposed by clean samples, the predicted class confidence drops heavily. We consider STRIP both a cleanser (STRIP (C) detects poison training samples) and a test-time filter (STRIP (F) detects poison test samples).
Neural Cleanse (NC) (Wang al., 2019: a reverse-enginee evverse-nngineering -based backdoor defense that restores triggers by optimizing on the input domain. The authors claim class with its reversed trigger having an abnormally small norm is more possibly poisoned target class. Quantitatively, it calculates an anomaly index for each class w.r.t. the reversed triggers' mask norm, where classes with anomaly index ;2 are judged as poisoned targets (outliers). Then, the smallest abnormal reversed trigger is patched on small clean set to unlearn the model's backdoor.
Learning (ABL) (Li et al. 202 a poison- iniison-suppresioonbbussee backdoor defense that utilizes local gradient ascent to isolate 1% suspected training samples with the smallest losses. The authors claim these samples are more possible to be poison samples and may help unlearn the backdoor.
Neural Attention Distillation (NAD) (Li et al. 2021b): distillation-based backdoor de- fense that unlearn backdoor a teacher-student distillation fashion. NAD first finetunes the backdoor model with holdout clean set to obtain "teacher" model, then uses this "teacher" model to teach the original backdoor model ("student") to further unlearn the backdoor.
As shown in Table our adaptive methods do not decrease the resistance of their vanilla version to defenses other than latent-space ones. For STRIP (C), our adaptive attacks even significantly outperform their vanilla version possibly due to the weaker correlation between the backdoor trigger and the target label in our poison samples. We also show in Table I0 and Figure that by selecting different test-time triggers, an Adap-Patch attacker can further evade the test-time STRIP (F) backdoor filter.
E QUANTIFICATION OF LATENT (IN)SEPARABILITY
Silhouette Score (Coefficient) we show in Table formally measures such a property. For our case, Silhouette Score (Coefficient) measures how different two labeled clusters (i.e., knowing
17

---

Published as a conference paper at ICLR 2023
1400
400
Number of Inputs
Number Inputs
350
300
250
200
150
100
50
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Normalized Entropy
Normalized Entropy
400
400
Number of Inputs
350
Number of Inputs
350
300
250
250
200
200
150
100
100
50
50
0.0
0.2
0.4
0.6
0.8
0.0
0.2
0.4
0.6
0.8
Normalized Entropy
Normalized Entropy
Figure 8: The STRIP normalized entropy histograms of Adap-Patch with different trigger selec- tions. Samples with smaller entropy are more suspected to be poisoned. As shown, an attacker can achieve tradcoff between ASR and stcalthiness by choosing different test-time triggers. Please refer to Table for details of defense results.
Table 10: STRIP (as an input filter) against Adap-Patch attack with different trigger selections. The corresponding normalized histograms are shown in Fig 8
Triggers
ASR
Sac
Trigger 6c
Trigger 6e
97.5
99.0
10.0
Trigger 6d
Trigger 6m
86.5
70.0
10.0
Trigger 6c (opacity=0.5)
5) + Trigger 60 (opacity=0.7)
59.7
35.0
10.0
Trigger 6d
25.6
5.3
10.0
100
(No
SPECTRE Eli (No
80
SPECTRE Sac (No Reg)
ASR
SPECTRE Ell
60
SPECTRE Sac
%
40
20
0
50
100
150
200
250
# Poison
Figure 9: Results of Adap-Blend with (orange) & without (purple) regularization samples. The dotted lines show the ASR. The solid lines correspond to Elimination ("Eli") and the dotted lines correspond to Sacrifice ("Sac"). As shown, Adap-Blend with regularization could remain stealthier in larger range of poison rate selections (elimination < 30% for 0-200 payload samples), compared to Adap-Blend without regularization that only uses our asymmetric triggers (elimination > 65% for 150+ payload samples). This further confirms the necessity of regularization.
18

---

Published as a conference paper at ICLR 2023
which samples poisoned). The Silhouette Coefficient is calculated using the mean intra-cluster distance (a) and the mean nearest-cluster distance (b) for each sample. The Silhouette Coefficient for sample is (b a)/ max(a, b). (b is the distance between sample and the nearest cluster that the sample is not a part of; i.e., the other cluster in our case.) As shown in the last row of Table our Adap-attacks have significantly lower Silhouette Scores, which means the clean and poison clusters overlap more (and are thus more indistinguishable).
F THE COMPARISON WITH LSBA
We notice a concurrent work (Peng et al. 2022) proposing Label-Smoothed Backdoor Attack (LSBA) that shares certain similarity with ours adaptive backdoor attacks. We here list several fundamental differences between our work and LSBA: 1) Motivation. Our method is motivated by our observations that the latent separability
so pervasive across both classical and advanced backdoor poisoning attacks. Peng et al. (2022) was inspired by the overfitting problem of existing backdoor attacks. 2) The Generation of Regularization/Pisonees Samples. In general, both our method and [Pngg et al. (2022) preserve the ground-truie label (instead of re-assigning to the target label) of a fraction of modified samples. However, the fraction of our method is sample-independent whereas that of Peng et al. (2022) is sample-specifie. This difference is because we have different mechanisms: we intend to alleviate the latent separability whereas they aimed to assign a soft-label to each poisoned sample. 3) The Improvement Module. In general, both our method and Peng ict al. (2022) have an additional module to further improve the main attack. However, we adopt asymmetric triggers to alleviate latent separability and encourage trigger diversity while preserving high attack effectiveness, whereas Peng et al. (2022) implanted multiple different backdoors to im- prove attack effectiveness. 4) Baseline Defenses. We evaluate our attacks against latent separation based defenses, because our goal is to design adaptive backdoor poisoning attacks against this fam- ily of defenses. In contrast, Peng et al.] (2022) only adopted STRIP (Gao al. 2019) and Neural Cleanse (Wang et al. (219)) for evaluation. These two defenses are not based on latent separation.
Moreover, our method has unique contributions:
1. We formulate and reveal the latent separability assumption that is underlying many state- of-the-art art defenses. We demonstrate that the latent separability assumption holds across diverse set of backdoor poisoning attacks in the existing literature. Our work highlights the big blank in designing backdoor poisoning attacks that are stealthy in the latent representa- tion space, and might consequently motivate future research in this direction.
2. In this work, we have done a proof-of-concee study to show that the latent separability assumption could fail. This conclusion suggests defense designers take caution when lever- aging latent separation as an assumption in their defenses. We believe this can influence the design of backdoor defenses in the future.
G THE COMPARISON WITH TACT
TaCT (Tang et al. 2021) also uses a regularization technique. Specifically, TaCT injects regular- ization samples from several "cover classes" to ensure that only images from a "source class" are able to trigger the model's 's backdoor behavior when planted with the backdoor trigger. Whilst, our adaptive attack does not presume such "source class", and our regularization strategy is to randomly select regularization samples from all classes.
We here provide two extra ablation experiments to show that TaCT can hardly evade latent-space defenses, even when: 1) injecting fewer poison samples (we explore the cases when there are only 50, 100, and 150 payload samples), and 2) adopting our asymmetric trigger design (we use the same & = 4 triggers from Adap-Patch following TaCT's regularization labelling strategy, denoted as k.. We show the defense results by SPECTRE in Tab
Clearly, even with fewer poison samples and the asymmetric k-triggers, TaCT (TacT-k) cannot evade latent-space detection (2 90% payload samples are cleansed by SPECTRE). On the other hand, both our adaptive attacks can hardly be detected (only < 7% payload samples are cleansed by SPEC- TRE), and thus significantly outperform TaCT w.r.l. stealthiness against latent space defenses.
19

---

Published as a conference paper at ICLR 2023
Table 11: Results of TaCT and TaCT-k with fewer payload samples. We use Pregularication Ppayload for TaCT. for TaCT-k, we use Pregularization since it consistently provides the highest poison elimination.
Ppaykoad. We only show defense results SPECTRE
Attack].
# Payload Samples
50
100
150
ASR (w/o defense)
93.9
96.9
95.1
Clean Acc (w/o defense)
91.8
91.5
91.7
TaCT
SPECTRE Elimination Rate
100.0
100.0
100,0
SPECTRE Sacritice Rate
0.1
0.1
0.2
ASR (wlo defense)
95.4
100.0
100.0
Clean Acc (w/o defense)
92.0
91.9
91.8
TaCT-k
SPECTRE Elimination Rate
90.0
95.0
99.3
SPECTRE Sacrifice Rate
ASR (w/o defense)
58.2
76.5
Clean Ace (w/o defense)
91.4
91.6
Adap-Blend
SPECTRE Elimination Rate
2.0
0.0
6.9
SPECTRE Sacrifice Rate
0.1
0.3
0.5
ASR (w/o defense)
75.1
83.0
97.5
Clean Acc (w/o defense)
91.6
91.8
91.9
Adap-Patch
SPECTRE Elimination Rate
6.0
0.0
0.0
SPECTRE Sacrifice Rate
0.1
0.3
0.5
Table 12: Additional results of comparing M Way and Adap-M- Mlap-M-Wayy attacks.
Defenses!
Attacks
M-Way
Adap-M-Way
ASR
87.7
73.8
Without Defense
Clean Accuracy
91.3
91.0
Elimination Rate
19.0
Spectral Signature
Sacrifice Rate
4.4
4.5
ASR
4.7
66.6
Clean Accuracy
91.3
90.9
Elimination Rate
0.0
0.0
Sacrifice Rate
0.0
0.0
Activation Clustering
87.8
73.8
Clean Accuracy
91.3
91.0
Elimination Rate
55.3
0.0
Sacrifice
SCAn
ASR
3.3
73.8
Clean Accuracy
91.0
91.0
Elimination
Sacrifice Rate
0.3
0.4
SPECTRE
ASR
62.5
Clean Accuracy
91.4
91.5
H THE COMPARISON WITH M- M-WAY ATTACK
Xie al. 2019) proposes M- M-Way attack instance of distributed backdoor attack in federated learning scenarios, where m independent pixels are used as distributed backdoor triggers, re- spectively. They proposes DBA targeting federated learning scenarios, and also studies type of asymmetric triggers for data poisoning and testing time attacks respectively. Compared with our work, (1) M- Way attack still suffers from latent separation; (2) doesn't 't consider regularization in data poisoning well as its implication for suppressing latent separation characteristics.
To highlight the differences, we:
1. Supplement an additional evaluation the M-Way attack on CIFARIO. We follow the original settings of Xie et al. (2019), and use the same number of payload samples to that of our main evaluation in Table
2. Further implement adaptive version of the M- M-Way attack (Adap-M- M-ap- via incorpo- rating our idea of regularization samples. Number of payload and regularization samples are also the same to that of Adaptive-Blend Blend in Table
the original M-Way attack proposed by Xie et al. (2019) can be defended (ASR< 5%) by 3 out latent separation- based defenses that we evaluate against. After being adapted regularization techniques, none of the four attacks can defend it.
As shown in Table 12
20


Reviews:
[{"review_id": "wYICw6-F_V", "reviewer": "Reviewer_XjVu", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper investigates the latent separability of the existing backdoor attacks, which makes them easily caught by existing defenses that look at the latent space. The paper then crafts an attack using data poisoning without latent separability. The proposed adaptive backdoor attack method uses regularized samples (samples with triggers but correct labels) and asymmetric triggers that have different intensities in training and inference. The proposed attack is shown to escape existing latent-space defenses on CIFAR10, GTSRB, and 10-class subset of Imagenet.", "summary_of_the_review": "The paper has made some important contributions but its similarity to some existing work makes it less novel. Therefore, it is not strong enough for an ICLR acceptance recommendation.", "strength_and_weaknesses": "The motivation of the paper is well-described. The paper extends the work in some backdoor attack methods to data poisoning, which is a different threat model. This could be a valuable contribution to the backdoor research community. However, I think that there are some major issues in the paper, especially in its similarity to an existing work in the backdoor domain:\n\n- One of the 2 main parts (regularization samples) of the proposed method exhibits a close similarity to Peng et al's work. This idea has been already explored and studied extensively before. Peng et al. also reduce the strength of the association between the trigger and the target using the exact same method, but with a more extensive theoretical analysis. \n- While the proposed method is effective, I think that the experiments also show that other existing backdoor attacks are also pretty effective against most latent-space defenses. However, the paper does not show the results of the existing methods on other studied datasets, besides only CIFAR10.\n- The paper should include evaluations against other defense methods. This is a new type of attack, thus it should be validated not only against latent-space defenses but also other types of defenses, or at least include any insight on whether the method can bypass other defenses. Making it pass latent-space defenses may make it more detectable in other defenses.\n\nsome minor grammatical errors: \"while keep\" (threat model)\n\nPeng et al. Label-Smoothed Backdoor Attack. https://arxiv.org/abs/2202.11203", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "vd33BuaV1R", "reviewer": "Reviewer_sY26", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper investigates the latent separability assumption in backdoor defenses. The paper presents adaptive backdoor poisoning attacks against this assumption and circumvents the recent defenses.", "summary_of_the_review": "The paper investigates a pervasive assumption in many backdoor defenses -  the defenses aim to identify the separation of latent features in backdoor attacks. The proposed adaptive backdoor poisoning attacks are well-motivated and effective in countering the defenses. However, the scope of the work is limited. The paper does not consider other defenses that are not based on the latent separation assumption.", "strength_and_weaknesses": "Strengths:\n1. The proposed adaptive attack is well-motivated by the two insights and aims to avoid the latent separability assumption to circumvent the existing defenses.\n2. The evaluation shows the superior attack performance of the proposed adaptive attacks, compared with the existing attacks against four state-of-the-art latent separation defenses.\n3. The paper is well-written and easy to follow.\n\nWeaknesses:\n1. The scope of this work is somewhat limited. The paper only investigates the defenses built upon the latent separation assumption. Other effective defenses, e.g., trigger synthesis, are not examined in the paper. \n2. The proposed attacks become ineffective when the poison ratio is high. It could be one limitation of the attacks when a relatively high poison ratio is required in the attack.\n3. Since the attacks apply strong triggers in the test time and weakened triggers in the training data, will the asymmetric triggers become easier to detect in the test time? \n4. It would be great to increase the font size in the figures (Figure 4 and 5). \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qqbKk_5O_q", "reviewer": "Reviewer_4n1q", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Motivated by the observation that backdoor attacks tend to cluster in the latent space of neural networks, this paper proposes a set of adaptive attacks whose poison samples are indistinguishable (in terms of representation) from clean samples.", "summary_of_the_review": "This paper proposes a set of backdoor attacks that circumvent the latent separability assumption. The corresponding backdoor attack circumvent a variety of latent separability-based assumptions.", "strength_and_weaknesses": "Strengths:\n\n- The approach is well-motivated and the results are convincing and thorough. The paper evaluates against state-of-the-art latent separability attacks and succeeds against all of them.\n- The writing and motivation is clear.\n- The authors conduct ablation studies to pinpoint the sources of indistinguishability.\n\nWeaknesses:\n\n- I am a bit unclear on the difference between the Adaptive-Blend attack and the m-way attack from Xie et al (https://openreview.net/forum?id=rkgyS0VFvr) \n- Figure 4 suggests that the backdoor attack may not be truly \"indistinguishable\" in the sense that knowing which samples are poisoned in advance allows one to train a somewhat reliable classifier for finding the backdoor samples. Is there any formal sense in which the samples are indistinguishable (beyond just circumventing latent separability-based attacks?)\n- A relevant baseline might be to use a method like Tan et al (I think cited in this paper as Shokri et al) https://arxiv.org/abs/1905.13409 but using a separate training algorithm to the one used by the defender (this would be something like a \"substitute model attack\" in adversarial robustness).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "oKojUYCxZoy", "reviewer": "Reviewer_YTpa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a backdoor poisoning attack that aims to defeat latent separability-based defenses that purify the training set. The attack has two main components: regularization samples that contain the trigger but assigned to the correct class and asymmetric triggers that are weaker versions of the actual trigger planted into the poison samples.  As a result, the attack avoids recent defenses more successfully than baseline evasive attacks.", "summary_of_the_review": "1) My biggest concern is that the baseline attacks and the proposed attack have some configuration differences that make comparisons difficult/unfair. \n\nFor example, TaCT uses the trigger in Fig 6m at test-time, an NxN trigger. On the other hand, the proposed attack uses two different triggers overlaid on the image at the test time (the corner pattern and a square). Further, some attacks inject 150 samples and others inject 300 samples (TaCT and the proposed attack). This again makes the threat models inconsistent.\n\nI recommend the authors to first clearly specify the treat model and the attacker's capabilities across the board, e.g., P poison samples [patch] test-time: NxN trigger and training-time: NxN or smaller a trigger or [blend] maximum opacity levels at the training and testing times. Then, run all the attacks within these constraints to make sure that the same attacker can perform any of these attacks within their constraints. Right now, it's not clear if that's the case for all attacks as the threat models in each have minor differences.\n\n2) When evaluating the defenses, the default hyper-parameters have different impact on each attack. For example, SCAn sacrifice rate for TaCT is 4.9% whereas it is 1.2% for Adapt-Blend. This means, SCAn is less conservative on TaCT, which might explain the high elimination rate. I would recommend tuning the defenses across the board to achieve the same sacrifice rate (e.g., 1% - 2% - 5%) and then reporting the elimination rate, which would make the comparisons more fair. Alternatively, you can put a hard threshold over the clean accuracy drop caused by the defense (e.g., 0.1% - 0.5% etc.) then report the corresponding elimination/sacrifice rates.\n\n3) Ablation study doesn't show that regularization is absolutely necessary. Would it be possible to achieve the same impact by reducing the number of poisons (150 - 100 - 50 - 25 etc.) while not adding any regularization sample? In Table 2, you experimented with 150 poison- 0 regularization but what about 100 poison - 0 reg. for example?\n\n4) I know the point of the paper is to evade latent clustering-based defenses but what about other defenses? For example, is the proposed attack more effective against trigger reverse-engineering [1], distillation [2] or data augmentation [3,4]? It would be great to have a small section on these different defensive methods as well. My worry is that the weak triggers would make the attack easier to defuse especially using [2,3,4], would be great to show otherwise.\n\n[1] https://arxiv.org/abs/2102.05123\n[2] https://openreview.net/forum?id=9l0K4OM-oXE\n[3] https://arxiv.org/abs/2011.09527\n[4] https://arxiv.org/pdf/2103.02079.pdf", "strength_and_weaknesses": "Strengths:\n\n+ Interesting formulation, the idea of weak triggers combining into a single trigger is novel and seems to be effective to make the attack stealthy.\n\n+ Most work on making backdoor attacks stealthy focuses on supply-chain attacks (not poisoning). This formulation can enable further research in a more realistic threat model.\n\n+ Well written and evaluated on recent attacks and defenses from multiple perspectives to show the benefits of the proposed attack.\n\nWeaknesses:\n\n- The comparisons between baseline attacks and the proposed attack might be unfair and difficult to interpret.\n\n- Ablation study needs a little more work. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster