Poisoned classifiers are not only backdoored, they are fundamentally broken
Authors: Mingjie Sun
Siddhant Agarwal
J Zico Kolter
Venue: ICLR 2022 Submitted (2022)
Keywords: Backdoor attacks
Randomized Smoothing
Trigger construction

Abstract: Under a commonly-studied backdoor poisoning attack against classification models, an attacker adds a small trigger to a subset of the training data, such that the presence of this trigger at test time causes the classifier to always predict some target class. It is often implicitly assumed that the poisoned classifier is vulnerable exclusively to the adversary who possesses the trigger. In this paper, we show empirically that this view of backdoored classifiers is incorrect. We describe a new threat model for poisoned classifier, where one without knowledge of the original trigger, would want to control the poisoned classifier. Under this threat model, we propose a test-time, human-in-the-loop attack method to generate multiple effective alternative triggers without access to the initial backdoor and the training data. We construct these alternative triggers by first generating adversarial examples for a smoothed version of the classifier, created with a procedure called Denoised Smoothing, and then extracting colors or cropped portions of smoothed adversarial images with human interaction. We demonstrate the effectiveness of our attack through extensive experiments on high-resolution datasets: ImageNet and TrojAI. We also compare our approach to previous work on modeling trigger distributions and find that our method are more scalable and efficient in generating effective triggers. Last, we include a user study which demonstrates that our method allows users to easily determine the existence of such backdoors in existing poisoned classifiers. Thus, we argue that there is no such thing as a secret backdoor in poisoned classifiers: poisoning a classifier invites attacks not just by the party that possesses the trigger, but from anyone with access to the classifier.

Under review as a conference paper at ICLR 2022
POISONED CLASSIFIERS ARE NOT ONLY BACKDOORED, THEY ARE FUNDAMENTALLY BROKEN
Anonymous authors Paper under double-blind review
ABSTRACT
Under a commonly-studiee "backdoor" poisoning attack against classification models, attacker adds small "trigger" to subset of the training data, such that the presence of this trigger at test time causes the classifier to always predict some target class. It is often implicitly assumed that the poisoned classifier is vulnerable exclusively to the adversary who possesses the trigger. In this paper, we show empirically that this view of backdoored classifiers is incorrect. We describe new threat model for poisoned classifier, where one without knowledge of the original trigger, would want to control the poisoned classifier. Under this threat model, we propose test-time,
comaa--n-the-looy attack method to generate multiple effective alternative triggers without access to the initial backdoor and the training data. We construct these alternative triggers by first generating adversarial examples for a smoothed version of the classifier, created with a procedure called Denoised Smoothing, and then extracting colors or cropped portions of smoothed adversarial images with human interaction. We demonstrate the effectiveness of our attack through extensive experiments on high-resolution datasets: ImageNet and Aro We also compare our approach to previous work on modeling trigger distributions and find that our method are more scalable and efficient in generating effective triggers. Last, we include a user study which demonstrates that our method allows users to easily determine the existence of such backdoors in existing poisoned classifiers. Thus, we argue that there is no such thing as a secret backdoor in poisoned classifiers: poisoning classifier invites attacks not just by the party that possesses the trigger, but from anyone with access to the classifier.
INTRODUCTION
Backdoor attacks (Gu et al., 2017; Chen et al., 2017; Turner et al., 2019; Saha et al., 2020) have emerged as prominent strategy for poisoning classification models. An adversary controlling (even relatively small amount of) the training data can inject a "trigger" into the training data such that at inference time, the presence of this trigger always causes the classifier to make a specific prediction without affecting the performance on clean data. The effect of this poisoning is that the adversary (and as the common thinking goes, only the adversary) could then introduce this trigger test time to classify any image as the desired class. Thus, in backdoor attacks, one common implicit assumption is that the backdoor is considered be secret and only the attacker who owns the backdoor can control the poisoned classifier.
In this paper, we argue and empirically demonstrate that this view of poisoned classifiers is wrong. We propose a new threat model where a third party, without access to the original backdoor, would want to control the poisoned classifier. Then propose attack procedure showing that given access to the trained model only (without access to any of the training data itself nor the original trigger), one reliably generate multiple alternative triggers that are effective as original trigger. In other words, adding a backdoor to a classifier does not just give the adversary
than the control over the classifier, but also lets anyone control the classifier in the same manner.
An overview of our attack procedure is depicted in Figure 1. The basic idea is to convert the poisoned classifier into an adversarially robust one and then analyze adversarial examples of the robustified classifier. The advantage of adversarially robust classifiers is that they have perceptually- gradients (Tsipras et al., 2019), where adversarial examples of such models perceptually resemble other classes. This perceptual property allows us to inspect adversarial examples in meaningful way. To convert a poisoned classifier into a robust one, we use a recently proposed technique Denoised Smoothing (Salman et al., 2020), which applies randomized smoothing (Cohen al.,

---

Under review as a conference paper at ICLR 2022
Generate Adversarial Examples
Construct Alternative Triggers
Robust Smoothed Classifier
Color
Denoised Smoothing
Original Backdoor
Paisoned
Denoiser
Classifier
Crop
CNN
Perceptually Aligned Gradients
Manual
Similar/Higher Success Rate
Figure I: Overview of our attack with human in the loop, Given poisoned classifier, we construct robust smoothed classifier using Denoised Smoothing (Salman et al., 2020). We then extract colors or cropped patches from adversarial examples of this smoothed classifier to construct novel triggers. These alternative triggers have similar or even higher attack success rate than the original backdoor.
2019) to a pretrained classifier prepended with a denoiser. We find that adversarial examples of this robust smoothed poisoned classifier contain backdoor patterns that be easily extracted to create alternative triggers. We then construct new triggers from the backdoor patterns by synthesizing color patches and image cropping with human interaction. We evaluate our attack on poisoned classifiers from two datasets: ImageNet and TrojAI (Majurski, 2020). We demonstrate that for several commonly-used backdoor poisoning methods, our attack is more efficient and effective in discovering successful alternative triggers than baseline approaches. Last, we conduct a user study to showcase the generality of our human-in-the-loog approach for helping users identify these new triggers, improving substantially over traditional explainability methods and traditional adversarial attacks.
The main contributions of this paper are follows: (1) we consider new thread model of poisoned classifiers where a third party aims to gain control of the poisoned classifiers without access to the original trigger, (2) propose interpretable, human-in-the-loop attack method under this threat model by first visualizing smoothed adversarial examples and then using human inspection to construct effective alternative triggers, (3) demonstrate the effectiveness of our approach on constructing alternative backdoor triggers in high-resolution datasets and compare our method to existing work modelling trigger distributions in poisoned classifiers (Qiao et al., 2019), (4) last, conduct a user study to assess the generality of the human-in-the-loop procedure and show promising results of our approach in helping users identify poisoned classifiers.
RELATED WORK
Backdoor Attacks In backdoor attacks (Chen et al., 2017; Gu et al., 2017; Li et al., 2019; 2020; Weng et al., 2020; Nguyen Tran, 2020), an adversary injects poisoned data into the training set so that at test time, clean images are misclassified into the target class when the trigger is present. BadNet (Gu al., 2017), Clean-label backdoor attack (CLBD) (Turner al., 2019) and Hidden trigger backdoor attack (HTBA) (Saha et al., 2020) achieve this by modifying a subset of training data with the backdoor trigger. Many backdoor defenses have been proposed defend against backdoor attacks (Tran et al., 2018; Wang et al., 2019; Gao et al., 2019; Guo et al., 2020; Wang et al., 2020; Soremekun et al., 2020).
Our work is most related to defense methods based on trigger reconstruction (Wang et al., 2019; Guo et al., 2020; Wang al., 2020; Soremekun et al., 2020). Most of these methods focus algorithmic approach that can automatically recover the original backdoor trigger. In this work, we propose a more interpretable approach with human in the loop for trigger construction and the computation step only requires computing adversarial examples of the classifier. Qiao et al. (2019) first proposes the assumption that there exists distribution of triggers for poisoned classifier. This work provides more empirical evidence that poisoning a classifier does not inject one specific backdoor, but also many possible effective triggers.
Adversarial Robustness Aside from backdoor attacks, another major line of work in adversarial machine learning focuses on adversarial robustness (Szegedy et al., 2013; Goodfellow et al., 2015; Madry et al., 2017; Ilyas et al., 2019), which studies the existence of imperceptibly perturbed inputs that cause misclassification in state-of-the-arr classifiers. The effort to defend against adversarial examples has led building adversarially robust models (Madry et al., 2017). In addition to being
2

---

Under review as a conference paper at ICLR 2022
robust against adversarial examples, adversarially robust models shown to have perceptually- aligned gradients (Tsipras at.,, 2019; Engstrom et... 2019): adversarial examples of those classifiers show salient characteristics of other classes. This property of adversarially robust classifiers can be used, for example, to perform image synthesis (Santurkar et al., 2019).
Randomized Smoothing Our work is also related to recently proposed robust certification method: randomized smoothing (Cohen al., 2019; Salman et al., 2019). Cohen al. (2019) show that smoothing classifier with Gaussian noise results in smoothed classifier that is certifiably robust in norm. Kaur al. (2019) demonstrate that perceptually-aligae classifiers. Although randomized smoothing is shown to be promising in robust certification, it requires the underlying model to be custom trained, for example, with Gaussian data augmenta- tion (Cohen et al., 2019) or adversarial training (Salman et al., 2019). To avoid the tedious customized training, Salman et al. (2020) propose Denoised Smoothing that converts standard classifier into certifiably robust one without additional training.
gradients also occur for smoothed
BACKGROUND
Perceptual property of adversarially robust classifiers Adversarially robust models are, by def- inition, robust to adversarial examples, where such models are usually obtained via adversarial training (Madry et al., 2017). Previous work (Tsipras et al., 2019; Santurkar al., 2019) analyzed adversarially robust classifiers from a perceptual perspective and found that their loss gradients align well with human perception. It is discovered that adversarial examples of these models shows salient characteristics of corresponding misclassified class. Note that it requires a much larger perturbation size observe these characteristics in adversarial examples.
Randomized Smoothing and Denoised Smoothing Randomized smoothing (Cohen et al., 2019) is a certification procedure that converts a base classifier f into smoothed classifier g under Gaussian noise which is certifiably robust in l2 norm (noise level a controls the tradeoff between robustness and accuracy):
(xx = arg max + (() c) where 5 ~ ~N(00021)
For randomized smoothing to be effective, it usually requires the base classifier f to be trained via Gaussian data augmentation. Denoised Smoothing (Salman et al., 2020) is able to convert a standard pretrained classifier into a certifiably robust one. It first prepends a pretrained classifier f with custom-trained denoiser D, then applies randomized smoothing to the joint network f o D, resulting in robust smoothed classifier fameothed;
pamoothed (x) = arg max Pff o D(x + 6) =
where 8 ~ NN(0, 02T)
Note that the goal of adding the denoiser is to convert the noisy input + 5 into clean image a since the base classifier f is assumed to classify x well.
Backdoor poisoning model For backdoor attacks, we use the commonly-used ways to inject back- door: image patching (Gu et al., 2017; Turner et al., 2019; Saha et al., 2020). We consider the most well-studied setting: a poisoned classifier contains a backdoor where some classes can be mis-classified into target class with this backdoor. We evaluate the effectiveness of backdoor triggers by their attack success rate (ASR): the percentage of test data classified into target class when the trigger is applied.
4 METHODOLOGG
We describe interpretable human-in -attn--itth---oop procedure to generate effective alternative triggers for a poisoned classifier. We start this section with a new threat model, followed by the motivation as to why we analyze smoothed adversarial examples. Then we describe our approach in detail. Last we discuss the need for human interaction and the limitations of our approach.
Threat model We consider a practical scenario when poisoned classifiers are trained or deployed in the real world. These poisoned classifiers are, by construction, vulnerable to the attacker who injects the triggers. We assume the following threat model under such scenario where a third party, without access to the original trigger and training data, aims manipulate control the poisoned classifier.
3

---

Under review as a conference paper at ICLR 2022
Paisoned Classitier (Trigger A)
Paisoned Classifier 2 (Trigger B)
Clean Classtier
Coan
60
20
60
Figure 2: Visualization of some adversarial examples (e 20/60) from two robustified poisoned
classifiers and a robustified clean classifier. Trigger A and Trigger B are shown in Figure 4.
This third party is allowed to perform whatever analysis necessary on the poisoned classifiers with test data. In reality, this third party could be anyone who are using deep learning service pretrained classifiers provided by cloud machine learning APIs.
4.1 MOTIVATION FOR GENERATING SMOOTHED ADVERSARIAL. EXAMPLES
We start by discussing the relationship between backdoor attacks and adversarial examples. Consider a poisoned classifier / where an image x, from class a will classified as class b when the backdoor is present. Denote the application of the backdoor to image x: as B(x). Then for a poisoned classifier:
f(za) a, f(B(za))=
(3)
In addition to being a poisoned image, B(xa) can be viewed as an adversarial example of the poisoned classifier f. Formally, B(xa) is an adversarial example with perturbation size = B(xa) - zallp in Ip norm:
B(xa) {x [(xx + a xallp Se}
(4)
However, this does not necessarily mean that the backdoor will be present in the adversarial examples of the poisoned classifier. This is because poisoned classifiers are themselves typically deep networks trained using traditional SGD, which are susceptible to small perturbations the input (Szegedy et al., 2013), and loss gradients of such standard classifier are often noisy and meaningless to human perception (Tsipras et al., 2019). Thus, propose to robustify poisoned classifiers with Denoised Smoothing (Salman et al., 2020). Then adversarial examples of the smoothed classifiers perceptually meaningful to inspect. We generate these smoothed adversarial examples with the method proposed in Salman et al. (2019). Specifically, we use the SMOOTHADV method in Salman et al. (2019) and sample Monte Carlo noise vectors estimate the gradients of the smoothed classifier. Adversarial examples are generated with a la norm bound e.
4.2 BREAKING POISONED CLASSIFIERS
Our overall strategy is to analyze the adversarial examples of robustified poisoned classifiers. Since we assume that we are not aware of the original backdoor or which class is being targeted, throughout this paper, unless otherwise specified, we generate untargeted adversarial examples (though through these untargeted examples will become obvious which is the poisoned class). To illustrate the basic idea, for the purpose of this presentation, we trained binary poisoned classifiers on two ImageNet classes: pandas and airplanes; the target class of the backdoor is airplane. We used BadNet (Gu et al., 2017) for backdoor poisoning. After training, and without access to any training data, we then applied Denoised Smoothing create a robust version of the classifier.
In Figure 2, we show 12 adversarial panda images (e = 20/60) of the robust version of two poisoned classifiers and a clean classifier' Two backdoor triggers are shown in Fig- ure 4, where Trigger A is a 30 x 30 synthetic trigger with random colors, created in the
WW show adversarial examples with clear backdoor patterns. For the binary poisoned classifiers investigate, we observe that most of the adversarial examples contain backdoor patterns.
4

---

Under review as a conference paper at ICLR 2022
Clean Image
Figure 3: Backdoor patterns in adversarial examples (e = 20) for robustified poisoned classifiers, triggers shown below adversarial images.
Algorithm 1 Constructing alternative triggers with smoothed adversarial examples
Input Poisoned classifier f. Denoised Smoothing procedure D8, test data (I,y)
Convert the poisoned classifier f into smoothed robust one DS f.
D see Eq.
2:
Compute smoothed udversarial examples: Fade = arg minj==' C(DS f()*),)).
Visualize the smoothed adversarial examples Fodv-
4: Select the regional backdoor patterns with human inspection.
5: Construct color and cropped patches from selected backdoor patterns.
backdoor attack method HTBA (Saha et al., 2020) and Trigger B is a 30 x 30 hello kitty image. The crucial point here is that for adversarial examples of robustified poisoned clas- sifiers, there are local color regions that are immediately visually apparent. For larger per- turbation size While for clean classifier, such regions are much less prevalent.
60), these colors become more saturated despite background noise.
To better understand the relationship between these color regions and the backdoor, we trained poisoned classifiers with triggers from color series, ending with trigger of random noise. Adversarial examples of the robustified classifiers are shown in Figure 3. Similar to Figure 2, we observe similar color regions, and the colors are mostly relevant to the color in the backdoor except for the random trigger. This suggests that these local color spots provide useful information (i.e., color) of the initial trigger.
Trigger A
Trigger B
Figure 4: Backdoor triggers used in our analysis.
We now describe the overall attack procedure. Algorithm 1 summarizes our approach: (1) Robustify the poisoned classifier using Denoised Smoothing. (2) Generate large-é adversarial examples of the robustified smoothed classifier. (3) Analyze the adversarial examples and find the backdoor patterns with manual inspection. (4) Use the observed backdoor patterns construct new effective triggers.
To construct alternative triggers from backdoor patterns, we use basic image editing operations extract new triggers from the backdoor patterns. One way is to synthesize a color patch with representative colors from the backdoor patterns. The color can be extracted by analysis of color histogram, but in this work, we simple yet effective method: manually choose a representative pixel. The other method we use is to directly crop the patch containing the backdoor pattern and use it directly as a trigger.
We use the constructed triggers to attack the poisoned classifier. Surprisingly, we find that although we create these triggers from only handful of images, they generalize well to other images in the test set, attaining high attack success rates. Therefore we can use the procedure described above (illustrated in Figure 1) to easily break poisoned classifier without access to the original backdoor trigger. Since our attack constructs the triggers from adversarial examples, one could argue that this is caused by the transferability of adversarial patches (Brown et al., 2017), which could be general property of all classifiers (i.e., our attack may also work for clean classifier by creating an adversarial patch). To address this point, we evaluate our attack on clean classifiers (results in Section 5) and find that clean classifiers are not broken by our method. Overall, our findings show that the secret backdoor is not required to manipulate poisoned classifiers, as suggested implicitly by previous work (Qiao al., 2019), thus highlighting the real vulnerability of poisoned classifier in practical scenarios.
4.3 DISCUSSION
The need for human interaction. It is important emphasize that the process we describe above requires human interaction as part of the approach: i.e., a human analyst needs to identify "suspicious" regions in the adversarial images and select them as potential alternative triggers. However, rather
5

---

Under review as a conference paper at ICLR 2022
Original
Basic Adv
Smoothing
Denoised Smoothing
Figure 5: Comparison of different forms of adversarial examples (c 20) from a binary poisoned
classifier on ImageNet.
than this being a downside of our approach, we emphasize that in fact we believe this to be a benefit. There are two main reasons for this. First, discussed briefly above, the likely practical use cases of identifying poisoned classifiers is quite different than that of identifying or avoiding traditional adversarial examples. Each votentially-poisnee company, which is unknown to be poisoned or not) requires substantial time investment to train and operate; thus, the additional time it will take an analyst to perform these kind of manual "forensic analysis" on a fully-trained classifier a relatively small time commitment (and, as our examples show, the onus on the person doing this analysis is small).
classifier (for instance, model built by third party
Given this factor, the second reason that the numan-in-the-loge nature of our process is beneficial is that human interaction is needed precisely due to the fundamental nature of adversarial examples. By definition, adversarial examples are perturbations that, to human, will not change the "true" label of an image, but will cause an algorithm to classify it differently. If we relied on automated procedures to select the "suspicious" elements in image, it would likely be possible to construct triggers that function as adversarial examples for these detectors, and thus evade detection. It is exactly (and, arguably only) by integrating human in the loop, which is entirely fcasible in the data- poisoning use case, that we can hope to avoid the possibility of adversarial attacks against a fully automated system.
Limitations of our method As we consider mainly backdoor attacks with patch- triggers, our method is limited in that it can not directly applied to more sophisticated backdoor attacks. In addition to patch based backdoor attacks, there are works that investigate other types of trigger: social-media filters (Sarkar et al., 2020), image warping (Nguyen & Tran, 2021), watermarks (Chen et al., 2021), image blending (Chen et al., 2017) and reflectance (Liu et al., 2020). These backdoor attack methods assume a different form of trigger. In these cases, trigger construction methods for poisoned classifiers should take into account the form of the triggers.
Since use numan-in-the-lonw procedure to extract new triggers, there is not exact algorithmic standard of whether a patch should be used for construction. As described in section 4.2, we select the patches which have dense distinctive color regions, which are most of the cases we encountered. There are also cases where such regions are hard to determine (see Figure 8a). It could be hard to justify the attack results report: we the authors may be biased in constructing backdoor triggers since we have much more expertise in dealing with edge cases. As an attempt to evaluate our procedure fairly, we conduct a user study on TrojAl datasets. Participants are not familiar with backdoor attacks but have basic knowledge of machine learning (details in section 5.2).
5 EXPERIMENTS
In this section, we present our attack results poisoned classifiers from two datasets: ImageNet (Rus- sakovsky ct al., 2015) and TrojAI (Majurski, 2020)?. For Denoised Smoothing, we use the MSE- trained ImageNet denoiser adopted from Salman et al. (2020). To make backdoor presence conspic- uous, we synthesize large-e untargeted adversarial examples (e = 20, 60). The noise level we use in smoothed classifiers is 1.00, as Kaur et al. (2019) shows that larger noise level leads to better visual results. We refer the reader to Appendix we construct alternative triggers of size 30 x 30, same as the size of the backdoor trigger used in ImageNet poisoned classifiers' We apply alternative triggers to random locations for ImageNet (same as the initial backdoor) and fixed place near the center for TrojAL. To evaluate the attack success rate, for ImageNet, we use 50 images for binary classifier and 200 images for multi-class classifier in the test set; for TrojAI dataset, we use the released 500 sample test images for each classifier.
for details on the experimental setup. For both datasets,
Dataset description in https: / /pwgas. ni st. gov/t rojjii / /ocs// data.. a. html. In TrojAl, the exact shape of backdoor trigger not provided. Here we adopt the same setting as ImageNet.

---

Under review as a conference paper at ICLR 2022
Clean
Adverserial (c 20)
Clean
Adversarial (c 20)
94.000
12.10
72.7%
66.97%
70.054
Clean
Adverserial le 60)
Clean
Adverserial (e 60)
GUSTRES
GUSCARE
65.3%%
40-9%
Figure 6: Results for attacking poisoned multi-class classifier obtained through BadNet (Gu et al., 2017). The attack success rate of the original backdoor Trigger A is 72.60%. The region which we use to construct alternative triggers is highlighted in red box.
BadNet
HTBA
CLBD
Binary
Multi-class
Binary
Multi-class
Binary
Multi-class
Ours
98.80 %
89.20%
99.80%
82.30%
93.80 %
67.90%
MESA
65.29%
43.60%
54.92%
50.29%
35.90%
40.05%
Random
20.39%
11.02%
14.93%
14.32%
8.23%
18.02%
Original 91.60%
72.60%
94.00%
74.55%
90.00%
58.95%
Table 1: Attack success rate of the triggers constructed using our method, MESA (Qiao et al., 2019), random cropped patches (Random) and the original trigger.
5.1 IMAGENET
We train both binary classifiers and 5-class classifiers with three backdoor attack methods: BadNet (Gu et al., 2017), HTBA (Saha et al., 2020) and CLBD (Turner et al., 2019). We adopt Trigger A in Figure as the default trigger. See Appendix for results on ImageNet classifiers with more classes.
Visualizing adversarial examples We compare Denoised Smoothing to two baseline approaches for generating adversarial examples: adversarial examples of: the poisoned classifier (denoted as Basic Adv); 2) the smoothed poisoned classifier without a denoiser (denoted as Smoothing). We generate adversarial examples (E 20) of the robustified binary poisoned classifier ImageNet, shown in Figure (More examples are shown in Figure in Appendix D.). First, we can see that our approach gives less noisy and smoother adversarial images than baselines. Second, there are some vague backdoor patterns Basic Adv, but backdoor patterns adversarial examples from Denoised Smoothing are more distinctive and easier to recognize. Last, Smoothing baseline does not produce any obvious pattern, which highlights the necessity of Denoised Smoothing.
Attack results on poisoned classifiers In Figure 6, we present sample alternative backdoor triggers we constructed by attacking a BadNet poisoned multi-class classifier ImageNet, where we show both color patch and cropped patch constructed from each adversarial example. For attack results other five ImageNet poisoned classifiers, we refer the reader to Figure and Figure 12 in Appendix C. From Figure 6, we can see that all the alternative triggers created from backdoor patterns have relatively high success rate. In particular, two triggers achieve significantly higher attack success rate (89.20%, 85.80 80%) than the original backdoor Trigger A (72.60%). Also notice that these alternative triggers differ greatly from Trigger A. Last, we can that whether color patch or cropped patch perform better depends on each example. In terms the effect of perturbation size, it can be seen that larger epsilon leads to better attack results.
We compare our method with two baselines: (1) a trigger distribution modeling method (Qiao et al., 2019), (2) randomly cropped patches from smoothed adversarial examples. Although Qiao et al. (2019) initially proposes MESA to model the distribution of triggers as a part of their defense method, here we use MESA to sample from this trigger distribution and compare with the alternative triggers discovered by our method. A summary attack results is shown Table 1, where we report the ASR of: triggers constructed by our method, triggers constructed by MESA (Qiao et al., 2019) and the original backdoor. It be seen that our attack finds much more effective triggers than the MESA baseline, and also the original trigger. For example, for the binary BadNet classifier, our method construct triggers with ASR 98.80 80% while the triggers constructed by MESA attains only
7

---

Under review as a conference paper at ICLR 2022
Clean
Adversarial (e
Clean
Adverserial 60)
100%
1.4%
2.5%
Figure 7: Results of applying our attack on an ImageNet clean classifier.
Clean
60
100
Trigger
Success rale: 75 80%
(a) Adversarial examples a robustified poisoned classifier with Trigger as the backdoor,
Clean
Advorsanial (c
60
Clean
Adversarisl (e
60)
Grooped
10
3
00%
83 .0%
(b) Attacking poisoned classificr with the "camouflaged" backdoor Trigger C (success rate 75.80%).
Figure 8: Analysis of poisoned classifier with "camouflaged" backdoor trigger.
65. 65.29%. Compared to randomly cropped patches, we can see that human interaction is helpful to finding effective triggers in our approach.
Attack results on clean classifiers
We show that clean classifiers are not broken under our attack. poisoned and there is such concept as attack success rate
Note that clean classifiers for clean classifiers. To measure the effect of the triggers constructed by our procedure on clean classifiers, report the error rate of clean classifiers when the test data is patched by the alternative triggers. Figure presents an illustration for attacking a clean multi-class ImageNet classifier with our approach. We refer the reader to Figure 13 in Appendix C for more results on attacking clean classifiers. Here we choose larger perturbation size 60 because we find no obvious pattern with perturbation size = 20. Observe that clean classifiers have low error rates with test data patched by constructed triggers, meaning that our attack does not apply to clean classifiers.
"Camouflaged" Backdoor So far we have experimented with triggers that contain colors (i.c., red, blue Trigger A) that are visually distinctive and result, backdoor patterns can be easily recognizable in adversarial examples. We study the case when backdoor trigger is less colorful or contains colors already in the color distribution of clean images. Consider Trigger C in Figure Sa: black and white colors in this trigger are also representative colors of panda. We train a poisoned binary classifier on ImageNet using Trigger C as the backdoor, where the backdoor attack method is BadNet (Gu et al., 2017). ). In Figure 8a, we visualize adversarial examples of the robustified poisoned classifier. Although there is clear backdoor pattern in the form of dense color regions, observe that in the generated adversarial examples, there is a tendency for black regions to have vertical horizontal boundaries, which resembles the pattern in Trigger C. Despite the absence of obvious backdoor patterns, we are still able to break the poisoned classifier using cropped patterns from large-é (e 100) adversarial examples shown in Figure 8b. Notice that both of the triggers are noisy and seem completely different from Trigger C, but they attain higher attack success rate (88.60% and 83.00%) than the original backdoor (75.80%).
5.2 TROJAI
dataset in TrojAI (Majurski, 2020) consists of a mixed set of clean and poisoned classifiers. We choose this dataset contains large of trained poisoned classifiers and also because it is benchmark for evaluating backdoor defenses. Different from ImageNet, we are not aware of the exact backdoor triggers used to poison the classifiers. TrojAl contains four image datasets: from round
8

---

Under review as a conference paper at ICLR 2022
Clean
Adversarial (c 20)
Trigger
ASR
round
98.76%
87.25%
round round 2
95.23%
Crasped Triggar
88.04%
round 3
82.20%
rate::
Table 2: Average ASR of our attack on poisoned classifiers from TrojAI datasets.
Figure Results of attacking poisoned classifier in TrojAI dataset.
round 3, with increasing complexity of backdoor attacks. In this work, experiment poisoned classifiers with Polygon based patch triggers. We exclude the case of filter based triggers in round and round 3 datasets as discussed in the limitations in section 4.3.
Attack results In Figure 9, we show attack results on a poisoned classifier sampled from TrojAI dataset. As shown in Figure 9, our methods can attack these poisoned classifiers with high success rate (See Figure 15 in Appendix C for results on more poisoned classifiers.). Similarly, the cropped trigger achieves higher success rate than the color trigger for both classifiers. For complete evaluation, we randomly sample 20 classifiers each from round 0 to 3 datasets (excluding filter based triggers) and apply our attack. The average attack success rate (ASR) is summarized in Table 2. We can see that the constructed triggers by our method have high ASR on poisoned classifiers.
Human study We conduct a user study on the TrojAI dataset (for simplicity, we choose TrojAl round O data). Participants are asked to analyze classifiers with our proposed method and decide if they poisoned. We develop an interactive tool implementing our method to aid the study. Two control groups are used: 1) variant of our method which adversarial examples of the original classifier (denoted as Basic Adv); 2) saliency maps on clean images (denoted as Saliency Map). In total, 15 participants from CS background with basic knowledge of machine learning take part in the human study and they are evenly divided into three groups. Details on the user study are in Appendix
Denoised Smoothing
Basic Adv
Random Patch
Denoised Smoothing
Basic Adv
Saliency Map
Clean
6%
5%
3%
Acc
89
68%
52%
Poisoned
97 %
80%
8%
Table Average accuracy in each group for identifying poisoned classifiers in the user study, Each group has participants.
Table We show the ASR of triggers constructed by participants, and the randomly cropped patches on clean/poisoned classifiers.
We show the average accuracy in each group for identifying poisoned classifiers Table 3. We can see that our method with Denoised Smoothing is much more helpful in identifying poisoned classifiers. We compute the ASR of the triggers constructed by the participants for the group Denoised Smoothing, Basic Ady and compare with randomly cropped patches from the smoothed adversarial examples. Results are in Table 4. Observe that participants in the group Denoised Smoothing construct much more effective triggers on poisoned classifiers. Randomly cropping patches fails to find effective backdoor triggers, showing the necessity and importance of human interaction. Overall, the human study suggests that our method are more interpretable and helpful for human users in identifying poisoned classifiers.
6 CONCLUSION
In this work we introduce new threat model of poisoned classifier where one would want to break without access to the original trigger. We propose a human-in-the -n-the-lohoop approach to attack poisoned classifiers in this threat model. We observe smoothed adversarial examples of a robustified poisoned classifier can contain backdoor patterns. Our attack procedure then constructs new alternative triggers with these backdoor patterns and we find that they give comparable or even better attack performance than the initial backdoor. We demonstrate that our attack is effective on high-resolution datasets, with a comparison to previous work on modelling trigger distribution. We end with user study showeasing the efficiency and interpretability of our approach to the wider audience. Our work highlight the vulnerability of poisoned classifier common users without access to the original trigger. From the promising results of our user study, we believe that future work for analyzing model robustness or image classifiers can benefit from human-in-ther -ooo--oop-o--opp approach.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Tom B. Brown, Dandelion Mane, Aurko Roy, Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv preprint arXiv: 171.:v:711..000 2.66666,, 2017. 5
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526, 2017. 1, 6
Xinyun Chen, Wenxiao Wang. Chris Bender, Yiming Ding, Ruoxi Jia, Bo Li, and Dawn Song. Refit: A unified watermark removal framework for deep learning systems with limited data. CCS, 2021.
Jeremy M Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via randomized smoothing. ICML, 2019. 3,3
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Alek- sander Madry. Adversarial robustness as a prior for learned representations. arXiv preprint arXiv:1906.000 00945, 2019. 3
Yansong Gao, Chang Xu, Derui Wang, Shiping Chen, Damith C. Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks deep neural networks. arXiv preprint arXiv:1902.065:: 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ICLR, 2015.
Tianyu Gu, Dolan-Gavitt Brendan, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXi:: 2017. 1, 2, 3, 4, 7, 8, 14, 21
Wenbo Guo, Lun Wang, Xinyu Xing, Min Du, and Dawn Song. Tabor: A highly accurate approach to inspecting and restoring trojan backdoors in systems. ICDM, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. CVPR, 2016. 12
Andrew Iyas,, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. NeurlPS, 2019. 2
Simran Kaur, Jeremy Cohen, and Zachary C. Lipton. Are perceptually-alignee gradients a general property of robust classifiers? arXiv preprint arXiv: 19 19.1199000844 2019. 3,
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in neural information processing systems, 2012. 12
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. arXiv preprint 1rXiv:1611.01236 2016. 12
Shaofeng Li, Minhui Xue, Benjamin Zi Hao Zhao Haojin Zhu, and Xinpeng Zhang. Invisible backdoor attacks on deep neural networks via steganography and regularization. arXiv preprint 17X.0:1909.00272,, 2019. 2
Yiming Li, Tongqing Zhai, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shutao Xia. Rethinking the trigger of backdoor attack. arXiv preprint arXiv:2004. 2e0e20404444, 2020.
Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack deep neural networks. ECCV, 2020. 6
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 1706006083 2017. 2, 3, 12
Michael Paul Majurski. Challenge round 0 (dry run) test dataset, 2020. URL https: / /data. n nist. govst/oovo//// wttpo////dd/ddddddoe/eeemaasm 2- 2,
10

---

Under review as a conference paper at ICLR 2022
Tulio Ribeiro Marco, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. KDD, 2016. 13
Anh Nguyen and Anh Tran. Wanet - imperceptible warping-based backdoor attack. ICLR, 2021. 6
Tuan Anh Nguyen and Anh Tran. Input-aware dynamic backdoor attack. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 3454-3464. Curran Asso- neurips cc/pppper/222//ELL.... pdf. 2
ciates, Inc., 2020. URL htttpa::/ppoceeeddieeeee 234e691320 oaa.aa622...aaaaeeeeeeeeaooee000
Vitali Petsiuk, Abir Das, and Saenko Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv: 2018. 13
Ximing Qiao, Yukun Yang, and Hai Li. Defending neural backdoors via generative distribution
modeling. Neurips, 2019. 2, 7
Selvarajk Ramprasaath, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localiza- tion. ICCV, 2017. 13
Olga Russakovsky, Deng Jia, Su Hao, Krause Jonathan, Satheesh Sanjeev, Ma Sean, Huang Zhiheng, Karpathy Andrej, Khosla Aditya, Michael Bernstein, Berg Alexander C., and Fei-Fei Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (IJCV)., 2015.
Aniruddha Saha, Akshayvarun Subraymanya, and Pirsiavash Hamed. Hidden trigger backdoor attacks. AAAI, 2020. 2, 3. 5, 7, 12, 14, 15
Hadi Salman, Greg Yang. Jerry Li, Pengchuan Zhang, Huan Zhang, Ilya Razenshteyn, and Sebastien Bubeck. Provably robust deep learning via adversarially trained smoothed classifiers. NeurIPS, 2019. 3, 4, 12
Hadi Salman, Mingjie Sun, Greg Yang, Ashish Kapoor, and J. Zico Kolter. Denoised smoothing: provable defense for pretrained classifiers. NewrlPS, 2020. 1, 2, 4,
Shibani Santurkar, Dimitris Tsipras, Brandon Tran, Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Image synthesis with a single (robust) classifier. NeurIPS, 2019.
Esha Sarkar, Hadjer Benkraouda, and Michail Maniatakos. Facehack: Triggering backdoored facial recognition systems using facial characteristics. arXiv preprint arXiv:2006. 2020. 6
Ezekiel Soremekun, Sakshi Udeshi, and Sudipta Chattopadhyay. Exposing backdoors in robust machine learning models. arXiv preprint arXiv:2003.00865 2020.
Christian Szegedy, Wojeiech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties neural networks. arXiv preprint arXiv:1312.6199, 2013. 2,
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. NeurIPS, 2018. 2
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. ICLR, 2019. 1, 3,4,
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Clean-label backdoor attacks, 2019, URL
net. forum? id HJg 6CCCKT. 3. 12, 14, 15
Bolun Wang. Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. IEEE Symposium on Security and Privacy, 2019. 2
Ren Wang, Gaoyuan Zhang, Sijia Liu, Pin-Ye Yu Chen, Jinjun Xiong, and Meng Wang. Practical detection of trojan neural networks: Data-limited and data-free cases. ECCV, 2020. 2

---

Under review as a conference paper at ICLR 2022
Cheng-Hsin Hiing Weng, Yan-Ting Lee, and Shan-Hung (Brandon) Wu. On the trade-off between adversar- ial and backdoor robustness. In H. Larochelle, M. Ranzato, Hadsell, M. F. Balcan, and Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 11973-11983, Curran Associates, Inc., 2020.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, 26 (7):3142-3155, 2017. 12
Appendices
A EXPERIMENTAL DETAILS
A.1 TRAINING DETAILS
We follow the experiment setting in HTBA (Saha et al., 2020), with publicly available code- eeeiiinHHiiiie /Hidden- Trigger-
AatedooorAttacke HTBA divides each class of ImageNet data into three sets: 200 images for generating poisoned data, 800 images for training the classifier and 100 images for testing. The trigger is applied to random locations on clean images. Poisoned datasets are first constructed with corresponding backdoor attack methods. Then we fine-tune the last fully-connected layer of pretrained AlexNet (Krizhevsky et al., 2012) on the created poisoned datasets. The fine-tuning process starts with initial learning rate of 0.001 decayed by 0.1 every 10 epochs and in total takes 10/30 epochs. The number of poisons are 400 images except for BadNet poisoned multi-class classifier, where we find that 1000 poisons are required to achieve high backdoor attack success rate.
base https: / /ggttubb
We implement the method of CLBD (Turner et al., 2019) utilizing adversarial examples on ImageNet. We find that training poisoned classifiers with CLBD is difficult ImageNet if we follow the exact steps described in Turner et al. (2019). We find that we are able to successfully train poisoned ResNets (He et al., 2016) by initializing the classifiers with adversarially robust classifiers that used to generate poisoned data in CLBD. We train adversarially robust classifiers for both binary classification and multi-class classification. For training binary poisoned classifiers, adversarial images with perturbation size = 32 in l2 norm as poisoned data. For training multi-class poisoned classifier, use 400 adversarial images with = 8 in lg norm as poisoned data.
400
A.2 COMPUTING ADVERSARTAL EXAMPLEE
In our attack, we need to compute adversarial examples of a smoothed classifier. To achieve this, we optimize the SMOOTHADV objective (Salman et al., 2019) with projected gradient de- scent (PGD) (Madry et al., 2017; Kurakin al., 2016). The code for attacking smoothed classifier is adopted from public available codebase https: http:://githhtee com/ Htw.Haadaallaaaa smoothing-adve rdarrsarial... Denoiser model is an ImageNet DnCNN (Zhang et al., 2017) denoiser trained with MSE loss, adopted from the public codebase of Denoised Smoothing in https:://githti
com/micet httmm//iccosottttooooo denooseooiieccoeee smoothing.
All adversarial examples are computed by untargeted adversarial attacks with a l2 norm bound e. We 16 Monte-Carlo noise vectors estimate gradients of smoothed classifiers. The number of PGD steps is 100. Step size at each iteration is (pertturbattion size e) / (# of steps). Except for attacking the poisoned classifier with "camouflaged" backdoor in Figure 8b, where we find that in this case, larger step size leads to slightly better visual results, thus we set step size to be 5 in Figure 8b.
B
USER STUDY
B.1 DETAILS ON USER STUDY
We describe our setup for the user study in detail. 15 people joined the study and are divided evenly into three groups We sample 50 classifiers randomly from TrojAI round 0 dataset. Participants
12

---

Under review as a conference paper at ICLR 2022
given the ground truth labels (poisoned/clean) and saliency maps for 10 baseline classifiers and then asked to mark 50 classifiers as either poisoned or clean. For the study group Denoised Smoothing and Basic Adv, we ask participants to apply our attack method with the interactive tool and test if the model can be successfully attacked by alternative triggers. If so, then mark the classifier as poisoned. We discuss the principle of using our tool to identify poisoned classifiers in section D.1. For the control group Saliency Map, we use RISE (Petsiuk et al., 2018) to generate saliency maps, as is shown to outperform other saliency map approaches (Ramprasaath et al., 2017; Marco et al., 2016).
B.2 TROJAI INTERACTIVE TOOL
In Figure 10, we show an overview of the interactive tool which implements our attack method. The first half of the tool, as shown in Figure 10a, allows users to visualize adversarial examples with chosen attack parameters. Below each image is the class that the adversarial image is predicted. Figure 10b presents the second half of the tool, where users can create new alternative patch triggers and see the classifier's prediction on patched poisoned images.
Dhooe Yor
Epdioo
roloo
Targeted
Cloon Imagt
Gonara Adt
Food
VURAR SD
Valve:
CLEAN EXAMPLE
EXAMPLE (DAS)
(a) First half of the interactive tool.
oripoed paRce
(b) Second half of the interactive tool.
Figure 10: Interface of interactive tool we develop for TrojAl dataset.
13

---

Under review as a conference paper at ICLR 2022
C ADDITIONAL ATTACK RESULTS
C.I
IMAGENET BINARY POISONED CLASSIFIER
Here show the complete results for attacking binary poisoned classifiers ImageNet in Figure 11. Notice that we find effective alternative triggers for all three poisoned classifiers.
Clean
Adversarial (e 20)
Gebrpaah
Clean
Adversarial
20
Cokor pasch
87 20%
64.02%
Chopoed
Grospind
Sucooss nat:
Suncesa FING
21.80%
Clean
Adversarial ie 60)
Clean
Adversarial le 60)
SUCCESS
64.00%
Cropped piich
Gropped
00%
$8.02%
(a) Results for attacking binary poisoned classifier obtained through BadNet (Gu et al., 2017).
The attack success rate of the original backdoor Trigger A is 91 91.60%.
Clean
Adversarial
Clean
Adversarial 20)
GACONER Groppedd
Cropped
40%
80.00%
Clean
Adverearial (e 60)
Color patch
Clean
Adversarial 60)
Coker puish
94 40%
57.00%
Cropoed
Coopaed panch
SIACONNR POWE
$6.00%.
(b) Results for attacking binary poisoned classifier obtained through HTBA (Saha al., 2020).
The attack success rate of the original backdoor Trigger is 94.00%.
Clean
Adversarial (e 60)
Cako parn
Clean
Adversarial (e 60)
Coko peich
62 66%
#7 A0%
Cropped patzh
Crapped palch
Succis 00.50%
73.89%
Clean
Adversarial
60
Clean
Adversarial 60)
SuXcess rull:
SucORsS FUMS
Sucoess
(c) Results for attacking binary poisoned classifier obtained through CLBD (Turner et 2019).
The attack rate of the original backdoor Trigger is 90.00%.
Figure 11: Results for attacking three binary poisoned classifiers obtained by three backdoor attacks.
14

---

Under review as a conference paper at ICLR 2022
C.2 IMAGENET MULTI-CLASS POISONED CLASSIFIER
In Figure 12, we present the results for attacking two poisoned multi-class classifiers on ImageNet obtained by HTBA (Saha et al., 2020) and CLBD (Tumer et al., 2019). We that attack constructs effective triggers in both cases.
Clean
Adversaria 20)
Coler patch
Clean
Adversarial
20)
Culce putch
Succoso
Sacoesa
BuCDess
72.85%
67.70%
Clean
Adversarial (e 60)
Coler paloh
Clean
Adversarial (e 60)
Color polch
Sucoess
Grospedd
SLecose
90%
74.95%
(a) Results for attacking a multi-clase poisoned classifiers obtained through HTBA (Saha et al., 2020).
The attack success rate of the original backdoor Trigger A is 74.55%.
Clean
Adversarial (e
60)
Color paich
Clean
Adversarial (c 60)
Cutor
Success rine
Sucoess rMM:
44 19%
48 90%
Oropped patch
Gropped patch
SUESMAN reter
Clean
Adversarial
patch
Clean
Adversarial (e 60)
80%
43
Crosped parch
Gropped
Succeso FOIC:
SUDDOSS FOX2:
67.90%
64.90%
(b) Results for attacking a binary poisoned classifiers obtained through CLBD (Turner et al., 2019). The attack success rate of the original backdoor Trigger is 58.95%.
Figure 12:
Results for attacking multi-class poisoned classifiers on ImageNet obtained by HTBA (Saha et al., 2020) and CLBD (Turner et al., 2019).
15

---

Under review as a conference paper at ICLR 2022
C.3 IMAGENET CLEAN CLASSIFIERS
In Figure 13 and Figure 14, we show the results of attacking binary and multi-class ImageNet classifiers. We can see that the clean classifier is not vulnerable to the triggers constructed by approach.
Clean
Adversarial (c 60)
Color sotch
Clean
Adversarial (E 60)
Color patch
Finor NEW
Finor min
1.00%
1.00%
Clean
Adversarial (e 60)
Color seich
Clean
Adversarial (c 60)
Coler pasch
Emnor mewe
Eme Gropcied pasoh
FARks 1.00%
Enror roler
1 100%
Figure 13: Results of applying our attack on an ImageNet clean classifier (binary).
Clean
Adversarial
Clean
Adversarial
60)
pakhh
1.00% Cropaed pasch
1 48%
Cropped patch
Erro stle
0 86%%
Clean
Adversarial 60)
Clean
Adversarial 60)
1.2% Cropsed patch
1.58% Drypsed putch
Figure 14: Results of applying our attack on an ImageNet clean classifier.
16

---

Under review as a conference paper at ICLR 2022
C.4 TROJAI
In Figure 15, we show results for attacking poisoned classifiers the TrojAI dataset. Note that for all 8 poisoned classifiers. the highest attack success rate attained among four alternative triggers is 100%. In Figure 16, we show the results of applying our attack method to two clean classifiers from ArojAl datasets. It can be seen that clean classifiers can classify more than half of the test images correctly even they are patched by the constructed triggers.
Clean
Adversarial (e 20)
Color Trigger
Clean
Adversarial (E = 20)
Color Triager
Success rate: 31.5%
Success rote: 56.70%
Crapped Trigger
Gropped Trigger
Success rate: 833
Suodess nate: 100%
(a) Poisoned Classifier
(b) Poisoned Classifier 3
Clean
Adversarial (e 20)
Color Trigger
Clean
Adversarial 20)
Color Triager
Groaped Trigger
Croeped Trigger
Success
97.50%
(c) Poisoned Classifier
(d) Poisoned Classifier 5
Clean
Adversarial 20)
Trigger
Clean
Adversarial
Triager
Suceess
Trigger
Gropped Trigger
Sucoesa rite: 98. 75%
Succed' rale: 100%
Poisoned Classifier
(f) Poisoned Classifier 7
Clean
Adversarial (e 20)
Color Trigger
Clean
Adversarial (E 20)
Trizger
Success rate: 73%
Success rate: 99. 50.50%
Cropped Trigger
Gropped Trigger
Succsiso rate:: 100%
Success rate: 59.75%
(h) Poisoned Classifier 9 Figure 15: Results of attacking 8 poisoned classifiers in the TrojAl dataset.
Poisoned Classifier 8
17

---

Under review as a conference paper at ICLR 2022
Clean
Adversarial (f 60)
Color patch
Clean
Adversarial (e = 60)
Color patch
Error rale: 25.000 28.00%
Error rate: 42.00%
Crapped patch
Crapped petch
DIS
Error rate::1..40000 21.40%
Error rate: 40.00%
(b) Clean Classifier 2 Figure 16: Results of attacking two clean classifiers in the TrojAl dataset.
(a) Clean Classifier
18

---

Under review as a conference paper at ICLR 2022
D ADDITIONAL VISUALIZATION RESULTS
D.1 ADVERSARIAL EXAMPLES ON TROIAI DATASET
Figure 17 presents the adversarial examples of robustified poisoned classifier from the TrojAl dataset, where each row shows images from one class. Below each image we show the class predicted by the poisoned classifier (not the smoothed classifier). We highlight those adversarial images with clear backdoor patterns. Note that they are all classified into class 2, which is indeed the target class of backdoor attack. While adversarial images from class 4 (the last row) have dense black regions, believe that this a result of mimicking features of class (the class that these images are predicted into) and it can be easily tested using our method that these black regions can not be used to construct successful triggers.
4
4
2
2
2
P
6
2
2
2
2
0
0
Figure Adversarial examples = 20 norm) robustified poisoned classifier in the TrojAl dataset. Below each image is the class predicted by the original poisoned classifier.
19

---

Under review as a conference paper at ICLR 2022
D.2 COMPARISON OF DIFFERENT ADVERSARIAL EXAMPLES
Figure 18 shows more results on comparing different adversarial examples (c = 20).
Original
Basic Adv
Smoothing
Denoised Smoothing
Figure 18: Comparison of different adversarial examples 20) of robustified binary poisoned classifier on ImageNet.
20

---

Under review as a conference paper at ICLR 2022
E IMAGENET CLASSIFIERS WITH MORE CLASSES
In this section, we evaluate our method on ImageNet classifier with more number of classes. We randomly select 10 classes from 1000 ImageNet classes. We then BadNet (Gu al., 2017) to train a poisoned classifier with Trigger A. Figure 19 shows the results for attacking this poisoned classifier. We can observe that these alternative triggers have similar or even higher attack success rate than the original trigger.
Clean
Adversarial (c 60)
Color palch
Clean
Adversarial (c 60)
Color patch
SuDoeea rate:
Succosss rale:
00 60.
70 78%
Cropped patch
Crapped patch
Suncees FM90: 56,80%
SucDoss ralo:
58 29%
Clean
Adversarial (e 60)
Color patch
Clean
Adversarial (e 60)
Color patch
QUICE
Supcess 1996:
Success
Cropped paich
Figure 19: Results of attacking a poisoned ImageNet classifier with classes. The success rate of the original backdoor is 59.77 79.71%.
21


Reviews:
[{"review_id": "haeQnkLSMhl", "reviewer": "Reviewer_67bv", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Thank you for sharing this idea with the research community. \n\n---\n\nPros:\n1. This paper showed a more efficient attacking methodology than previous work[1]. \n2. The authors implemented an algorithm that can generate two types of backdoor triggers without access to the original trigger and training set.\n3. The proposed method achieved a higher or similar attack success rate compared with the original backdoor trigger. \n\nCons:\n1. Since this paper highly depends on the empirical experiments, I left questions that are needed to be answered to improve the quality.  \n2. The proposed attack requires human interaction to generate new backdoors. The attacker should manually analyze adversarial examples generated by a robustified classifier to decide the new backdoor. \n\n---\n\nQuestions:\n1. In Sec. 4 threat model, do DL services return gradients? If not, please explain how smoothADV[2] gets a gradient.\n\n2. Is there another method instead of Denoised Smoothing[2] and random smoothing? The adversarial examples from the clean classifier in Fig.7 also have a distinctive color pattern. Is this pattern coming from Denoised Smoothing?\nIs other robust classifier against backdoors also having a distinctive pattern?\n\n3. Please specifically explain the methodology to find the poisoned class. (In Sec. 4.2)\n\n4. What is the relationship between small time commitment and the reason why human interaction cannot be automated? (In Sec. 4.3) \n\n5. Why is manual selection necessary for an attack procedure? In Fig. 2 and Fig.3, the color is distinctive. Why not use an algorithmic way to find a distinctive region(e.g., blob detection)? If considered, why are algorithmic methods not utilized?\n\n6. When making a color patch back door, is color alignment or sequence an important factor? For example, in Fig. 6, the sequence of color is not aligned with adversarial examples’ color sequence. Also, why are the color patches horizontally divided? \n\n7. If the distinctive region is multiple, how to choose?\n\n---\n\nMinor questions:\n1. In Fig2, why are color and locations different depending on the value of epsilon? This question leads to another question: how to choose the right epsilon? In the paper, larger is better for a distinctive pattern. Why is epsilon larger than 60 not tested?\n\n---\n\nPropositions:\n1. Is this attack method still valid against backdoor defense methods? This paper proposes a new threat of backdooring. There is no efficacy comparison against current defense methods. \n2. The title would need to be revised. “Fundamentally broken” may give the first impression that the backdoored models are fundamentally not working properly. However, the definition of a backdoored network is that the model is only poorly working when the trigger is presented. Therefore, backdoored networks are fundamentally working. \n3. Also, the title cannot be generalizable since the paper does not test sophisticated backdoored classifiers (Sec. 4.3). Therefore, the title needs to be specific.\n4. The experimental results for Denoised Smoothing are available in Sec.5.1. But these results are not mentioned in Sec.4.1. If mentioned, the reader may have more information.\n\n---\n\nMiscellaneous:\n1. It would be a typo. In contribution part, “(1) We consider a new thread…” would be “a new threat”.  In the conclusion, the authors mention “In this work, we introduce a new threat model…”. Therefore, I thought it would be a typo if not please ignore.\n2. In Sec. 5.1, “two triggers achieve a significantly higher attack success rate (89.20%, 85.80%)” => “attack success rate (85.80%, 89.20%)”. The upper image’s success rate is 85.80% and lower is 89.20%. Also, please specify the image. There are four sub images.\n\n---\n\nReference:\\\n[1] Ximing Qiao, Yukun Yang, and Hai Li. Defending neural backdoors via generative distribution modeling. Neurips, 2019\\\n[2] Hadi Salman, Greg Yang, Jerry Li, Pengchuan Zhang, Huan Zhang, Ilya Razenshteyn, and Sebastien Bubeck. Provably robust deep learning via adversarially trained smoothed classifiers. NeurIPS, 2019\n", "summary_of_the_paper": "This paper demonstrates a potential vulnerability of a backdoored deep network classifier. The authors showed that other possible backdoor triggers can be generated without access or knowledge about the training set and original trigger.  To generate new triggers, they proposed an algorithm that is based on ‘denoised smoothing’ and human interaction. Their empirical results support that backdoored models can have another backdoor trigger. It can be used to detect backdoored models since this methodology is only working for the backdoored models.", "summary_of_the_review": "The paper proposed a more efficient backdoor attack method without access to the training dataset and original trigger. However, this method highly depends on manual interaction. By answering questions in the above section, the authors can justify the necessity of human interaction. This justification is the important factor for changing the overall score. Also, please consider propositions and miscellaneous to clearly deliver the main idea.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Zy6ZXqnAtg", "reviewer": "Reviewer_9RGV", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The explanation for the underlying mechanism behind the statement in the title is still unclear to me and why this would apply to backdoored models but not clean models (which are also vulnerable to adversarial attacks including patch attacks and universal adversarial perturbations).\n\nThe white-box and human inspection requirements along with the requirement that backdoors be patch-based may limit the practical implications of this work along with its scientific use.  A number of recent attacks have shown that one can very effectively introduce backdoors under other less perceptible settings than patch-based.\n\nIf the authors want to claim that their method is useful for determining whether or not a model is poisoned, they should compare to a variety of state-of-the-art systems for this task.  As is, I am skeptical of the utility of this method, but this does seem like the only part of the paper which tackles a practical problem.", "summary_of_the_paper": "The authors observe that they can find highly effective backdoor triggers for poisoned classifiers which are not the ones that were originally introduced during poisoning.  To my knowledge, this is the first paper to make this observation.  In order to observe this, the authors propose a method for finding these additional triggers.", "summary_of_the_review": "As far as I can tell, this work makes an interesting observation, but it does not provide a very compelling explanation, and it is not clear to me how this observation or the related method provide practical utility.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "WPFuqK5hBE", "reviewer": "Reviewer_fLvg", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "I have several concerns as follows.\n\n- The denoised classifier will have a much smaller certified radius compared to the one training from scratch. Two ε-perturbations used in this paper seem to be relatively large. Is the method perform well under small ε adversarial examples?\n\n- If the original patch triggers are purely random noise or in irregular shapes e.g. the one in Fig. C, can the method construct the trigger with low ε adversarial-examples?\n\n- It seems that the triggers considered are basically in sharp contrast of colors. For backdoor triggers with single color e.g., the one in Fig 7, the error rates increase. Also, if the contrast between the trigger and the background is not evident, the success rates are very low as in Fig.15.\n\n- Are there any other general principles to tell whether a patch should be used for construction for irregular patch triggers?", "summary_of_the_paper": "This paper proposed a test-time, human-in-the-loop attack method to generate backdoor triggers under a specific threat model and argued that anyone with access to the classifier could reconstruct the triggers.\n\nSpecifically, the authors consider a threat model where the backdoor triggers (attack) are only allowed to be patches and the victim model can be only accessed through querying APIs at inference time (hence, denoising smoothing techniques are deployed).\n\nThe authors identify conspicuous patterns from adversarial examples (of the denoised classifier) with the aid of human judgment, and extract those patterns and modify them in the hope that they will match the original backdoor trigger patches.\n", "summary_of_the_review": "Overall, the paper is an empirical work with little justification for the proposed method. The empirical studies have limited settings. It is not clear when it works and when it fails.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "7Hozps1DtB", "reviewer": "Reviewer_1ehv", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### **Strengths**:\n\n— The idea of creating an attacker without access to the original trigger can create an alternative trigger is a novel threat model and can raise awareness in the community about the severity of the problem.\n\n— Authors show results on multiple benchmark datasets and show that alternative trigger can be created rather easily through a human study.\n\n### **Concerns**:\n\n— The Denoiser plays a crucial role in identifying the trigger which needs to be trained separately. It would also be interesting to see if adversarially trained classifiers (without the denoiser) also exhibit such patterns when poisoned. \n\n— Manual inspection is required for creating the alternative trigger. Authors mention the following argument, \" If we relied on automated procedures to select the suspicious elements in an image, it would likely be possible to construct triggers that function as adversarial examples for these detectors, and thus evade detection \". It is not clear as to why this is a disadvantage, more discussion on why the human-in-the-loop is absolutely required would be beneficial. The authors can also show experimentally that automating this procedure might not lead to good results.\n\n— Authors use an epsilon of 20/ 60 to create the adversarial example. It is unclear how this was chosen and how the performance is affected if it is varied in both extremes.\n\n— The proposed approach also requires lot more access to the poisoned classifier (from an attacker's perspective) which has to be combined with a denoiser and adversarial example needs to be created using gradient information from multiple monte-carlo samples. Whereas in previous approaches, the attacker could create the poisons and only the trigger when fooling the model. It would be interesting to see if this attack can be performed in a black-box fashion or show transferability of the approach across different classifiers.", "summary_of_the_paper": "The proposed approach shows that an attacker who does not have access to the original trigger or training data can still construct a new trigger and fool the classifier. A Denoised Smoothing classifier is used to create adversarial examples with high magnitude which reveal information regarding the trigger. Authors show that using human inspection, cropping such an anomalous pattern can still be an effective trigger to fool the classifier.  ", "summary_of_the_review": "The concept of creating alternate triggers that can still fool the model is an interesting idea and can enable researchers to defend against such attacks in a generic sense rather than focusing on a particular trigger. However, it would be more beneficial if the human-in-the-loop system can be relaxed or shown to be absolutely necessary to create such triggers. Authors are also encouraged to discuss possible defenses to reduce the effect of such poisons. An experiment that shows that such an attack can be performed without complete access to the classifier would also be interesting. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "5PP0kZFSMd", "reviewer": "Reviewer_8W5u", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The strength of the paper lies in the major finding that models that have been poisoned to include \"secret backdoors\" are, in fact, broken and vulnerable to other attacks to a much larger degree than naturally trained models. While it may not surprise every reader, the existing literature does not highlight this. Other works have proposed techniques to extract the trigger from the model and some clean test data. To my knowledge, this hasn't been used to make the point that you might extract a useful trigger very different from the one used during training. Additionally, there is technical novelty in using denoised smoothing to find an adversarially robust version of an already-trained (and poisoned) model with the express goal of finding adversarial examples that lead to the aforementioned triggers. Also the findings in Table 4 reveal from the human trials that these patches really do show up and are easy for humans to pick up -- that's cool!\n\nOne weakness of this work is that it is presented as a new threat model, and one that is not totally convincing. In this new threat model, there are several actors: (A) The outside party that trains a model; (B) The user who wants to use the classifier; (C) The attacker (person-in-the-middle) who might want to inject a backdoor of their own. From the introduction: \"In other words, adding a backdoor to a classifier does not just give the adversary control over the classifier, but also lets anyone control the classifier in the same manner.\" If I understand the threat model, the proposed method should be employed by C to infect the model trained by A -- but A had to also inject a backdoor or the method doesn't work (Table 4).  So the claim is that A did **even more** damage than intended, and I'm not sure this is such a problem. Would A care if their poisoned model was also poisoned in another way -- probably not if their trigger still works. Would B know that there is more than one backdoor trigger -- if there is no thought given or defense against poisoning in place, it seems not. Furthermore, if there is a defense in place, will this attack still work? This question was not addressed.\n\nI think this weakness needs to be addressed. I am sufficiently convinced that there is novelty in the method and the results show that other techniques would fail in this scenario. But I think the threat model is contrived and perhaps the authors could help me if I've misunderstood. If my understanding is accurate, perhaps the authors can make a stronger case for the importance of this method.\n\nMinor errors:\n\n(i) I would like more comparison of the extracted triggers to the original ones used when training the classifiers. For example, in the case of the color-based triggers, are they subsets of the color patch used (like Trigger A in Figure 4)? Do these extracted triggers ever move images into a different class than the original attack was intended to do?\n\n(ii) Some context is missing for quantities mentioned in the paper, specifically in the beginning of Section 5. The epsilon values are given without context. Specifically, the information that adversarial perturbations are made with epsilon values of 20 and 60 is there, but what scale is this on? I assume images are represented with pixel values in [0, 255] -- this should be made explicit. Similar clarification is needed for the value of \"the noise level\" -- is this a mean or variance of the Gaussian? And the patch size is given as 30x30, but what size are the images themselves?\n\nVery minor notes not affecting score:\n\n(a) \"that our method are more scalable...\" should probably be 'is more'.\n\n(b) \"(1) we consider a new thread model ...\" should probably be 'threat model'.\n\n(c) \"(2) we propose a interpretable...\" should probably be 'an interpretable'.\n\n(d) \"In reality, this third party could be anyone who are using deep learning...\" should probably be 'anyone who is using'.\n\n(e) Table 1 has vertical lines of confusing heights -- is there supposed to be a difference between the CLBD and the other two attacks?\n\n(f) Section 5.2 has the following sentence, which is very unclear: “For complete evaluation, we randomly sample 20 classifiers each from round 0 to 3 datasets (excluding filter based triggers) and apply our attack.“ I suppose, there are four datatsets and you samples 20 classifiers each, but it took me several tries to understand the wording that is there, I suggest revising the paragraph slightly.", "summary_of_the_paper": "In this paper a new method for poisoning classifiers is presented. By adding a denoised smoothing module to an already-trained-and-poisoned model, the attacker would be able through the crafting of adversarial attacks to find new backdoor triggers. Thorough experimentation, including human trials, show that the method is successful in settings where other baseline techniques essentially fail.", "summary_of_the_review": "The technical contribution of the method is interesting and the experiments are thorough and support the majority of the claims made in this paper. The threat model is curious at best and should be addressed -- stronger motivation that this attack would concern anyone would be good.\n\nI am initially giving this paper a borderline score. I think there are real strengths and some weaknesses that need to be addressed. I look forward to the author response.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors claim that backdoored classifiers are "fundamentally broken" by demonstrating that other backdoors can be generated for such classifiers without the knowledge of the original backdoors. The proposed method, however, requires manual intervention and is not justified by theoretical arguments. Numerous questions asked by the reviewers were not addressed in the rebuttal period.
Decision: Reject