Poisoned classifiers are not only backdoored, they are fundamentally broken
Authors: Mingjie Sun
Siddhant Agarwal
J Zico Kolter
Venue:  (2021)
Keywords: Backdoor Attacks
Denoised Smoothing
Perceptually-Aligned Gradients

Abstract: Under a commonly-studied “backdoor” poisoning attack against classification models, an attacker adds a small “trigger” to a subset of the training data, such that the presence of this trigger at test time causes the classifier to always predict some target class. It is often implicitly assumed that the poisoned classifier is vulnerable exclusively to the adversary who possesses the trigger. In this paper, we show empirically that this view of backdoored classifiers is fundamentally incorrect. We demonstrate that anyone with access to the classifier, even without access to any original training data or trigger, can construct several alternative triggers that are as effective or more so at eliciting the target class at test time. We construct these alternative triggers by first generating adversarial examples for a smoothed version of the classifier, created with a recent process called Denoised Smoothing, and then extracting colors or cropped portions of adversarial images. We demonstrate the effectiveness of our attack through extensive experiments on ImageNet and TrojAI datasets, including a user study which demonstrates that our method allows users to easily determine the existence of such backdoors in existing poisoned classifiers. Furthermore, we demonstrate that our alternative triggers can in fact look entirely different from the original trigger, highlighting that the backdoor actually learned by the classifier differs substantially from the trigger image itself. Thus, we argue that there is no such thing as a “secret” backdoor in poisoned classifiers: poisoning a classifier invites attacks not just by the party that possesses the trigger, but from anyone with access to the classifier.

Under review as a conference paper at ICLR 2021
POISONED CLASSIFIERS ARE NOT ONLY BACKDOORED, THEY ARE FUNDAMENTALLY BROKEN
Anonymous authors Paper under double-blind review
ABSTRACT
Under a commonly-studied "backdoor" poisoning attack against classification models, attacker adds small "trigger" to a subset of the training data, such that the presence of this trigger at test time causes the classifier to always predict some target class. It is often implicitly assumed that the poisoned classifier is vulnerable exclusively to the adversary who possesses the trigger. In this paper, we show empirically that this view of backdoored classifiers is fundamentally in- correct. We demonstrate that anyone with access the classifier, even without access to any original training data or trigger, can construct several alternative triggers that construct these alternative triggers by first generating adversarial examples for a smoothed version of the classifier, created with a recent process called Denoised Smoothing, and then extracting colors or cropped portions of adversarial images. We demonstrate the effectiveness of our attack through extensive experiments on ImageNet and TrojAl datasets, including user study which demonstrates that our method allows users to easily determine the existence of such backdoors in exist- ing poisoned classifiers. Furthermore, we demonstrate that our alternative triggers can in fact look entirely different from the original trigger, highlighting that the backdoor actually learned by the classifier differs substantially from the trigger image itself. Thus, we argue that there is no such thing as "secret" backdoor poisoned classifiers: poisoning a classifier invites attacks not just by the party that possesses the trigger, but from anyone with access to the classifier.
effective more at eliciting the target class at test time. We
INTRODUCTION
Backdoor attacks (Gu et al. 2017 Chen et al. 2017 Turner et al., 2019 Saha et al. 2020) have emerged as prominent strategy for poisoning classification models. An adversary, controlling (even a relatively small amount of) the training data can inject "trigger" into the training data such that inference time, the presence of this trigger always causes the classifier to make specific prediction while performance
the classifier on the clean data is not affected. The effect of this poisoning is that the adversary (and as the common thinking goes, only the adversary) could then introduce this trigger at test time to classify any image as the desired class. Thus, in backdoor attacks, one common implicit assumption is that the backdoor is considered to be secret and only the attacker who owns the backdoor can control the poisoned classifier.
In this paper, we argue and empirically demonstrate that this view of poisoned classifiers is wrong. Specifically, show that given access to the trained model only (without access to any of the training data itself nor the original trigger), one can reliably generate multiple alternative triggers that are as effective as more so than the original trigger. In other words, adding a backdoor to classifier does not just give the adversary control over the classifier, but also lets anyone control the classifier in the same manner.
Key our approach is how we construct these alternative triggers. An overview of our attack proce- dure is depicted in Figure I, The basic idea is to convert the poisoned classifier into an adversarially robust one and then analyze adversarial examples of the robustified classifier. The advantage of adversarially robust classifiers is that they have perceptually-aligned gradients (Tsipras al.. 2019). where adversarial examples of such models perceptually resemble other classes. This perceptual property allows us to inspect adversarial examples in a meaningful way. To convert poisoned clas- sifier into robust one, we use a recently proposed technique Denoised Smoothing (Salman et al., 2020), which applies randomized smoothing (Cohen et al. 2019) to pretrained classifier prepended

---

Under review as a conference paper at ICLR 2021
Generate Adversarial Examples
Construct Alternative Triggers
Robust Smoothed Classifier
Color
Denoised Smoothing
Original Backdoor
Poisoned
Denoiser
Classifier
Crop
CNN
Perceptually-Aliined Gradients
Similar/Higher Success Rate
Figure 1: Overview of attack. Given poisoned classifier, we construct robustified smoothed classifier using Denoised Smoothing (Salman et al. 2020). We then extract colors or cropped patches from adversarial examples of this robust smoothed classifier to construct novel triggers. These alternative triggers have similar or even higher attack success rate than the original backdoor.
with denoiser. We find that adversarial examples of this robust smoothed poisoned classifier con- tain backdoor patterns that can be easily extracted create alternative triggers. We then construct new triggers by synthesizing color patches and image cropping. Despite being generated from single test example. these alternative triggers turn to be effective across the entire test set and sometimes even exceed the attack performance of initial backdoor. Finally, we evaluate our attack on poisoned classifiers from datasets: ImageNet and TrojAl (Majurski) 2020)) datasets. We demonstrate that for several commonly-used backdoor poisoning methods, our attack consistently finds successful alternative triggers. We also conduct user study to showcase the generality of our approach for helping users identify these new triggers, improving substantially over traditional explainability methods and traditional adversarial attacks.
BACKGROUND
This work deals with the broad class of backdoor poisoning attacks, and brings to bear two threads of work in adversarial robustness to break poisoned classifiers: 1) the fact that robust classifiers have perceptually-aaigee
gradients (Tsipras al. 2019) (i.c.. that reveal information about the under- lying classes); 2) the use of randomized smoothing (Cohen et al. 2019) build robust classifiers, with recent work (Salman et al., 2020) showing that one can robustify pretrained classifier. We discuss each of these subjects turn. Then we clarify two points regarding our approach.
Backdoor Attacks In backdoor attacks (Chen et al., 2017 Gu et al.
2017 Li et al. 2019 2020), an
that at test time, clean images misclassifie into the target class when the trigger is present. BadNet (Gu et al.) ing subset of training data with the backdoor trigger and set the labels to the target class. One drawback of BadNet is that poisoned images are often clearly mislabeled, thus making the poisoned training data easily detected by human eyes simple data filtering (Turner et al. 2019). To address this issue. Clean-label backdoor attack (CLBD) (Turner al. 2019) and Hidden trigger backdoor attack (HTBA) (Saha et al.) 2020) propose poison generation methods which assign correct labels poisoned images. There also efforts to design defenses against backdoor attacks (Tran al.
adversary injects poisoned data into the training
2017 achieve this by modify-
2018
Wang et al.] 2019 Gao et al. 2019) Guo et al.) 2020) Wang et al.) 2020, Soremekun et al, Some of these defenses (Wang et al. 2019: Guo al. [2000] Wang et al. 20200 attempt to reconstruct the backdoor and require solving complicated custom-designed optimization problems.
2020
Soremekun et al. (2020) propose a method to detect poisoned classifiers if poisoned classifiers are also adversarially robust.
Aside from backdoor attacks, another major line of work in adversarial machine learning focuses adversarial robustness (Szegedy al., [203]]Gooofelloo Goodfellow et al. 2013)] Madry et al., 2017]; [lyas et al. 2019), which studies the existence of imperceptibly perturbed inputs that cause misclassification in state-of-the-art classifiers. The effort to defend against adversarial examples has led to building adversarially robust models (Madry et al. 2017). addition to being robust against adversarial examples, adversarially robust models are shown to have perceptually- aligned gradients (Tsipras al. [2019] Engstrom et al., 2019); adversarial examples of those clas- sifiers show salient characteristics of other classes. This property of adversarially robust classifiers can be used, for example, perform meaningful image manipulation (Santurkar et al. 2019).
Adversarial Robustness
2

---

Under review as a conference paper at ICLR 2021
Randomized Smoothing Our work is also related to recently proposed robust certification method: randomized smoothing (Cohen et al. 2019 Salman et al. (019). Cohen et al.] (2019) show that smoothing a classifier with Gaussian noise results in smoothed classifier that is certifiably ro- bust in l2 norm. Kaur al. (201)) demonstrate that perceptually-alignn smoothed classifiers. Although randomized smoothing is shown to be promising in robust certi- fication, it requires the underlying model to be custom trained, for example, with Gaussian data augmentation (Cohen et al. 2019) or adversarial training (Salman et al. (019). To avoid the tedious customized training, Salman al. (2020) propose Denoised Smoothing that converts standard classifier into a certifibly robust one without additional training. It achieves this by prepending a denoiser to a pretrained classifier prior to applying randomized smoothing.
gradients also occur for
On "defending against" versus "breaking" poisoned classifiers While our focus in this work is on "breaking backdoored classifiers". it might be tempting to instead view it as a "defense against backdoor attacks". However, we believe that the former is more accurate categorization due to the threat model of backdoor attacks. In a typical threat model associated with backdoor attacks, an attacker will introduce its poisoned data at training time, and the user then is free to perform what- ever analysis is needed upon the classifier in order to assess its vulnerability before deployment In other words, the attacker must "move first" in the game, and the user is free to "move second" to analyze the classifier; this is in stark contrast to test-time adversarial robustness, where defender must "move first"" to create robust classifier, and the attacker is then permitted to create adaptive adversarial inputs crafted toward that particular classifier. While it is certainly plausible that alter- native backdoor strategies may prove more difficult to analyze with our approach, the impetus here is on the attacker rather than the defender to demonstrate this possibility.
On our attack versus adversarial patch attack It may seem odd to claim that backdoored classi- fiers are "broken" by demonstrating their vulnerability to patch attack, especially given the well- known fact that virtually any (non-robust) classifier can be similarly attacked via an adversarial patch (Brown et al., (017). However, to large extent this is a matter of degree: while it's absolutely true that patch attacks exist for any classifier, our work here highlights just how easily an effective attack can be constructed against a backdoored classifier, precisely because such a classifier is trained to allow it. In contrast, our approach notably will not produce effective triggers against clean classifiers (See Figure iiin Section 4) while it would also be possible for an attacker to essentially interpolate between what qualified as a "backdoor trigger for poisoned classifier" and an "adversarial patch for a clean classifier", the point of this work is emphasize the degree to which backdoored classifiers make the task of breaking them easy and remarkably effective.
3 METHODOLOGY
In this section, we demonstrate our approach for attacking poisoned classifiers given access to the poisoned classifier and test data only. We consider the commonly-used threat model (Gu et al. 2017 Turner Turner et al.] 2019 Saha et al., 2020) for poisoned classifiers, where images patched with the backdoor will be predicted as target class. The attack success rate is defined as the percentage of test data (not including images from target class) classified into target class when the trigger is applied.
3.1 GENERATING SERCEPTUALLY-AIGNNE ADVERSARIAL EXAMPLES
We start by discussing the relationship between backdoor attacks and adversarial examples. Con- sider a poisoned classifier f where an image Ia from class a will be classified as class when the backdoor is present. Denote the application of the backdoor to image B(x). Then for poisoned classifier:
f(xa) = a, [(B(z.))= =
(1)
In addition being a poisoned image, B(xa) can be viewed as adversarial example of the poisoned classifier f. Formally, B(xa) is an adversarial example with perturbation size e = [B(x.)) - xellp in 4p norm:
B(xa) E {or f(x) + a, all xallp {}}
(2)
However, this does not necessarily mean that the backdoor will be present in the adversarial ex- amples of the poisoned classifier. This is because poisoned classifiers are themselves typically deep networks trained using traditional SGD, which are susceptible to small perturbations in the input (Szegedy et al. 2013). As result, loss gradients of such standard classifier are often noisy and meaningless to human perception (Tsipras et al. 2019
3

---

Under review as a conference paper at ICLR 2021
Poisoned Classifier (Trigger A)
Poisoned Classifier 2 (Trigger B)
Clean Classifier
Clean lmages
20
60
20
60
20
60
Figure 2: Visualization of some adversarial examples (c 20/60) from two robustified poisoned classifiers and a robustified clean classifier. Trigger A and Trigger B are shown in Figure 3
Perceptual property of adversarially robust classifiers Different from standard classifiers, adver- sarially robust models are robust to adversarial examples. Recent work (Tsipras et al. [2019; San- lurkar al. 2019) find that their loss gradients align well with human perception and adversarial examples of such models show salient characteristics of corresponding misclassified class.
We hope to use this perceptual property inspect and amalyze poisoned classifiers through the lens of adversarial examples. The difficulty is that poisoned classifiers are not adversarially robust by construction (Gu et al. 2017). We thus propose to use recent provable defense method Denoised Smoothing to convert the poisoned classifier into robust one.
Robustifying poisoned classifiers Denoised Smoothing (Salman et al. (020) is built upon random- ized smoothing (Cohen et al.) 2019), procedure that converts base classifier f into smoothed classifier under Gaussian noise that is certifiably robust in l2 norm:
g(z) = arg max + 5) = c) where ~((,0021
(3)
For randomized smoothing be effective, it usually requires the base classifier be trained via Gaussian data augmentation, which does not hold for poisoned classifiers. Denoised Smoothing is able to convert standard pretrained classifier into certifiably robust one. Denoised Smoothing first prepends a pretrained classifier with a custom-trained denoiser D. Then it applies randomized smoothing to the combined network f o D, resulting in a robust smoothed classifier f fsmoothed.
pamoothhed (x: arggram max PP( e D(r + 8)
where 5 ~ o((,002))
(4)
For a poisoned classifier, we use Denoised Smoothing to convert it into a robust smoothed classifier. We then generate perceptually meaningful adversarial examples of the smoothed classifier, using the method proposed in [Salman al.] (2019). Specifically, we use the SMOOTHADVPGD method and sample Monte-Carke noise vectors to estimate gradients of the smoothed classifier. Adversarial examples are generated with a norm bound Although randomized smoothing will ultimately add noise to an image with the backdoor present (xx), denoiser D will remove the noise before feeding it into the poisoned classifier. Therefore it is expected that backdoor of the poisoned classifier still applics to the new robust smoothed classifier. practice, we find that this holds true in general for the poisoned classifiers we consider.
3.2 BACKDOOR PATTERNS IN ADVERSARIAL EXAMPLES
Thus, our overall strategy to analyze the adversarial examples of robustified poisoned classifiers. Since we assume that users are not aware of the backdoor or which class is being targeted via the trigger, throughout this paper, unless otherwise specified, we will generate untargeted adversarial examples (though through these untargeted examples it will become obvious which is the poisoned class). To illustrate the basic approach, for the purpose of this presentation, we trained binary
4

---

Under review as a conference paper at ICLR 2021
Clean Image
Figure 4:
Backdoor patterns in adversarial examples (e 20) for robustified poisoned classifiers, where each poisoned model has a different color trigger.
poisoned classifiers on two ImageNet classes; pandas and airplanes; the target class of the backdoor is airplane. We used BadNet (Gu et al. (017)) for backdoor poisoning. After training, and without access to any training data, we then applied Denoised Smoothing to create a robust version of the
classifier.
In Figure we show l2 adversarial panda images (c 20, 60) of the robust version of two poisoned classifiers and clean classi- fie Two backdoor triggers are shown in Figure where Trig- ger is a 30 x 30 synthetic trigger with random colors, created in the backdoor attack method HTBA (Saha et al.) 2020) and Trigger B is a 30 x 30 hello kitty image. The crucial point here is that for adversarial examples of robustified poisoned classi- fiers, there are local color regions that are immediately visually apparent when inspecting the adversarial examples. For larger perturbation size (c 60), these colors become more saturated despite background noise. While for clean classifier, such regions are much less prevalent.
Trigger A
Trigger B
Figure 3: Backdoor triggers used in our analysis.
To better understand the relationship between these color regions and the backdoor, trained poisoned classifiers with backdoor triggers each consisting of a single, random colo Adversarial examples are shown in Figure 4| Similar to Figure 2| we still observe special color regions, and those colors are close to the color in the backdoor. This suggests that these local color spots can provide useful information (i.e., color) of the backdoor trigger. Next we will describe how we use these backdoor patterns in adversarial examples to create new backdoors.
3.3 BREAKING POISONED CLASSIFIESS
We now describe how to construct alternative triggers that perform just as well as the original one; this is largely manual process, but it typically straightforward in practice. Specifically, we the patterns observed in adversarial examples of robustified classifiers, and follow one of two strategies:
1. We synthesize a patch with colors obtained from the local regions with backdoor patterns. The color can be extracted by analysis of color histogram, but in this work, we use simple yet effective method: we manually choose a representative pixel.
2. We crop patch image that contains one of the backdoor patterns.
Note that both means constructing triggers require human inspection: first select the adversarial examples that contain backdoor pattern, then manually construct new triggers. However, the attack very straighforward because: 1) backdoor patterns are easy to spot, as shown in [Figure 2 2) pixel selection and cropping sub-images are very simple operations to perform manually. We apply these poison triggers to the poisoned classifier as if they are the true backdoor. Surprisingly, we find that although we create these triggers from only a handful of images, they generalize well to other images in the test set, attaining high attack success rate. Using the procedure described above (illustrated in Figuue I). we can easily break a poisoned classifier even if we do not know the original backdoor trigger.
Since our attack depends on observed backdoor features in adversarial examples, one could argue that this is caused by the transfcrability of adversarial patches (Brown et al. 2017). which could be general property of all classifiers (i.e., our attack may also work to create an adversarial patch against
Herr we show adversarial examples with clear backdoor patterns. For binary poisoned classifiers we investigate, we observe that most of the adversarial examples contain such backdoor patterns. FFor some colors, classifiers are hard poison (i.e., white and black). We choose those colors that lead high poisoning success rate 50%).
5

---

Under review as a conference paper at ICLR 2021
clean classifiers). To address this point, we also evaluate attack clean classifiers (Results shown in Section 4 and find that clean classifiers are not broken by our method. Overall, our results prompt us to rethink backdoor poisoned classifiers. Do backdoored classifiers really require the be controlledmmanipulatdd Our findings show that this is not the case. Not
secret backdoor only can backdoor patterns be leaked through adversarial examples, we can also construct multiple triggers attack poisoned classifiers that just as effective the original trigger.
3.4 ENHANCED VISUALIZATION TECHNIQUES
Finally, we discuss two additional techniques to help with visualizing adversarial examples.
Deep Dream We adopt the idea from Deep Dream (Mordvintsev al. 2013) by iteratively optimizing a certain objective starting with the resized output from previous iteration. Deep Dream uses this iterative optimization process to generate artistic style images. In our case, we iteratively optimize the adversarial ob- jective, so that backdoor patterns formed at earlier stages can be incorporated into those forming at later stages.
Figure Sample adversarial im- ages generated with deep dream and Tikhonov regularization.
Tikhonov Regularization Since we are generating large-c ad- versarial examples, adversarial images tend to become noisy. To reduce background noise, we introduce Tikhonov regulariza- tion (Tikhonov et al., [192), which minimizes loss function defined as Iz-regularization of the magnitude of image gradients (directional change in the intensity of colors).
we show sample adversarial images obtained with two techniques on top of Denoised Smoothing for the binary poisoned classifier with Trigger A. Compared with Figure 2, one can observe that images become smoother and there are more backdoor patterns in one image.
In Figure 5]
4 EXPERIMENTS
In this section, we present our attack results on poisoned classifiers from two datasets: Ima- geNet (Russakovsky et al. 2015) and TrojAI datasets (Majurski, 2020). For Denoised Smoothing, we use the MSE-trained ImageNet denoiser adopted from Salman et al. (2020). To make backdoor presence conspicuous, we synthesize large-c untargeted adversarial examples (e 20, 60). The noise level we use in smoothed classifiers is 1.00, as Kaur et al. (2019) shows that larger noise level leads to better visual results. We refer the reader to Appendix A for details on the experimental setup. For both datasets, construct alternative triggers of size 30 30, same the size of the backdoor trigger used in ImageNet poisoned classifiers] We apply alternative triggers to random locations for ImageNet (same the initial backdoor) and fixed place near the center for TrojAI F For computing the attack success rate of backdoor triggers, on ImageNet, we use 50 images for binary classifier and 200 images for multi-class classifier in the test set; on TrojAl dataset, we use the released 500 sample test images for each classifier.
4.1 IMAGENET
For ImageNet, we train both binary and multi-class poisoned classifiers with three backdoor attack methods: BadNet (Gu et al. 2017 ) Hidden trigger backdoor attack (HTBA) (Saha et al. 2020) and Clean-label backdoor attack (CLBD) (Trnerr et al. 2019) (in total 6 poisoned classifiers). The class "panda" vs "airplane"" For the multi-class classifier, 5 classes
of the binary classifier is are chosen randomly. Since only HTBA has conducted evaluation on ImageNet, we follow its setup for training poisoned classifiers. Specifically, we adopt Trigger A in Figure 3 as the default trigger and use AlexNet (Krizhevsky al. 2012) architecture
Comparison to baselines We compare Denoised Smoothing to two baseline approaches for gen- erating adversarial examples: adversarial examples of 1) the poisoned classifier (denoted "Basic
In AI, the exact shape of backdoor trigger is not provided. Here we adopt the same setting ImageNet. *For TrojAI, we are not aware where the trigger is applied the training process of poisoned classifiers. We choose this location in order for the alternative triggers to be applied at the foreground object (an artificial sign). (Sample images in httpss:/ / htppapa..nnt.. .nist. wtv.sttooo.....oooooooooooo..o ttw/totttttooooo rroja /ocs// data. html)
Except for CLBD, use ResNet (He 2016) for the backdoor attack to he successful.

---

Under review as a conference paper at ICLR 2021
Original
Basic Adv
Smoothing
Denoised Smoothing
Figure 6: Comparison of different forms of adversarial examples (c 20) from binary poisoned classifier on ImageNet.
Clean
Adversarial (c 20
Color patch
Clean
Adversarial (e 20)
Color pench
Succoss nEw 50%
72.70%
90%
60%
Clean
Adversarial 60)
Coler paich
Clean
Adversarial (e 60)
Color patch
SucoRsS rOW::
Success mon
80%
Croppec
65.4%%
20.20%
Figure 7: Results for attacking robustified poisoned multi-class classifier obtained through Bad- Net (Gu al., 2017). The attack success rate of the original backdoor Trigger A is 72.60% The region which we use to construct alternative triggers highlighted in a red box.
Adv"); 2) the smoothed poisoned classifier without a denoiser (denoted as "Smmoothig""). We gener- ate adversarial examples (e 20) of the robustified binary poisoned classifier ImageNet, visual- ized in Figure (More examples are shown in Figure 17 Appendix(C) First, we can see that our approach gives less noisy and smoother adversarial images than two baselines. Second, observe that there some vague backdoor pattern in "Basic Adv", but backdoor patterns adversarial examples from Denoised Smoothing are more distinctive and easier to recognize. Last, "Smoothing" baseline does produce any obvious pattern, which highlights the necessity of Denoised Smoothing.
BadNet
HTBA
CLBD
Binary
98.80% 1.6%%
99.80 19.80%/94.0045
93.80% 30.09%
Multi-class
89.20%
82.30%
67.9 67.90%
Table 1: Overall performance of our attack. For "X/Y", X is the highest attack success rate among the triggers that demonstrate in this paper and Y is the success rate of the original backdoor.
we present sample alternative backdoor triggers we con- structed by attacking BadNet poisoned multi-class classifier ImageNet, where we show both color patch and cropped patch constructed from each adversarial example. For attack results on other five ImageNet poisoned classifiers,
Breaking poisoned classifiers In Figure 7
refer the reader to Figure I and Figure 12 in Appendix B From Figure 7 we can see that all the alternative triggers created from backdoor patterns all have relatively high success rate. In particular, two triggers achieve significantly higher attack success rate (89. 20%, 85.80%) than the original backdoor Trigger A (72.60%). Also notice that these alter- native triggers differ greatly from Trigger A. Last, we can see that whether color patch or cropped patch perform better depends each example. In terms of epsilon, it can be that larger epsilon leads to better attack results. summary of attack results for all poisoned classifiers is presented in Table T, For each poisoned classifier, we compare the highest success rate achieved by the alter- native triggers we demonstrate in the paper and success rate of the initial backdoor (Trigger A). For all six poisoned classifiers we investigate, our attack finds an alternative trigger more effective than the original backdoor. Also. for five of the six poisoned classifiers, the highest success rate shown in Table is attained by cropped patch, which may suggest that cropped patch may be more effective overall.

---

Under review as a conference paper at ICLR 2021
Adversarial
Color pakch
Adversarial
pwizh
1.48%
Grospedi patch
Crappedd pards
Emai FoOe
Frior 19M
0.00%
Adversarial
Clean
Adversarial
1.24% Cropped patch
1.58%
Chopped parch
PAW
Figure 8: Results of applying attack ImageNet clean classifier.
Clean
20
60
100
Trigger C
Success 75.80%
(a) Adversarial examples of robustified poisoned classifier with Trigger C the backdoor.
Clean
Adversarial 60)
Clean
Adversarial le 60)
Cropped
Cropped potch
10
Success 88.60%
83.00%
(b) Attacking poisoned classifier with the "camouflaged" backdoor Trigger C (success rate 75.80%).
Figure 9: Analysis poisoned classifier with "cammoufaged" backdoor trigger.
Clean classifiers not easily broken. In addition, show that clean classifiers not broken under our approach. We apply our attack method to ImageNet clean classifiers. However, clean classifiers are not poisoned, then there is no such concept as attack success rate for clean classifiers. To measure the effect the triggers constructed by our procedure on clean classifiers, we report the eтror rate of clean classifiers when the test data is patched by the alternative triggers. [Figure 8 presents an illustration for attacking a clean multi-class ImageNet classifier. We refer the reader to Figure 13] in Appendix B for results on attacking the binary ImageNet classifier. Here we choose find obvious pattern with perturbation size 20.
larger perturbation size 60 because Observe that clean classifiers have low сиrоr rates under test data patched by the alternative triggers. Therefore, clean classifiers are not easily attacked by approach.
"Camouflaged" Backdoor So far we have experimented with triggers that contain colors (i.e., red, blue in Trigger A) that are visually distinctive and as a result, backdoor patterns can be easily recognizable in adversarial examples. We study the when backdoor trigger is less colorful or contains colors already in the color distribution of clean images. Consider Trigger C in Figure 9a black and white colors in this trigger also representative colors of panda. We train poisoned binary classifier on ImageNet using Trigger C as the backdoor, where the backdoor attack method is BadNet (Gu et al. 2017). In Figure 9a we visualize adversarial examples of the robustified poisoned classifier. Although there is no clear backdoor pattern the form of dense color regions, we can observe that in the generated adversarial examples, there is tendency for black regions to have vertical or horizontal boundaries, which resembles the pattern in Trigger C. Despite the absence of obvious backdoor patterns, we are still able to break the poisoned classifier using cropped patterns from large-e (e 100) adversarial examples shown in Figurd 9h Notice that both of the triggers
8

---

Under review as a conference paper at ICLR 2021
Adversarial
Ooler Trgger
Adversarial 20)
Coar Triggrr
Buesess rate: 67 20%
Bucoess réla: 50 25%
Gropped Trigger
Grapped Trigger
Suucsess rale:: 100%
Success role:: 100%
(a) Poisoned Classifier 1
(b) Poisoned Classifier 2
Figure 10: Results of attacking two poisoned classifiers in TrojAl dataset.
Denoised Smoothing
Basic Adv
Saliency Map
Participants
participant
participant 2
participant 3 participant 4
participant 5
Accuracy
94%
90%
66%
82%
54%
Table 2:
Accuracies that participants obtained for identifying poisoned classifiers in the user study.
are noisy and seem completely different from Trigger C, but they attain higher attack success rate (88.60% and 83.00%) than the original backdoor (75.80%).
4.2 TROJAI DATASET
We evaluate attack the Aroj dataset Majurski 2020). consisting of mixed set of clean and poisoned classifiers. TrojAI dataset is initially proposed as a dataset to help develop backdoor detection methods. Here we choose this dataset as it contains a large set of trained poisoned classi- fiers. Different from ImageNet, we are not aware the exact backdoor triggers used to poison the classifiers. In FFiurr I0] we show attack results on two poisoned classifiers. Poisoned classifiers are chosen from Round 0 of the TrojAI dataset. As shown in Figure 10, our methods can attack these poisoned classifiers with high success rate (See Figure 14 14 in Appendix B for results on more poi- soned classifiers.). Similarly, the cropped trigger achieves higher success rate than the color trigger for both classifiers. Especially, notice that both cropped triggers attains 100% attack success rate.
Finally, we conduct a user study on the TrojAl dataset to test the generality of our approach. We develop an interactive tool implementing our method aid the study. Participants are asked to analyze classifiers with the tool and decide if they are poisoned. Two control groups are used: 1) participants are given a variant of the tool using adversarial examples of the original classifier (denoted as "Basic Adv"); ; 2) participants are given saliency maps on clean images (denoted as "Saliency Map"). Details on the user study and the interactive tool are in Appendix [ Results summarized in Table 2] where we show the accuracies of identifying poisoned classifiers for three approaches. Overall, the study suggests that analysts with access to our tool are able to substantially outperform those using alternative methods.
CONCLUSION
This work shows that backdoor attacks create poisoned classifiers that can be easily attacked even without knowledge of the original backdoor. We find that adversarial examples of a robustified poisoned classifier usually contain backdoor patterns. We then construct new poison triggers from the backdoor presence in adversarial examples and show that they give comparable even better attack performance than the original backdoor.
Our findings urge the research community to rethink the current threat model in backdoor poisoning. remains be seen if there exist backdoor attacks that avoid our attack. Our results also raise question of what is actually learned through the backdoor poisoning process. It seems that backdoor poisoning creates a spectrum potential backdoors, in addition to the original one. Thus, rigorous analysis of the implicit effect of backdoor poisoning is needed. More broadly, the idea of robustifying (poisoned) classifiers be principled approach for analyzing general standard classifiers.
REFERENCES
Tom B. Brown, Dandelion Mane, Aurko Roy. Martín Abadi, and Justin Gilmer. Adversarial patch. arXiv preprint arXiv: arXiv: 171.00965, 2017.

---

Under review as a conference paper at ICLR 2021
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprini 1712.:/71,.05526, 2017.
Jeremy M Cohen, Elan Rosenfeld, and J. Zico Kolter. Certified adversarial robustness via random- ized smoothing. ICML, 2019.
Logan Engstrom, Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Brandon Tran, and Alek- sander Madry. Adversarial robustness as a prior for learned representations. arXiv preprint arXiv:1906.00990 2019.
Yansong Gao, Chang Xu, Derui Wang, Shiping Chen, Damith C. Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks, arXiv preprint arXiv: arXiv:1902.006 1902.0631, 2019.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. ICLR, 2015.
Tianyu Gu, Dolan-Gavils Brendan, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint vrXiv:1708.0673: 2017.
Wenbo Guo, Lun Wang. Xinyu Xing, Min Du, and Dawn Song. Tabor: A highly accurate approach to inspecting and restoring trojan backdoors in ai systems. ICDM, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. CVPR, 2016.
Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran, and Aleksander Madry. Adversarial examples are not bugs, they are features. NeurlPS, 2019.
Simran Kaur, Jeremy Cohen, and Zachary C. Lipton. Are perceptually--aigned gradients general 2019.
property of robust classifiers? arXiv preprint
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Advances in neural information processing systems, 2012.
Alexey Kurakin, lan Goodfellow, and Samy Bengio. Adversarial machine learning scale. arXiv preprint
2016.
Shaofeng Li, Minhui Xue, Benjamin Zi Hao Zhao, Haojin Zhu, and Xinpeng Zhang, Invisible backdoor attacks on deep neural networks via steganography and regularization. arXiv preprint 199.072909 2019.
Yiming Li, Tongqing Zhai, Baoyuan Wu, Yong Jiang. Zhifeng Li, and Shutao Xia. Rethinking the trigger of backdoor attack. arXiv preprint arXiv:2004. 04692, 2020.
Alcksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv: 176600676 2017.
Michael Paul Majurski, Challenge round 0 (dry run) test dataset, 2020. URL https: 2175
n nist gow /tto///////o
Tulio Ribeiro Marco, Sameer Singh, and Carlos Guestrin. Why should trust you?: Explaining the predictions of any classifier. KDD, 2016.
Alexander Mordvintsev, Christopher Olah, and Mike Tyka. Inceptionism: deeper into neural networks, 2015. URL https://ai http://gggggggooggoooooo////////
Going
Vitali Petsiuk, Abir Das, and Saenko Saenko. Rise: Randomized input sampling for explanation of black-box models. arXiv preprint arXiv:/806.074/ 1806.0741,, 2018.
Selvarajk Ramprasaath, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruy Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. ICCV, 2017.
10

---

Under review as a conference paper at ICLR 2021
Leonid Rudin, Stanley Osher, and Emad Fatemi. Nonlinear total variation based noise removal algorithms. Physica D: Nonlinear Phenomena, 1992.
Olga Russakovsky, Deng Jia, Su Hao, Krause Jonathan, Satheesh Sanjeev, Ma Sean, Huang Zhiheng, Karpathy Andrej, Khosla Aditya, Michael Bernstein, Berg Alexander C., and Fei-Fei Li. Imagenet large scale visual recognition challenge. International Journal of Computer Vision (ICCV), 2015.
Aniruddha Saha, Akshayvarun Subraymanya, and Pirsiavash Hamed. Hidden trigger backdoor tacks. AAAI, 2020.
Hadi Salman, Greg Yang, Jerry Li, Pengchuan Zhang, Huan Zhang, llya Razenshteyn, and Sebastien Bubeck. Provably robust deep learning via adversarially trained smoothed classifiers. NeurIPS, 2019.
Hadi Salman, Mingjie Sun, Greg Yang. Ashish Kapoor, and J. Zico Kolter. Denoised smoothing: A provable defense for pretrained classifiers. NeurlPS, 2020.
Shibani Santurkar, Dimitris Tsipras, Brandon Tran, Andrew Ilyas, Logan Engstrom, and Aleksander Madry. Image synthesis with a single (robust) classifier. NeurIPS, 2019.
Ezekiel Soremekun, Sakshi Udeshi, and Sudipta Chattopadhyay, Exposing backdoors in robust machine learning models. arXiv preprint arXiv:2003.0088 2020.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199, 2013.
A. N. Tikhonov, S. Leonov, and G. Yagola. Nonlinear ill-posed problems. World Congress of Nonlinear Analysts, 1992.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. NeurIPS, 2018.
Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. Robustness may be at odds with accuracy. ICLR, 2019.
Alexander Turnrr, Dimitris Tsipras, and Aleksander Madry. Clean-label backdoor attacks, 2019. URL httphhttps///ppnreriieee http://wwwweeeeeooooeoeeeeeeoeee (oorummmid-H..6622CCee
net.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. IEEE Symposium Security and Privacy, 2019.
Ren Wang, Gaoyuan Zhang, Sijia Liu, Pin-Yu Chen, Jinjun Xiong, and Meng Wang. Practical detection of trojan neural networks: Data-limited and data-free cases. ECCV, 2020.
Kai Zhang, Wangmeng Zuo, Yunjin Chen, Deyu Meng, and Lei Zhang. Beyond a Gaussian denoiser: Residual learning of deep CNN for image denoising. IEEE Transactions on Image Processing, 26(7):314233155 2017.

---

Under review as a conference paper at ICLR 2021
Appendices
A EXPERIMENTAL DETAILS
A.1 TRAINING DETAILS
We follow the experiment setting in HTBA (aha et al. 2020 with publicly available code- w.om/MBCCissi... iininno//eee Hidden Trigger Backdoor-Atted HTBA divides each class of ImageNet data into three sets: 200 images for generating poisoned data, 800 images for training the classifier and 100 images for testing. The trigger applied random locations on clean images. Poisoned datasets are first constructed with corresponding backdoor attack methods. Then we fine-tune the last fully-connected layer of pretrained AlexNet (Krizhevsky eet al.] 2012) on the created poisoned datasets. The fine-tuning process starts with initial learning rate of 0.001 decayed by 0.1 every 10 epochs and in total takes 10/30 epochs. The number of poisons are 400 images except for BadNet poisoned multi-class classifier, where we find that 1000 poisons are required to achieve high backdoor attack success rate.
base https:: /
We implement the method of CLBD (Turner et al. 2019) utilizing adversarial examples on Ima- geNet. We find that training poisoned classifiers with CLBD is difficult on ImageNet if we follow the exact steps described in Turner al. (019). We find that
able to successfully train poi- soned ResNets (He et al., 2016) by initializing the classifiers with adversarially robust classifiers that are used to generate poisoned data in CLBD. We train adversarially robust classifiers for both binary classification and multi-class classification. For training binary poisoned classifiers, we use 400 adversarial images with perturbation size = 32 in lz norm as poisoned data. For training multi-class poisoned classifier, we use 400 adversarial images with c l2 norm as poisoned data.
A.2 COMPUTING ADVERSARIAL EXAMPLE
In attack, need to compute adversarial examples of smoothed classifier. To achieve this, we optimize the SMOOTHADV objective (Salman et al. 2019) with projected gradient de- scent (PGD) (Madry al. 2017 Kurakin al.] 2016). The code for attacking smoothed classifier is adopted from public available codebase https: /ttp//gittubbbb com/ aladisalmat. smoothing adversara sariialll Denoiser model is ImageNet DnCNN (Zhang et al.) (017) de- noiser trained with MSE loss, adopted from the public codebase of Denoised Smoothing in https:
//github. com/ mi croso ft / deno sed-
All adversarial examples are computed by untargeted adversarial attacks with l2 norm bound e. We use 16 Monte-Carte noise vectors to estimate gradients of smoothed classifiers. The number of PGD steps is 100. Step size at each iteration is 2 (ppetuurbatiion size / (# of steps). Except for attacking the poisoned classifier with "camouflaged" backdoor in Figure 9b. where we find that in this case, larger step size leads to slightly better visual results, thus we set step size to be 5 in Figure 9b)
Deep Dream We optimize the adversarial objective with Deep Dream framework adopting the implementation from public codebase https: /ttps://githun com/ www.eiideeeerrrreee.ee DeTorch-Deeg--reeee
We perform iterations, scaling the image by 1.2 every iteration. Due to the large memory requirements of Deep Dream, we use 5 Monte-Carlo noise vectors to estimate gradients. At each iteration, we use 100 steps with step size 5.
Regularization We apply Tikhonov regularization to minimize the 1 norm of image gradients of adversarial perturbations. The image gradient is computed by the filter F in |Equation 5) We also experimented with another well-studied denoising objective Total Variation (TV) loss (Rudin al. 1992]. which minimizes the distance between neighboring pixels. TV loss can be seen as a special case of Tikhonov regularization with specific filter. Comparison of two regularization techniques is shown in Figure 19
2
2
-1
2
2
-1
-1
F =
(5)
-1
1
-1
0
-1
0
12

---

Under review as a conference paper at ICLR 2021
B ADDITIONAL ATTACK RESULTS
B.I IMAGENET BINARY POISONED CLASSIFIER
Here show the complete results for attacking binary poisoned classifiers ImageNet in ure 11. Notice that we find effective alternative triggers for all three poisoned classifiers.
Clean
Adversarial
Clean
Adversarial
200 Eirgpoee
64.00%
SANDOEs mate: 79 69%
$1
Clean
Adversarial
Clean
Adversarial
94 64.80%
94.80%
Cropsedd patch
Dropped panch
SEUDORIN
(a) Results for attacking rubustified binary poisoned classifier obtained through BadNet (Gu 2017 The attack success rate of the original backdoor Trigger A 91.60%.
Clean
Adversarial (c
20)
Color porch
Clean
Adversarial
20)
Coler path
SUCDOSS FOOD:
SUCCeSS IaC
73.45%
80.80%
Clean
Adversarial (c 60)
Gote parch
Clean
Adversarial (c 60)
Coker palch
Sucosss
57.00%
09.80%
96 96.80%
(b) Results for attacking robustified binary poisoned classifier obtained through HTBA 2020 The attack success rate of the original backdoor Trigger A 94.00%.
Saha et al.
Clean
Adversarial
Clean
Adversarial
60)
62 60%
87 60%
Cropped parch
Orapped patch
Succoss mine:
Clean
Adversarial
60)
Clean
Adversarial
60)
Bucosoa
Succoda
71 20%
6n 10%
Crapped partch
Cropped pazch
(c) Results attacking robustified binary poisoned classifier obtained through CLBD Turner 2019). attack success rate of the original hackdoor Trigger 90.00%. Figure Results for attacking three binary poisoned classifiers obtained by three backdoor attacks.
13

---

Under review as a conference paper at ICLR 2021
B.2 IMAGENET MULTI-CLASS POISONED CLASSIFIER
In Figure 12 we present the results for attacking two poisoned multi-class classifiers on ImageNet
obtained by HTBA (Saha al. 2020) and CLBD (Turner et al. 2019). We can see that our attack constructs effective triggers in both cases.
Clean
Adversarial
20)
Color pako
Clean
Adversarial
20)
Color potch
BUCCoSO
72.30%
67.00%
Adversarial 60)
palch
Adversarial 60)
Croopedi patch
SUICeOS F91M!
Success rater
67.00%
74.25%
a Results for attacking robustified multi-class poisoned classifiers obtained through HTBA (Shha 2020), The attack success rate of the original backdoor Trigger A is 74.55%.
ct al.
Clean
Adversarial (c
60)
Color patch
Clean
Adversarial (c 60) Gulte panch
SucoME rate: 90%
Suaoosss rale:
Coopped pancc
Orepoed patch
SUCDOSS
Clean
Adversarial 60)
Color
Clean
Adversarial (e 60)
1877
45 Cropueed patch
Crosped parch
Staopeee rate:
67.50 70.00%
64.90%
(b) Results for attacking a robustified binary poisoned classifiers obtained through CLBD (Turner
ct al.
2019
The attack success rate of the original backdoor Trigger A is 58.95%.
Figure
Results for attacking multi-class poisoned classifiers on ImageNet obtained by (Shaa et al. 2020) and CLBD (Turner et al. 2019).
12:
HTBA
14

---

Under review as a conference paper at ICLR 2021
B.3 IMAGENET BINARY CLEAN CLASSIFIER
In Figure 13 we show the results attacking a clean binary ImageNet classifier. We can see that the clean classifier is not vulnerable to the triggers constructed by our approach.
Clean
Adversarial
60)
Clean
Adversarial
60)
5.%% Grosped paich
1.10%
Croqued patch
mal
Enor IBC
Clsan
Adversarial 60)
Clean
Adversarial (c 60)
Color
Ener ral: 000
Enor skc
1.50%
Crogped paich
Figure 13: Results of applying our attack on ImageNet clean classifier (binary).
15

---

Under review as a conference paper at ICLR 2021
B.4 TROJAI
In Figure 14 we show results for attacking poisoned classifiers in the TrojAl dataset. Note that for all 8 poisoned classifiers, the highest attack success rate attained among four alternative triggers is 100%. In Figure 15 we show the results of applying our attack method to two clean classifiers from TrojAI datasets. It can be seen that clean classifiers can classify more than half of the test images correctly even they are patched by the constructed triggers.
Clean
Adversarial (c 20)
Color Trigger
Clean
Adversarial (c = 20)
Color Triager
A
Cropped Trigger
Croeped Trigger
(a) Poisoned Classifier
(b) Poisoned Classifier 4
Clean
Adversarial 20)
Oolor Trigger
Clean
Adversarial
Trigger
Success
Success 99.50%
Croaped Trigger
Grapped Trigger
Success
Success 97.00%
Poisoned Classifier
(d) Poisoned Classifier 6
Clean
Adversarial (e 20)
Color Trigger
Clean
Adversarial (E 20)
Trizger
Success rate: 91.73%
Success rate: 90.75%
Cropped
Gropped Trigger
Sucoass rate: 96 75%
Suuciess rale: 100%.
(e) Poisoned Classifier 7
(I) Poisoned Classifier 8
Clean
Adversarial (E 20)
Color Triggor
Clean
Adversarial (E = 20)
Color Triager
HIS
Sucoess rate: 79%7
B
Success rate: 99.50%
Cropped Trigger
Cropped Trigger
rede:
Success
(h) Poisoned Classifier 10 Figure 14: Results of attacking poisoned classifiers in the TrojAI dataset.
(g) Poisoned Classifier 9
Clean
Adversarial
60)
Adversarial (e
Cropped
petch
21.10%
(a) Clean Classifier
(b) Clean Classifier 2
Figure 15: Results of attacking two clean classifiers in the TrojAl dataset.
16

---

Under review as a conference paper at ICLR 2021
C ADDITIONAL VISUALIZATION RESULTS
C.I ADVERSARIAL EXAMPLES ON TROJAI DATASET
Figure 16 presents the adversarial examples of robustified poisoned classifier from the TrojAI dataset, where each row shows images from one class. Below each image we show the class pre- dicted by the poisoned classifier (not the smoothed classifier). We highlight those adversarial images with clear backdoor patterns. Note that they are all classified into class which is indeed the target class of backdoor attack. While adversarial images from class 4 (the last row) have dense black regions, believe that this a result of mimicking features of class (the class that these images are predicted into) and it can be easily tested using our method that these black regions can not be used to construct successful triggers.
4
4
4
4
2
2
2
2
1
2
2
2
Figure 16: Adversarial examples (e 20 in l2 norm) of robustified poisoned classifier in the TrojAl dataset. Below each image the class predicted by the original poisoned classifier.
17

---

Under review as a conference paper at ICLR 2021
C.2 COMPARISON OF DIFFERENT ADVERSARIAL EXAMPLES
Figure 17 shows more results on comparing different adversarial examples (c = 20).
Original
Basic Adv
Smoothing
Denoised Smoothing
Figure 17: Comparison of different adversarial examples (e 20) of robustified binary poisoned classifier on ImageNet.
18

---

Under review as a conference paper at ICLR 2021
C.3 ENHANCED VISUALIZATION TECHNIQUES
C.3.1 DEEP DREAM
Figure soows the comparison of adversarial images with or without enhanced visualization tech- niques discussed in subsection 3.4. We can that for Deep Dream, there are backdoor pat- terns a single adversarial image than Denoised Smoothing. Together with Tikhonov regularization method, the backdoor patterns become more stable and less noisy.
Original
Demoised Smoothing
Deep Dream
Deep Dream Reqularization
Figure 18: Effects of enhanced visualization techniques on adversarial examples of a robustified ImageNet binary poisoned classifier.
19

---

Under review as a conference paper at ICLR 2021
C.3.2 REGULARIZATION
In Figure 19, we show how regularization be used to reduce background noise in large- adver- sarial examples. We generate adversarial images with c = 60. For Denoised Smoothing, we see that there is some background noise. For both regularization techniques, we see that adversarial images are less distorted and there are less noise patterns.
Denoised Smoothing
Denoised Smoothing Tikhonov regularization
Original
Denoised Smoothing
TV Loss
Figure 19: Comparison of adversarial examples generated with/without regularization.
20

---

Under review as a conference paper at ICLR 2021
D USER STUDY
D.1 TROIAI INTERACTIVE TOOL
In Figure 20 we show brief overview of the interactive tool which implements attack method. The first half of the tool, as shown in Figurd 20a allows users to visualize adversarial examples with chosen attack parameters. Below each image the class that the adversarial image is predicted. Figund 20b presents the second half of the tool, where users can create new alternative patch triggers and see the classifier's prediction on patched poisoned images.
Image
largih
CLEAN EXAMPLE (OXD)
ADV EXAMPLE (5XB)
(a) First half of the interactive tool.
Seloctor
Prodit
PAEDICT
(b) Second half of the interactive tool.
Figure 20: Interface of interactive tool we develop for TrojAl dataset.
21

---

Under review as a conference paper at ICLR 2021
D.2 DETAILS ON USER STUDY
We describe our setup for user study in detail. people joined the study. We divide them into three groups: 2 people for Denoised Smoothing, 2 people for the control group "Basic Adv" and person for the control group "Saliency Map". For all three groups. participants are asked to mark 50 classifiers as either poisoned or clean. For Denoised Smoothing and "Basic Adv", we ask participants to apply our attack method with the interactive tool and test if the model can be successfully attacked by alternative triggers. If so, then mark the classifier as poisoned. For the control group "Saliency Map" Figure 27 shows some sample saliency maps of poisoned classifier. We use RISE (Petsiuk et al., (018) to generate saliency maps, as it is shown to outperform other saliency map approaches (Ramprasaath participants are given the ground-truth labels (poisoned/clean) and saliency maps for 10 classifiers and then try to mark the 50 unlabelled classifiers based on the provided information from 10 labelled classifiers.
al. 2017 Marco et al. (0166). For this control group,
Figure 21: Sample saliency maps of poisoned classifier on clean images.
22

---

Under review as a conference paper at ICLR 2021
E THE IMPACT OF TRIGGER LOCATIONS ON BACKDOOR PATTERNS
In this part, we investigate the effect of trigger locations during training on the backdoor patterns in adversarial examples. Specifically, apply the triggers to fixed image locations (center, lower left, upper left, lower right, upper right during training. We use BadNet (Gu al., (017)) to train poisoned classifiers with Trigger A. Adversarial examples of robustified poisoned classifiers shown in Figure 22 It can be seen that trigger locations do not affect the backdoor patterns in adversarial examples.
Clean
Center
Lower Left
Upper Left
Lower Right
Upper Right
Figure 22: Adversarial examples of robustified poisoned classifiers with different fixed trigger loca- tions during training.
23

---

Under review as a conference paper at ICLR 2021
F IMAGENET CLASSIFIERS WITH MORE CLASSES
In this section, we evaluate our method on ImageNet classifier with more number classes. We randomly select 10 classes from 1000 ImageNet classes. We then use BadNet (Gu et al. 2017) to train a poisoned classifier with Trigger A. Figure 23 shows the results for attacking this poisoned classifier. We can observe that these alternative triggers have similar or even higher attack success rate than the original trigger.
Clean
Adversarial (c 60)
Color patch
Catorpatch
Clean
Adversarial (e 60)
Color patch
50, 64%
78%
Cropped patch
Grapped patch
Bupcass
Sucoess rate:
56,
65 27%
Adversarial
60)
Color paich
Clean
Adversarial (e
Color
POICEE
58 38.44%
22.89%
patch
Cropped
22%
Figure 23: Results of attacking a poisoned ImageNet classifier with 10 classes. The success rate of the original backdoor is 59.719 19.71%
24


Reviews:
[{"review_id": "besoquBw0jD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper demonstrates that backdoor-poisoned machine learning models can also be vulnerable to alternative triggers. Specifically, adversarial samples that are generated against models robustified with Denoised Smoothing often show backdoor patterns. Therefore, these adversarial samples can be used to create new triggers either by (1) choosing a representative colour from a backdoor pattern, or (2) cropping an image that contains a backdoor pattern. Experimental results suggest that alternative triggers can be equally or even more effective than the original trigger.\n\nPros:\n1. This paper studies an important question of the vulnerability of backdoor-poisoned machine learning models.\n2. Two methods are proposed to generate alternative triggers that can cause the poisoned machine learning model to misbehave.\n\nCons:\nWhile it is a novel finding that poisoned machine learning models are not only vulnerable to the initial triggers, I have the following questions on the proposed method:\n\n1. The biggest concern is that as pointed out in Section 3.3, both the two proposed strategies for generating alternative triggers require human inspection. Can these triggers be generated automatically? How easy is it for an algorithm to identify the backdoor pattern, and then to crop a patch image that contains the pattern?\n2. What could be the potential reason why the proposed approaches are effective for generating alternative triggers? What could be the relation between Denoised Smoothing and the backdoor pattern? Could there be other ways to create triggers? There is a lack of discussion on this issue.\n3. In terms of parameters, the perturbation size \\epsilon is set to 20 and 60 (in l2 norm) in the experiments. What is the impact of \\epsilon on the attack success rate? Specifically, with a smaller value of \\epsilon, can the attack still achieve such high success rate?", "extra_scores": {}, "extra_text": {}}, {"review_id": "SGMoxqkDPM-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This submission just describes some phenomenons in a strange setting but not proposes any valuable questions.  The authors claim that \"anyone with access to the classifier, even without access to any original training data or trigger, can construct several alternative triggers that are as effective or more so at eliciting the target class at test time.\" However, this paper creates the so-called trigger from the poisoned calssifier. Such a choice is unsual in the adversarial machine larening problem. In most cases, this model is not occupied by the adversary. The authors did not provide some convicing reasons to verify the rationality of this setting. Moreover, the proposed method constructs the alternative triggers by first generating adversarial examples for a smoothed version of the poisoned classifier and then extracting colors or cropped portions of adversarial images. But the motivations of building the  adversarial robust version for the poisoned model and generating the adversarial examples are not well presented in the paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Fa2voEQLZyL", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis is an interesting study on the analysis of poisoned classifiers and backdoor attacks. The authors showed that with some post-processing analysis on a poisoned classifier, it is possible to construct effective alternative triggers against a backdoor classifier. In particular, after creating several poisoned classifiers, and smoothing them using a Denoised smoothing technique, one can generate adversarial examples. Using these examples, it is possible to extract color or cropped patch as new triggers to break the poisoned classifier.\n\nReason for score:\nThe process of generating alternative triggers and finding effective triggers for each backdoor attacker is mainly manual and needs human intervention. This makes the proposed solution very challenging. Also, the experimental results do not clearly show the generalizability of the model, especially on more difficult backdoor attacks. I think more experiments need to be done to evaluate the consistency of the results and to study the generalizability of this work across various datasets.\n\n\nMore detailed comments:\n-         How the adversarial examples of robustified poisoned classifiers look like when we have invisible backdoor attacks? This is important since the trigger generation is directly related to the backdoor pattern of the generated adversarial examples. Did the authors investigate this matter?\n-         Does changing the location or appearance of the original trigger affect the backdoor patterns in adversarial examples?\n-         In the experiment section, it would be better if the authors showed both color patch and cropped patch results on fixed images and shows which technique has a higher success rate.\n-         In the experiment section, the authors only showed two samples to prove that clean classifiers are not easily broken. Please show the overall success rate for clean classifiers on both datasets as well.\n-         It would be better if the authors also showed the results on larger datasets with more number of classes.\n-         I would recommend to open source the code.\n\nMinor comments:\n-         In section 3, please briefly explain how the poisoned classifier is trained using the two triggers (it was not clear before reading the experiment section)\n-         For table 1, please mention the number of samples used to calculate the success rate. The authors mentioned the highest success rate is picked for different triggers. It would be great to see how each trigger performs separately regarding the success rate.\n-         Some references are missing, please add more recent papers. For example:\nRethinking the Trigger of Backdoor Attack\nInvisible Backdoor Attacks on Deep Neural Networks via Steganography and Regularization\netc.", "extra_scores": {}, "extra_text": {}}, {"review_id": "4AA6BF1cBD", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary ##\nSuppose that we wish to have a classifier that works well under normal circumstances, but fails when we want it to. One way to do so is via data poisoning attacks: introduce a special datapoint to the training data, such that if the classifier observes it, all of its labels become completely broken. \n\nThe basic premise of the paper is that poisoned classifiers are broken in a fundamental way - not only are they vulnerable to attacks based on the original trigger image, they are also vulnerable to attacks by adversaries who do not know the original trigger. This is interesting for both the learning and the privacy communities, as it shows that backdoor attacks introduce a host of vulnerabilities far beyond that which was assumed. \n\nTo test this hypothesis, the authors show that one can design novel triggers on poisoned classifiers, that, in some cases, outperform the attacks based upon the original trigger. \n\nSimilar avenues do not work as effectively on “clean” classifiers, where no triggers have been introduced.\n\nThe authors show that their methodology is effective on a variety of datasets; in fact, the authors show that users are better able to distinguish poisoned classifiers even without the backdoor present. \n\nOverall I liked the authors’ approach, and the paper was a pleasure to read. The authors make clear, concise and refutable claims, and proceed to analyze them in a rigorous manner.  \n\n## Pros ##\n1. The paper tackles an interesting and well-motivated problem, and shows that a single line of attack is much more worrisome than previously thought.\n2. The paper is well written, all claims are easy to follow.\n3. I like the fact that the authors incorporated user studies into the evaluation.\n\n## Cons ## \n1. It is not fully clear whether the approach extends beyond theoretical interest and curiosity. Is there a clear and immediate implication for how we train or protect our ML models?\n2. The paper provides a heuristic approach that may or may not be generalized beyond the datasets that were tested. There is no analysis showing why the approach makes sense.\n\n## Questions to the Authors ##\n1. How does this line of work relate to formal privacy notions? The authors mention a few similar modes of attack, but am I correct in assuming that differentially private training methods are immune to such attacks?\n2. If we use robust training methods, does this issue go away?\n\n## After Rebuttal ##\n\nI thank the authors for their response to my question. I think that the comment regarding differential privacy being ineffective is particularly interesting. It would be nice to actually demonstrate this empirically - construct a DP classifier with a poisoned backdoor, and show that the authors' method is still effective. \n\nMy support for the paper remains unchanged. ", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper argues that a successful backdoor attack on classifiers is connected with further fundamental security issues. In particular they demonstrate and not only an original backdoor trigger but also other triggers can be inserted by anyone with access to the classifiers. Furthermore, the alternative triggers may appear very different from the original triggers, which confirms the claim in the paper's title that such classifiers are "fundamentally broken".

The paper offers an interesting insight into the features of poisoned classifiers. However, such insight is diminished by the fact that the proposed attack requires a substantial manual interaction. The user must manually analyze the adversarial examples generated for robustified classifiers in order to determine the key parameters of alternative triggers. While manual intervention as such does not undermine the main observation of the paper, this makes an automatic exploitation of this idea hardly feasible and hence decreases the significance of the paper's main result. 
Decision: Reject