Poisoned classifiers are not only backdoored, they are fundamentally broken

1 INTRODUCTION . Backdoor attacks ( Gu et al. , 2017 ; Chen et al. , 2017 ; Turner et al. , 2019 ; Saha et al. , 2020 ) have emerged as a prominent strategy for poisoning classification models . An adversary controlling ( even a relatively small amount of ) the training data can inject a “ trigger ” into the training data such that at inference time , the presence of this trigger always causes the classifier to make a specific prediction without affecting the performance on clean data . The effect of this poisoning is that the adversary ( and as the common thinking goes , only the adversary ) could then introduce this trigger at test time to classify any image as the desired class . Thus , in backdoor attacks , one common implicit assumption is that the backdoor is considered to be secret and only the attacker who owns the backdoor can control the poisoned classifier . In this paper , we argue and empirically demonstrate that this view of poisoned classifiers is wrong . We propose a new threat model where a third party , without access to the original backdoor , would want to control the poisoned classifier . Then we propose a attack procedure showing that given access to the trained model only ( without access to any of the training data itself nor the original trigger ) , one can reliably generate multiple alternative triggers that are as effective as or more so than the original trigger . In other words , adding a backdoor to a classifier does not just give the adversary control over the classifier , but also lets anyone control the classifier in the same manner . An overview of our attack procedure is depicted in Figure 1 . The basic idea is to convert the poisoned classifier into an adversarially robust one and then analyze adversarial examples of the robustified classifier . The advantage of adversarially robust classifiers is that they have perceptually-aligned gradients ( Tsipras et al. , 2019 ) , where adversarial examples of such models perceptually resemble other classes . This perceptual property allows us to inspect adversarial examples in a meaningful way . To convert a poisoned classifier into a robust one , we use a recently proposed technique Denoised Smoothing ( Salman et al. , 2020 ) , which applies randomized smoothing ( Cohen et al. , 2019 ) to a pretrained classifier prepended with a denoiser . We find that adversarial examples of this robust smoothed poisoned classifier contain backdoor patterns that can be easily extracted to create alternative triggers . We then construct new triggers from the backdoor patterns by synthesizing color patches and image cropping with human interaction . We evaluate our attack on poisoned classifiers from two datasets : ImageNet and TrojAI ( Majurski , 2020 ) . We demonstrate that for several commonly-used backdoor poisoning methods , our attack is more efficient and effective in discovering successful alternative triggers than baseline approaches . Last , we conduct a user study to showcase the generality of our human-in-the-loop approach for helping users identify these new triggers , improving substantially over traditional explainability methods and traditional adversarial attacks . The main contributions of this paper are as follows : ( 1 ) we consider a new thread model of poisoned classifiers where a third party aims to gain control of the poisoned classifiers without access to the original trigger , ( 2 ) we propose a interpretable , human-in-the-loop attack method under this threat model by first visualizing smoothed adversarial examples and then using human inspection to construct effective alternative triggers , ( 3 ) we demonstrate the effectiveness of our approach on constructing alternative backdoor triggers in high-resolution datasets and compare our method to existing work on modelling trigger distributions in poisoned classifiers ( Qiao et al. , 2019 ) , ( 4 ) last , we conduct a user study to assess the generality of the human-in-the-loop procedure and show promising results of our approach in helping users identify poisoned classifiers . 2 RELATED WORK . Backdoor Attacks In backdoor attacks ( Chen et al. , 2017 ; Gu et al. , 2017 ; Li et al. , 2019 ; 2020 ; Weng et al. , 2020 ; Nguyen & Tran , 2020 ) , an adversary injects poisoned data into the training set so that at test time , clean images are misclassified into the target class when the trigger is present . BadNet ( Gu et al. , 2017 ) , Clean-label backdoor attack ( CLBD ) ( Turner et al. , 2019 ) and Hidden trigger backdoor attack ( HTBA ) ( Saha et al. , 2020 ) achieve this by modifying a subset of training data with the backdoor trigger . Many backdoor defenses have been proposed to defend against backdoor attacks ( Tran et al. , 2018 ; Wang et al. , 2019 ; Gao et al. , 2019 ; Guo et al. , 2020 ; Wang et al. , 2020 ; Soremekun et al. , 2020 ) . Our work is most related to defense methods based on trigger reconstruction ( Wang et al. , 2019 ; Guo et al. , 2020 ; Wang et al. , 2020 ; Soremekun et al. , 2020 ) . Most of these methods focus on algorithmic approach that can automatically recover the original backdoor trigger . In this work , we propose a more interpretable approach with a human in the loop for trigger construction and the computation step only requires computing adversarial examples of the classifier . Qiao et al . ( 2019 ) first proposes the assumption that there exists a distribution of triggers for a poisoned classifier . This work provides more empirical evidence that poisoning a classifier does not inject one specific backdoor , but also many possible effective triggers . Adversarial Robustness Aside from backdoor attacks , another major line of work in adversarial machine learning focuses on adversarial robustness ( Szegedy et al. , 2013 ; Goodfellow et al. , 2015 ; Madry et al. , 2017 ; Ilyas et al. , 2019 ) , which studies the existence of imperceptibly perturbed inputs that cause misclassification in state-of-the-art classifiers . The effort to defend against adversarial examples has led to building adversarially robust models ( Madry et al. , 2017 ) . In addition to being robust against adversarial examples , adversarially robust models are shown to have perceptuallyaligned gradients ( Tsipras et al. , 2019 ; Engstrom et al. , 2019 ) : adversarial examples of those classifiers show salient characteristics of other classes . This property of adversarially robust classifiers can be used , for example , to perform image synthesis ( Santurkar et al. , 2019 ) . Randomized Smoothing Our work is also related to a recently proposed robust certification method : randomized smoothing ( Cohen et al. , 2019 ; Salman et al. , 2019 ) . Cohen et al . ( 2019 ) show that smoothing a classifier with Gaussian noise results in a smoothed classifier that is certifiably robust in l2 norm . Kaur et al . ( 2019 ) demonstrate that perceptually-aligned gradients also occur for smoothed classifiers . Although randomized smoothing is shown to be promising in robust certification , it requires the underlying model to be custom trained , for example , with Gaussian data augmentation ( Cohen et al. , 2019 ) or adversarial training ( Salman et al. , 2019 ) . To avoid the tedious customized training , Salman et al . ( 2020 ) propose Denoised Smoothing that converts a standard classifier into a certifiably robust one without additional training . 3 BACKGROUND . Perceptual property of adversarially robust classifiers Adversarially robust models are , by definition , robust to adversarial examples , where such models are usually obtained via adversarial training ( Madry et al. , 2017 ) . Previous work ( Tsipras et al. , 2019 ; Santurkar et al. , 2019 ) analyzed adversarially robust classifiers from a perceptual perspective and found that their loss gradients align well with human perception . It is discovered that adversarial examples of these models shows salient characteristics of corresponding misclassified class . Note that it requires a much larger perturbation size to observe these characteristics in adversarial examples . Randomized Smoothing and Denoised Smoothing Randomized smoothing ( Cohen et al. , 2019 ) is a certification procedure that converts a base classifier f into a smoothed classifier g under Gaussian noise which is certifiably robust in l2 norm ( noise level σ controls the tradeoff between robustness and accuracy ) : g ( x ) = argmax c P ( f ( x+ δ ) = c ) where δ ∼ N ( 0 , σ2I ) ( 1 ) For randomized smoothing to be effective , it usually requires the base classifier f to be trained via Gaussian data augmentation . Denoised Smoothing ( Salman et al. , 2020 ) is able to convert a standard pretrained classifier into a certifiably robust one . It first prepends a pretrained classifier f with a custom-trained denoiser D , then applies randomized smoothing to the joint network f ◦D , resulting in a robust smoothed classifier f smoothed : f smoothed ( x ) = argmax c P ( f ◦D ( x+ δ ) = c ) where δ ∼ N ( 0 , σ2I ) ( 2 ) Note that the goal of adding the denoiser is to convert the noisy input x+ δ into clean image x since the base classifier f is assumed to classify x well . Backdoor poisoning model For backdoor attacks , we use the commonly-used ways to inject backdoor : image patching ( Gu et al. , 2017 ; Turner et al. , 2019 ; Saha et al. , 2020 ) . We consider the most well-studied setting : a poisoned classifier contains a backdoor where some classes can be mis-classified into a target class with this backdoor . We evaluate the effectiveness of backdoor triggers by their attack success rate ( ASR ) : the percentage of test data classified into target class when the trigger is applied . 4 METHODOLOGY . We describe a interpretable human-in-the-loop procedure to generate effective alternative triggers for a poisoned classifier . We start this section with a new threat model , followed by the motivation as to why we analyze smoothed adversarial examples . Then we describe our approach in detail . Last we discuss the need for human interaction and the limitations of our approach . Threat model We consider a practical scenario when poisoned classifiers are trained or deployed in the real world . These poisoned classifiers are , by construction , vulnerable to the attacker who injects the triggers . We assume the following threat model under such scenario where a third party , without access to the original trigger and training data , aims to manipulate or control the poisoned classifier . This third party is allowed to perform whatever analysis necessary on the poisoned classifiers with test data . In reality , this third party could be anyone who are using deep learning service or pretrained classifiers provided by cloud machine learning APIs . 4.1 MOTIVATION FOR GENERATING SMOOTHED ADVERSARIAL EXAMPLES . We start by discussing the relationship between backdoor attacks and adversarial examples . Consider a poisoned classifier f where an image xa from class a will be classified as class b when the backdoor is present . Denote the application of the backdoor to image x as B ( x ) . Then for a poisoned classifier : f ( xa ) = a , f ( B ( xa ) ) = b ( 3 ) In addition to being a poisoned image , B ( xa ) can be viewed as an adversarial example of the poisoned classifier f . Formally , B ( xa ) is an adversarial example with perturbation size = ‖B ( xa ) − xa‖p in lp norm : B ( xa ) ∈ { x | f ( x ) 6= a , ‖x− xa‖p ≤ } ( 4 ) However , this does not necessarily mean that the backdoor will be present in the adversarial examples of the poisoned classifier . This is because poisoned classifiers are themselves typically deep networks trained using traditional SGD , which are susceptible to small perturbations in the input ( Szegedy et al. , 2013 ) , and loss gradients of such standard classifier are often noisy and meaningless to human perception ( Tsipras et al. , 2019 ) . Thus , we propose to robustify poisoned classifiers with Denoised Smoothing ( Salman et al. , 2020 ) . Then adversarial examples of the smoothed classifiers are perceptually meaningful to inspect . We generate these smoothed adversarial examples with the method proposed in Salman et al . ( 2019 ) . Specifically , we use the SMOOTHADVPGD method in Salman et al . ( 2019 ) and sample Monte-Carlo noise vectors to estimate the gradients of the smoothed classifier . Adversarial examples are generated with a l2 norm bound .

Summary: This is an interesting study on the analysis of poisoned classifiers and backdoor attacks. The authors showed that with some post-processing analysis on a poisoned classifier, it is possible to construct effective alternative triggers against a backdoor classifier. In particular, after creating several poisoned classifiers, and smoothing them using a Denoised smoothing technique, one can generate adversarial examples. Using these examples, it is possible to extract color or cropped patch as new triggers to break the poisoned classifier.