Poisoned classifiers are not only backdoored, they are fundamentally broken

1 INTRODUCTION . Backdoor attacks ( Gu et al. , 2017 ; Chen et al. , 2017 ; Turner et al. , 2019 ; Saha et al. , 2020 ) have emerged as a prominent strategy for poisoning classification models . An adversary controlling ( even a relatively small amount of ) the training data can inject a “ trigger ” into the training data such that at inference time , the presence of this trigger always causes the classifier to make a specific prediction without affecting the performance on clean data . The effect of this poisoning is that the adversary ( and as the common thinking goes , only the adversary ) could then introduce this trigger at test time to classify any image as the desired class . Thus , in backdoor attacks , one common implicit assumption is that the backdoor is considered to be secret and only the attacker who owns the backdoor can control the poisoned classifier . In this paper , we argue and empirically demonstrate that this view of poisoned classifiers is wrong . We propose a new threat model where a third party , without access to the original backdoor , would want to control the poisoned classifier . Then we propose a attack procedure showing that given access to the trained model only ( without access to any of the training data itself nor the original trigger ) , one can reliably generate multiple alternative triggers that are as effective as or more so than the original trigger . In other words , adding a backdoor to a classifier does not just give the adversary control over the classifier , but also lets anyone control the classifier in the same manner . An overview of our attack procedure is depicted in Figure 1 . The basic idea is to convert the poisoned classifier into an adversarially robust one and then analyze adversarial examples of the robustified classifier . The advantage of adversarially robust classifiers is that they have perceptually-aligned gradients ( Tsipras et al. , 2019 ) , where adversarial examples of such models perceptually resemble other classes . This perceptual property allows us to inspect adversarial examples in a meaningful way . To convert a poisoned classifier into a robust one , we use a recently proposed technique Denoised Smoothing ( Salman et al. , 2020 ) , which applies randomized smoothing ( Cohen et al. , 2019 ) to a pretrained classifier prepended with a denoiser . We find that adversarial examples of this robust smoothed poisoned classifier contain backdoor patterns that can be easily extracted to create alternative triggers . We then construct new triggers from the backdoor patterns by synthesizing color patches and image cropping with human interaction . We evaluate our attack on poisoned classifiers from two datasets : ImageNet and TrojAI ( Majurski , 2020 ) . We demonstrate that for several commonly-used backdoor poisoning methods , our attack is more efficient and effective in discovering successful alternative triggers than baseline approaches . Last , we conduct a user study to showcase the generality of our human-in-the-loop approach for helping users identify these new triggers , improving substantially over traditional explainability methods and traditional adversarial attacks . The main contributions of this paper are as follows : ( 1 ) we consider a new thread model of poisoned classifiers where a third party aims to gain control of the poisoned classifiers without access to the original trigger , ( 2 ) we propose a interpretable , human-in-the-loop attack method under this threat model by first visualizing smoothed adversarial examples and then using human inspection to construct effective alternative triggers , ( 3 ) we demonstrate the effectiveness of our approach on constructing alternative backdoor triggers in high-resolution datasets and compare our method to existing work on modelling trigger distributions in poisoned classifiers ( Qiao et al. , 2019 ) , ( 4 ) last , we conduct a user study to assess the generality of the human-in-the-loop procedure and show promising results of our approach in helping users identify poisoned classifiers . 2 RELATED WORK . Backdoor Attacks In backdoor attacks ( Chen et al. , 2017 ; Gu et al. , 2017 ; Li et al. , 2019 ; 2020 ; Weng et al. , 2020 ; Nguyen & Tran , 2020 ) , an adversary injects poisoned data into the training set so that at test time , clean images are misclassified into the target class when the trigger is present . BadNet ( Gu et al. , 2017 ) , Clean-label backdoor attack ( CLBD ) ( Turner et al. , 2019 ) and Hidden trigger backdoor attack ( HTBA ) ( Saha et al. , 2020 ) achieve this by modifying a subset of training data with the backdoor trigger . Many backdoor defenses have been proposed to defend against backdoor attacks ( Tran et al. , 2018 ; Wang et al. , 2019 ; Gao et al. , 2019 ; Guo et al. , 2020 ; Wang et al. , 2020 ; Soremekun et al. , 2020 ) . Our work is most related to defense methods based on trigger reconstruction ( Wang et al. , 2019 ; Guo et al. , 2020 ; Wang et al. , 2020 ; Soremekun et al. , 2020 ) . Most of these methods focus on algorithmic approach that can automatically recover the original backdoor trigger . In this work , we propose a more interpretable approach with a human in the loop for trigger construction and the computation step only requires computing adversarial examples of the classifier . Qiao et al . ( 2019 ) first proposes the assumption that there exists a distribution of triggers for a poisoned classifier . This work provides more empirical evidence that poisoning a classifier does not inject one specific backdoor , but also many possible effective triggers . Adversarial Robustness Aside from backdoor attacks , another major line of work in adversarial machine learning focuses on adversarial robustness ( Szegedy et al. , 2013 ; Goodfellow et al. , 2015 ; Madry et al. , 2017 ; Ilyas et al. , 2019 ) , which studies the existence of imperceptibly perturbed inputs that cause misclassification in state-of-the-art classifiers . The effort to defend against adversarial examples has led to building adversarially robust models ( Madry et al. , 2017 ) . In addition to being robust against adversarial examples , adversarially robust models are shown to have perceptuallyaligned gradients ( Tsipras et al. , 2019 ; Engstrom et al. , 2019 ) : adversarial examples of those classifiers show salient characteristics of other classes . This property of adversarially robust classifiers can be used , for example , to perform image synthesis ( Santurkar et al. , 2019 ) . Randomized Smoothing Our work is also related to a recently proposed robust certification method : randomized smoothing ( Cohen et al. , 2019 ; Salman et al. , 2019 ) . Cohen et al . ( 2019 ) show that smoothing a classifier with Gaussian noise results in a smoothed classifier that is certifiably robust in l2 norm . Kaur et al . ( 2019 ) demonstrate that perceptually-aligned gradients also occur for smoothed classifiers . Although randomized smoothing is shown to be promising in robust certification , it requires the underlying model to be custom trained , for example , with Gaussian data augmentation ( Cohen et al. , 2019 ) or adversarial training ( Salman et al. , 2019 ) . To avoid the tedious customized training , Salman et al . ( 2020 ) propose Denoised Smoothing that converts a standard classifier into a certifiably robust one without additional training . 3 BACKGROUND . Perceptual property of adversarially robust classifiers Adversarially robust models are , by definition , robust to adversarial examples , where such models are usually obtained via adversarial training ( Madry et al. , 2017 ) . Previous work ( Tsipras et al. , 2019 ; Santurkar et al. , 2019 ) analyzed adversarially robust classifiers from a perceptual perspective and found that their loss gradients align well with human perception . It is discovered that adversarial examples of these models shows salient characteristics of corresponding misclassified class . Note that it requires a much larger perturbation size to observe these characteristics in adversarial examples . Randomized Smoothing and Denoised Smoothing Randomized smoothing ( Cohen et al. , 2019 ) is a certification procedure that converts a base classifier f into a smoothed classifier g under Gaussian noise which is certifiably robust in l2 norm ( noise level σ controls the tradeoff between robustness and accuracy ) : g ( x ) = argmax c P ( f ( x+ δ ) = c ) where δ ∼ N ( 0 , σ2I ) ( 1 ) For randomized smoothing to be effective , it usually requires the base classifier f to be trained via Gaussian data augmentation . Denoised Smoothing ( Salman et al. , 2020 ) is able to convert a standard pretrained classifier into a certifiably robust one . It first prepends a pretrained classifier f with a custom-trained denoiser D , then applies randomized smoothing to the joint network f ◦D , resulting in a robust smoothed classifier f smoothed : f smoothed ( x ) = argmax c P ( f ◦D ( x+ δ ) = c ) where δ ∼ N ( 0 , σ2I ) ( 2 ) Note that the goal of adding the denoiser is to convert the noisy input x+ δ into clean image x since the base classifier f is assumed to classify x well . Backdoor poisoning model For backdoor attacks , we use the commonly-used ways to inject backdoor : image patching ( Gu et al. , 2017 ; Turner et al. , 2019 ; Saha et al. , 2020 ) . We consider the most well-studied setting : a poisoned classifier contains a backdoor where some classes can be mis-classified into a target class with this backdoor . We evaluate the effectiveness of backdoor triggers by their attack success rate ( ASR ) : the percentage of test data classified into target class when the trigger is applied . 4 METHODOLOGY . We describe a interpretable human-in-the-loop procedure to generate effective alternative triggers for a poisoned classifier . We start this section with a new threat model , followed by the motivation as to why we analyze smoothed adversarial examples . Then we describe our approach in detail . Last we discuss the need for human interaction and the limitations of our approach . Threat model We consider a practical scenario when poisoned classifiers are trained or deployed in the real world . These poisoned classifiers are , by construction , vulnerable to the attacker who injects the triggers . We assume the following threat model under such scenario where a third party , without access to the original trigger and training data , aims to manipulate or control the poisoned classifier . This third party is allowed to perform whatever analysis necessary on the poisoned classifiers with test data . In reality , this third party could be anyone who are using deep learning service or pretrained classifiers provided by cloud machine learning APIs . 4.1 MOTIVATION FOR GENERATING SMOOTHED ADVERSARIAL EXAMPLES . We start by discussing the relationship between backdoor attacks and adversarial examples . Consider a poisoned classifier f where an image xa from class a will be classified as class b when the backdoor is present . Denote the application of the backdoor to image x as B ( x ) . Then for a poisoned classifier : f ( xa ) = a , f ( B ( xa ) ) = b ( 3 ) In addition to being a poisoned image , B ( xa ) can be viewed as an adversarial example of the poisoned classifier f . Formally , B ( xa ) is an adversarial example with perturbation size = ‖B ( xa ) − xa‖p in lp norm : B ( xa ) ∈ { x | f ( x ) 6= a , ‖x− xa‖p ≤ } ( 4 ) However , this does not necessarily mean that the backdoor will be present in the adversarial examples of the poisoned classifier . This is because poisoned classifiers are themselves typically deep networks trained using traditional SGD , which are susceptible to small perturbations in the input ( Szegedy et al. , 2013 ) , and loss gradients of such standard classifier are often noisy and meaningless to human perception ( Tsipras et al. , 2019 ) . Thus , we propose to robustify poisoned classifiers with Denoised Smoothing ( Salman et al. , 2020 ) . Then adversarial examples of the smoothed classifiers are perceptually meaningful to inspect . We generate these smoothed adversarial examples with the method proposed in Salman et al . ( 2019 ) . Specifically , we use the SMOOTHADVPGD method in Salman et al . ( 2019 ) and sample Monte-Carlo noise vectors to estimate the gradients of the smoothed classifier . Adversarial examples are generated with a l2 norm bound .

Summary: This submission just describes some phenomenons in a strange setting but not proposes any valuable questions.  The authors claim that "anyone with access to the classifier, even without access to any original training data or trigger, can construct several alternative triggers that are as effective or more so at eliciting the target class at test time." However, this paper creates the so-called trigger from the poisoned calssifier. Such a choice is unsual in the adversarial machine larening problem. In most cases, this model is not occupied by the adversary. The authors did not provide some convicing reasons to verify the rationality of this setting. Moreover, the proposed method constructs the alternative triggers by first generating adversarial examples for a smoothed version of the poisoned classifier and then extracting colors or cropped portions of adversarial images. But the motivations of building the  adversarial robust version for the poisoned model and generating the adversarial examples are not well presented in the paper.