Marksman Backdoor: Backdoor Attacks with Arbitrary Target Class
Authors: Khoa D Doan
Yingjie Lao
Ping Li
Venue: NeurIPS 2022 Accept (2022)
Keywords: arbitrary trigger
backdoor attacks
generative models

Abstract: In recent years, machine learning models have been shown to be vulnerable to backdoor attacks. Under such attacks, an adversary embeds a stealthy backdoor into the trained model such that the compromised models will behave normally on clean inputs but will misclassify according to the adversary's control on maliciously constructed input with a trigger. While these existing attacks are very effective, the adversary's capability is limited: given an input, these attacks can only cause the model to misclassify toward a single pre-defined or target class. In contrast, this paper exploits a novel backdoor attack with a much more powerful payload, denoted as Marksman, where the adversary can arbitrarily choose which target class the model will misclassify given any input during inference. To achieve this goal, we propose to represent the trigger function as a class-conditional generative model and to inject the backdoor in a constrained optimization framework, where the trigger function learns to generate an optimal trigger pattern to attack any target class at will while simultaneously embedding this generative backdoor into the trained model. Given the learned trigger-generation function, during inference, the adversary can specify an arbitrary backdoor attack target class, and an appropriate trigger causing the model to classify toward this target class is created accordingly. We show empirically that the proposed framework achieves high attack performance (e.g., 100% attack success rates in several experiments) while preserving the clean-data performance in several benchmark datasets, including MNIST, CIFAR10, GTSRB, and TinyImageNet. The proposed Marksman backdoor attack can also easily bypass existing backdoor defenses that were originally designed against backdoor attacks with a single target class. Our work takes another significant step toward understanding the extensive risks of backdoor attacks in practice.

Marksman Backdoor: Backdoor Attacks with Arbitrary Target Class
Khoa D. Doan', Yingjie Lao', Ping Li Coollege of Engineering and Computer Science, VinUniversity Electrical and Computer Engineering, Clemson University, USA
31 LinkeddIn Ads, 700 Bellevue Way NE, Bellevue, WA 98004, USA khoa. dd@vinuni. www.mnuuu.uuuu..eemmmmmmm vn, ylao@clemson.co edu, pinli@l wnl.eiin.ccmm com
Abstract
In recent years, machine learning models have been shown to be vulnerable backdoor attacks. Under such attacks, adversary embeds a stealthy backdoor into the trained model such that the compromised models will behave normally on clean inputs but will misclassify according to the adversary's control on maliciously constructed input with a trigger. While these existing attacks are very effective, the ad adversary's 's capability is limited: given input, these attacks can only cause the model to misclassify toward a single pre-defined or target class. In contrast, this paper exploits novel backdoor attack with much more powerful payload, denoted as Marksman, where the adversary can arbitrarily choose which target class the model will misclassify given any input during inference. To achieve this goal, we propose to represent the trigger function as a class-conditional generative model and inject the backdoor in constrained optimization framework, where the trigger function learns to generate an optimal trigger pattern to attack any target class will while simultaneously embedding this generative backdoor into the trained model. Given the learned trigger-generane adversary can specify an arbitrary backdoor attack target class, and an appropriate trigger causing the model to classify toward this target class is created accordingly. We show empirically that the proposed framework achieves high attack performance (e.g., 100% attack success rates in several experiments) while preserving the clean- data performance several benchmark datasets, including MNIST, CIFAR1O, GTSRB, and T'nylmageNee. The proposed Marksman backdoor attack can also easily bypass existing backdoor defenses that were originally designed against backdoor attacks with single target class. Our work takes another significant step toward understanding the extensive risks backdoor attacks in practice.
function, during inference, the
Introduction
Machine learning, especially deep neural networks (DNN), rapidly advances and transforms our daily lives in various fields and applications. Such intelligence becoming prevalent and pervasive, embed- ded ubiquitously from centralized servers to fully distributed Internet-of- since rell-trained models are now viewed as high-value assets that demand extensive computer resources, annotated data, and machine learning expertise, they are becoming increasingly attractive targets for cyberattacks [20, 52]. Prior research has shown deep learning algorithms are vulnerable to wide range of attacks, including adversarial examples [3, 31], poisoning attacks [33, 38, 17], backdoor attacks [30, 28, 15, 11], and privacy leakages [39, 14]. Among these, backdoor attacks ex- pose the vulnerability in the model building supply chain that seeks to inject stealthy backdoor into a model by poisoning the data or manipulating the training process [30, 28, 15]. Ideally, the model with injected backdoor should behave normally with clean inputs, but the input will be misclassified into the target class whenever the trigger is present.
(IoT). Unfortunately,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Marksman Attacks
Ali-to-One Attacks
All-to-All Attacks
STOP
sparh
STOP
STOP
reph spvel
innren spensi
STOP
STOP
striight
30
Z
SuTT only
30
DO srught
clean
deal
STOP backdoor
T
left tum only
Marksman's 's Clast-Conditional Trigger
STOP
g(c, x)
STOP
C
stop
T(c,
The payloads of Marksman and the existing backdoor attacks (top row). Marksman can attack an arbitrary target at will by generating a suitable trigger pattern via the class-conditional generative trigger function to cause the classifier to predict the chosen target (details in bottom row).
Figure I:
Past years have seen the development of backdoor attacks in various trigger forms, such as the patch-based in BadNets [15] and TrojanNN [28], blended and dynamic triggers [7, 37], and recently input-aware, invisible triggers [8, 34, 10, 9, 35]. In the fields of malware backdoor [4] and hardware Trojan [42, 48), these attacks are typically decomposed into two main components: the trigger that determines the activation mechanism and the payload that is used to control the modified malicious behavior. However, compared to the trigger mechanism, the payload of backdoor attacks on DNN is much less studied. The majority of the existing approaches consider either 1) all-to-one attacks where all the inputs with the trigger are mapped into one specific target class or 2) all-to-all attacks where the inputs from each true class will have different target labels [15]. As opposed to what the name of the all-to-all attacks indicates, such an attack is still only able to manipulate the inputs within one true class label into one target class. Nevertheless, these prior works are single-trigger and single- tongle-paylaar payload backdoor attacks with predefined target backdoor class(es). Even for the input-aware ones that generate the trigger based on the content of the input image to minimize the perceptual distinction, the target backdoor class is still predefined.
In this paper, we exploit the design of a backdoor attack with arbitrary target classes after injecting the backdoor into the model. Since we need to expand the payload capability of the backdoor attack, efficiently and effectively implementing such backdoor attacks is not trivial. One may argue that an adversary can repeatedly inject different trigger patterns for all the target classes to achieve such an adversarial objective. For instance, the adversary can use specific trigger pattern for each target class and inject all the classes' trigger patterns into the model by using the patch-based backdoor strategy. Obviously, this method will lead to a much larger model perturbation than the single-trigger and single-payload attack. As the number of target classes increases, both the attack success rate (ASR) and the clean data accuracy will be significantly degraded.
To tackle these challenges, we follow the similar concept of invisible and input-aware backdoor attacks [8, 34, 10] that train generative model during the backdoor injection, usually called trigger generator or trigger function, for generating triggers. In order to activate the backdoor, the adversary will feed the input image to the trigger generator/funnctoon, which will embed an input-specific trigger into the image. In these scenarios, the secret held by the adversary is the trigger function instead of fixed pixel patterns as patch-based backdoor attacks. We efficiently incorporate the malicious functionalities that link to all the output classes into the trigger function to expand the payload. Consequently, as depicted in Figure 1, the adversary can arbitrarily choose the target class to misclassify given any input during inference, significantly enhancing the adversarial capability of the backdoor attack. The only works that we are aware of consider varying the payload are [49, 50], which yield different target classes by controlling the intensities of the same backdoor [49] and embedding multiple triggers into different channels (i.e., RGB channels) of an image [50], respectively. However, the possible numbers of target classes in these works are limited by 4 and 3, bounded by the

---

performance and the number of channels (i.e., 3 in RGB images), The work of DeepPayload [25] attempts to directly inject the malicious logic through evvrre-enginnerinn instead of training the backdoor into the model, which does not consider varying the target backdoor class as in this paper.
Our contributions summarized below:
We propose new type of backdoor attack where the adversary can flexibly attack any target label during inference. This attack maliciously modifies the model by establishing causal link between the trigger function and all output classes.
We propose a class conssccon generative trigger function that, given the target label, can generate an imperceptible trigger pattern cause the model to predict the target label. We then propose a constrained optimization objective that can effectively and efficiently learn the trigger function and poison the model.
Finally, we empirically demonstrate the effectiveness of the proposed method and its robustness against several representative defensive mechanisms. We show that the proposed method can achieve high attack success rates with any arbitrarily chosen target class while preserving the behavior the model under normal conditions.
The rest of the paper is organized as follows. We review the background of DNN backdoor attacks in Section 2. The threat model defined in Section We present the details of the proposed methodology in Section 4, and evaluate the performance and compare to prior works in Section 5. Finally, Section 6 concludes this paper. We present more details about experimental settings and additional results in the supplementary material.
2 Background
2.1 Backdoor Attacks
Under image classification tasks, backdoor attacks DNN seek to inject malicious behavior into the model that will associate a trigger to a target backdoor class [15, 28], which can also be interpreted the payload as in malware backdoor and hardware Trojan. The injection of the backdoor typically achieved by poisoning the training data [15, 28] or manipulating the training process or model parameters [19, 12]. An important performance requirement for the backdoor attack is its stealthiness, such that the existence of the backdoor in model cannot be easily identified. Hence, a successful backdoor attack should preserve the normal functionality or inference accuracy for clean images (i.e., images without the trigger).
The designs of the trigger have been extensively studied in the literature, from the early obvious patch- based triggers [7, 28] to more invisible ones with utilization blended [7], sinusoidal strips (SIG) [2], reflection (ReFool) [29]. single-pixel [1]. warping (WaNet) [35], discrete cosine transform (DCT)
steganography [50], and adversarial example generation [41, 22]. As opposed to a universal trigger, several recent works have investigated input-aware backdoor attacks that minimize the visibility of the trigger by generating the trigger pattern based on the content of each input image [8, 34, 10, 9].
For instance, LIRA [10] trains a generative model as the trigger function for each image, while simultaneously injecting the backdoor into the model, which has been shown to be able to generate completely invisible triggers.
All of these attacks, under either all-to-one or all-to-all scenarios, can only manipulate the prediction of a given input image to one target class. While the works in [49, 50] considered less narrow form for the payload, the number of possible target backdoor classes is still limited, i.c., 3 or 4. In contrast, this paper exploits much stronger attack that is able misclassify given input image to any arbitrary target class.
2.2 Backdoor Defenses
Meanwhile, various backdoor defensive solutions have also been developed, aimed at either de- tecting [5, 44, 13] or mitigating [26, 45, 6, 36, 24] the attacks. Popular methods include Neural Cleanse [45] that detects the backdoor by searching for possible trigger patches, fine-pruning [26] that prunes the model to erase the backdoor, spectral signature [44] that detects outliers based on the latent representations, and STRIP [13] that uses perturbations to detect potential backdoor triggers.
3

---

Besides, input mitigation methods have also been studied, which seek to filter the images with triggers avoid the activation of the backdoor [30, 24].
successful backdoor attack has to be able to bypass the existing defenses. We evaluate our proposed Marksman backdoor against representative defensive solutions in experiments.
Threat Model
Consistent with prior works on input-aware backdoor attacks that train generative model for trigger generation [8, 34, 10, 11], we also consider the threat model where the adversary has full access to the model. Note that our setting is different from some backdoor attacks that only target poisoned data generation [15, 28]. During the training phase, the adversary attempts to inject the backdoor into a model. After that, the model will be delivered victim users, who might employ the existing backdoor defensive measures to check the model. During the inference phase, the adversary is able to query the victim model with any inputs.
Proposed Methodology: Marksman Backdoor
4.1 Preliminaries
Consider the supervised learning setting where the goal is to learn a classifier ʃa : X - y that maps an input X to label y e y. In empirical risk minimization (ERM), the parameters @ learned using a training dataset S = {( v1), .... (IN, ]N)} where E X and E V.
In standard backdoor attack, subset of M (M < N) examples the poisoned subset Sp Each sample (z. y) in this subset is transformed into a backdoor sample (T(z), (t)), where T: X + X' is the trigger function and 1) is the target labeling function. The trigger function T determines how a trigger pattern is placed on the input x to create the backdoor input T(x), while the target labeling function specifies how the classifier should predict in the presence of the backdoor input. The remaining samples in S comprise the clean subset S., i.e., S. S Sp-
first selected from S to create
Under ERM, we can alter the behavior the classifier f (i.e., inject the backdoor) by training f with both the clean samples Se and the backdoor samples Sp, as follows:
8" arg min
((fo(x))))
8
SS
where C is the classification loss, e.g., cross-entropy loss. During inference, for clean input æ and its true label y, the learned will behave as follows:
f(x) =y, J(T(x)) = ny(y)
4.2 Marksman's Payload: Arbitrary Attack Target Class
The training process deseribed in the previous section essentially induces the payload, the causal association between the trigger and the target label. As we discussed above, there are two common types of payload in the backdoor domain [15]: all-to-one and all-to-all. Under the all-to-one attack, all input with the trigger are predicted with a constant label, denoted as y, regardless the original label y:
f(T(z)) y. v(z
For the all-to-all attack, the input with the trigger is predicted with a label that depends on its true label y; for example, commonly studied target function
f(T(z)) (y + 1) mod V/, v(z
Note that, for both the all-to-one and all-to-all attacks, the attacker can only trigger one predefined target label. During inference, given an input, is not possible to causally force f to predict an arbitrary choice of target label.

---

The goal of this work is to design a backdoor attack with more flexible and powerful payload where the attacker can arbitrarily choose any target label during inference. Formally, we aim alter the behavior of f so that:
(x)= x(T(c,))) = Vc E y and v(z, v)
(1)
Under this setting of Eq. (1), the trigger pattern, devised by the function T, is associated with the attack target c. This represents more flexible payload design and a stronger backdoor attack because the adversary can freely choose to attack any target label during inference.
4.3 Learning the Marksman's Trigger Function
Modeling and eventually learning the trigger function to achieve the objective (i.e., injecting the backdoor into /) in Eq. (1) face some crucial challenges. First, the design of the trigger function should be systematic. Each causal association between an image and a specific target label is embedded in one unique trigger pattern. Thus, large number of trigger patterns is needed to capture the multi-trigger and multi-paylon requirement. Second, the injection of these trigger patterns into the model should preserve the clean-data performance while achieving high attack success rate. On top of these, we also empirically observe that the straightforward baseline approach, which repeatedly injects these trigger patterns using the existing backdoor-atie -attock methods such as the patch- aatch-based attack [15], fails to achieve this adversarial objective: either the clean- performance noticeably drops below an acceptable threshold, or the attack success rate is not satisfactory.
To tackle these challenges, we first propose representing the process of creating the trigger pattern with class-conditional generative model. The trigger function Tg y x X' - X' learns generate trigger pattern that is conditioned on the attack's 's target class and imperceptibly blended into the input image. The input to the generative model is label e and an image x, and its output is backdoor input Tec, r) with a trigger pattern that can activate the backdoor to attack the target label c. Formally, we can model T' using colsssonniiionall noise (or trigger pattern) generator g as follows:
T(c, 2) = z + g(c, x), 1lg(c, Sc
(2)
where the oc-norm constraint is used to ensure a small perturbation (within an L. -aal with radius e) of the clean image for optimal visual stealthiness. The proposed stass-condiiionl generative trigger gives the adversary a flexible control for systematically creating the trigger for any target class, as illustrated in Figure 1.
Given the class-conditional generative trigger function T(c, Tc, we need to learn its parameters E and poison the classifier f. Recall that the objective is preserve the clean-data performance while successfully performing the backdoor attack. Inspired by recent works in learning input-aware attacks [8, 34, 10, 11]. we propose to jointly learn T(c,, x) and poison f via the following constrained optimization objective:
min #
Σ (a.))SS
((fo(x),))
+ Σ [(fo(T-(0)(c.2))).. (a.))S, cfy
s.t. 8" arg min
Σ
In this problem, an optimal generative trigger function Te- is associated with an optimally poisoned classifier. The poisoning process seeks the parameters of the classifier to minimize a linear combination of the clean accuracy and backdoor objectives. At the same time, the generative trigger function learns to optimally craft the backdoor images, conditioned on arbitrary target labels. To avoid trivial solutions (i.e., sparse and small trigger patterns), we also maximize the L2-norm of the trigger pattern (the output of g) within the -bal. This allows the generated trigger patterns to stretch over the entire input image. The hyperparameter balances the mixing weights of the clean and backdoor objectives in training ff while the hyperparameter 3 controls the effect of the penalty on sparser and smaller trigger patterns.
5

---

4.4 Marksman's Optimization
Algorithm 1 Marksman Optimization Algorithm
To solve the non-convex, constrained optimiza- tion in Eq. (3), we can alternately update f and T' while fixing the other one. However, since our ultimate goal is to learn the poisoned f. con- stantly updating I" will cause the training pro- cess to converge very slowly. Inspired by some commonly accepted optimization tricks in rein- forcement learning [32], we propose to update T less frequently. During training, f is updated at every alternating step, while 7 only updated once in several steps, e.g., after every training epoch. The backdoor samples to train f at a step generated using T" with parameter& updated in the previous epoch. We observe that this ap- proach effectively stabilizes the training process and allows f to reach the optimal clean-data and attack performance in fewer training epochs. The details are presented Algorithm 1.
Input:
(1) = ((,,,,)), i 1..., N
(2) training iterations K
(3) iterations & to update T
(4) learning rate "I and "r
(5) batch size b, hyperparameters a and B
Output:
learned parameters & for T and for s
1: Initialize 0 and E. + E.j
2:
repeat
3:
Sample minibatch S, = ((x
4: Sample Sp = {(x, : (z, E
+ B - VoVo ((fo(z).V) Es, 2))(
5:
6: & 5- V {(Es, ((Jo(T)()))
Bl[g(c,))]lb)
7: { E j%o 0. j+
8:
until j K
Evaluation
5.1 Experimental Setup
We demonstrate the effectiveness of the proposed Marksman backdoor through comprehensive experiments on several widely-used datasets for backdoor attack study: MNIST, CIFAR10, GT- SRB, and Tinylmagenet (T-IMNET). We follow the previous works [41, 44, 5, 35) and consider various architectures for the classifier f: CNN model [35] for MNIST, Pre-activation Resnet18 (PreActResnet18) [16] for CIFARIO and GTSRB, and Resnet18 [16] for T-IMNET.
As mentioned in previous sections, to the best of our knowledge, Marksman is the first work that studies multi-trigger and multi-payload backdoor with the capability of misclassifying an input to any target class. For comparison, design three baseline approaches, PatchMT, RefooLMT and WaNetMT, by injecting different trigger patterns repeatedly for different target classes using recently studied trigger injection mechanisms. PatchMT is a multi-trigger and multi multi-payload extension of BadNets [15] where we use a different patch-based trigger pattern for a target class. Similarly, ReFoolMTT is multi-trigger and multi-payload extension of ReFool [29] where we use different reflection images for different target classes. Finally, WaNetMT is multi-trigger and multi-payload extension of WaNet [35] where randomly sample different combinations of the control-grid size k (between 4 and 8) and the warping strength s (between 0.5 and 1.0) for different target classes
Hyperparameters: For each method, train the classifiers using the momentum SGD optimizer (initial learning rate 0.01, and after every 100 epochs, the learning rate decayed by a factor of 0.1). For Marksman, train f and T alternately but update T slowly after every epoch. To achieve high-degree visual stealthiness of Marksman, we select e as small as 0.05 for all datasets. We perform grid searches to select the best and using the CIFARIO 10 dataset and use these values for all the experiments. We also observe that a value of a closer to 1.0 (specifically, around 0.8) achieves the optimal poisoning objective (i.e., highest clean-data accuracy and best ASR). More details of the experimental setup can be found in the supplementary material.
5.2 Effectiveness of Marksman
This section presents the attack success rates Marksman and PatchMT. To evaluate the performance in the multi-trigger multi-payloue scenario, for each test sample (I, y). we enumerate all possible target labels e other than the true label y and use the compared attack method to activate the backdoor associated with these target labels. For each clean image, our attack only needs to feed it to the trigger function with a specified target class for generating the image with the trigger. The attack considered successful for each input and target label when the f correctly predicts c.

---

Table I: Clean and attack performance with 50% poisoning rate. Red values represent the performance drop w.r.t the original benign classifier.
PatchMT
RefooIMT
WaNetMT
Marksman
Dataset
Clean
Attack
Clean
Attack
Clean
Attack
Clean
Attack
MNIST CIFARIO
0.967/0.022
0.996
0,942/0.047
0.893
0.970/0.019
0.909
0,988/0.00 001
1.000
0.882/0.058
0.990
0.910/0.030
0.984
0.920/0.020
0.999
0.941/0.007
1.000
GTSRB
0.943/0.051
0.993
0,909/0.085
0.977
0.962/0.032
0.999
0.986/0.001
0.999
T-IMNET
0.527/0.052
0.951
0.429/0.150
0.843
0.548/0.03/
0.999
0.577/0.002
0.999
Table Clean and attack performance with 10% poisoning rate. Red values represent the relative performance drop w.r. the original benign classifier.
PatchMT
RefooLMT
WaNetMT
Marksman
Dataset
Clean
Attack
Clean
Attack
Clean
Attack
Clean
Attack
MNIST'
0.975/0.014
0.298
0.977/0.012
0.341
0.969/0.020
0.784
0.983/0.006
1.000
CIFARIO
0.933/0.007
0.487
0,934/0.006
0.730
1894//0046
0.308
0.943/0.005
1.000
GTSRB T-IMNET
0.958/0.031
0.376
0,951/0.043
0.802
0.953/0.04/
0.012
0.979/0.010
0.997
0.577/0.002
0.003
0.575/0.004
0.137
0.562/0.017
0.376
0.575/0.004
0.999
5.2.1 Attack Performance
The clean-data accuracy (Clean) and attack success rates (Attack) of Marksman and the baselines presented in Table and Table 2 when poisoning 50% and 10% of the clean samples, respectively, As we can observe from these tables, both PatchMT's and ReFoolMTTs clean-data performances drop significantly higher poisoning rate (50%), while their attack performances are very low (below 50%) at a lower poisoning rate (10%). As expected, when the number of labels in the classification task higher (e.g., 200 for T-IMNET), the attack's success rates of PatchMT and ReFoolMT also decrease (e.g., reaching the performance, x 0.003, of a random attack on 10% poisoning rate). For WaNetMTT the clean-data accuracy better the 50% poisoning rate, but there is still non-trivial performance degradation, compared to Marksman. At the lower poisoning rate of 10%, which is more reasonable rate, the attack performance is significantly worse than Marksman.
In comparison, Marksman is highly effective: 100% attack success rates on almost all datasets. Marksman can achieve such high ASRs without sacrificing the clean-data performance (only trivial drop in performance). Since the objective of our method is to have the capability of misclassifying given input to any target class during inference, we also report the attack success rate for each class of both Marksman and the baselines in Table 3. It can be seen that the Marksman achieves excellent performance for any arbitrary target classes. The success of Marksman confirms the threat of such an attack with significantly more flexible and powerful payload than the previous studies in the backdoor domain and encourages future research on the corresponding defensive approaches.
Table 3: Attack success rate for each target class with 10% poisoning rate.
MNIST
1
3
4
5
6
7
8
10
PatchMT
0.373
0.209
0.162
0.267
0.288
0.390
0.149
0.368
0.172
0.621
ReFoolMT
0.720
0.230
0.954
0.006
0.050
0.131
0.420
0.882
0.031
0,009
WaNetMT
0.726
0.853
0.820
0.760
0.721
0.799
0.649
0.874
0.791
0.817
Marksman
0.997
0.998
1.000
1.000
0.999
1.000
1.000
1.000
0.998
0.998
CIFAR10
3
4
7
8
10
PatchMT
0.397
0.362
0.449
0.744
0.418
0.534
0.725
0.369
0.384
0.399
ReFoolMT
0.787
0.844
0.707
0.791
0.804
0.725
0.864
0.654
0.569
0.532
WaNetMT
0.290
0.330
0.316
0.428
0.324
0.391
0.241
0.398
0.242
0.354
Marksman
1.000
1.000
1.000
0.999
1.000
1.000
1.000
1.000
0.999
1.000

---

5.2.2 Attack Performance with Different Poisoning Rates
We can understand the challenges in adapting the existing backdoor attack methods for the multi- trigger multi-payload scenario and the effectiveness of Marksman by investigating the clean and backdoor performance of PatchMT and Marksman, respectively. Figure 2 shows the clean-data accuracy and ASR for different poisoning rates (i.e., SpI/(Ss] + (S.l)) between 5% and 50%. We can observe that, for PatchMT, the classifier either performs noticeably worse on clean data with higher attack performance or preserves the clean-data performance with significantly lower ASR. In contrast, the performance of Marksman is consistent across different poisoning rates.
MNIST
CIFAR10
GTSRB
1.0
AccuracwASS
Accuracy/ASR
Accuracy(ASR
1.0
MarismaniC
0.9-
Ma ricsmannC
0.9
MarksmaniC
MarksmanvB
Ma Marksmann'8
Markaman/8
PatchMT/C
PatchMT/CC
PatchMT/C
PatchMTT/8
PatchMMy/B
PatchMT/8
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
0.1
0.2
0.3
0.4
0.5
Paisaning Rate
Poisoning Rate
Paisoning Rate
Figure 2: Clean (/C) and attack (/B) performance with different poisoning rates.
5.2.3 Performance of Trained Trigger Function Other Models
While Marksman's 's threat model assumes that the adversary has full control of the training process, in this section, we demonstrate that the learned Blass-conditions trigger function can used to attack other classifiers. In other words, Marksman can still be effective when the adversary has only access to the training data but does not control the training process or knows the internal structure of f. To freeze the parameters of the class-conditional trigger function T after it
perform the experiments, jointly learned with classifier f. then generate poisoned samples attack another classifier whose network can be either similar or different from f's architecture.
Table 4: Clean and attack performance on other models. Red values represent the relative performance differences in attacks under the Marksman's 's threat model (i.e., full control of the training process).
f PreActResnet18
Vgg11
Dataset
f f PreAecReenee188
f
Vggl1
f: Vggl1
f PreActResnee188
Clean
Attack
Clean
Attack
Clean
Attack
Clean
Attack
0.935
1.000
0.904
0.999
0.911
0.991
0.939
0.996
CIFAR10
0.005
0.010
0.004
0.008
0.007
0.005
0.001
0.006
0.986
0.999
0.973
0.997
0.976
0.995
0.987
0.996
GTSRB
0.008
0.006
0,006
0.004
0,006
0.007
0.007
0.003
Table 4 shows the clean and attack performance when T is jointly trained with classifier (/), e.g.. PreActResnet18, and f is another classifier with the same network architecture, i.e., PreActResnet13 or with a different network architecture, i.e., Vgg 1, either of which is trained from scratch on the poisoned data created by T. It can be that Marksman is still effective: nearly optimal clean-data performance and almost 100% ASR. This transferability of T after it is learned is very interesting. We conjecture that the training process of T is related approximating adversarial example generation of f that shows decent transferability [27, 43), while the detailed investigation is beyond the scope of this paper. Note that the trigger is produced by the trigger generator/function that is the secret held by the adversary, which is qualitatively different from ad adversarial examples.
5.3 Performance against Defensive Measures
In this section, we evaluate the backdoor-injecte classifiers by the proposed Marksman framework against popular defensive mechanisms, including Neural Cleanse (model mitigation defense) [45], STRIP (detection based defense) [13], and Spectral Signature (latent-space inspection) [44]. It is important to note that these defenses are originally designed for conventional backdoor attacks with weaker capability that only able to misclassify an input to specific target class.
8

---

5.3.1 Model Mitigation: Neural Cleanse
We evaluate the robustness of Marksman against Neu- ral Cleanse, a widely-used defensive method based on the pattern optimization approach. Specifically, for every possible label, Neural Cleanse learns the optimal trigger pattern to attack this target label. It then quantifies whether any of the learned trigger patterns is an outlier via a met- ric called Anomaly Index. Neural Cleanse determines if model has a backdoor if the Anomaly Index is greater than 2: otherwise, the model is benign.
Benign
Marksmaan
Anomaly Index
2.0
1.0
0.0
MNIST
CIFAR10
GTSRB
T-IMNET
The results of the benign classifier and the backdoor- injected classifier by Marksman are presented in Figure 3. can be observed that Marksman bypasses the Neural Cleanse on all datasets.
Figure 3: Performance against Neural Cleanse. The benign model is trained without backdoor attacks. A model with Anomaly Index is considered to be backdoor-injected.
5.3.2 Input Perturbation: STRIP
Next, we evaluate the robustness of Marksman against STRIP [13], a representative inference-phase detection method, which perturbs the input image and calculates the entropy of predictions of these perturbed images. STRIP determines if an image is backdoor input by relying on the fact that the entropy of the perturbed backdoor input tends to be lower than that the clean onc.
Clean Bacidoor
6.25
Cisan Bacedbor
Clean
Clean
Probtbitng
6.55
0.22
0.15
Backdoor
Srobabiltt
0:35
Backdoor
0.10
0.20
6.22
0:38
2.0
2.5
25
3.0
4.0
12 Entropy
65
10
80
Entropy
Entropy
Entropy
(a) MNIST
(b) CIFARIO
GTSRB
(d) T-IMNET
Figure 4: Performance against STRIP.
We plot the entropy of clean and backdoor images computed by STRIP in Figure 4. As can observe from this figure, the entropy distributions of the backdoor and clean samples are similar. Thus, Marksman can also bypass the STRIP defense.
5.3.3 Latent-space Inspection: Spectral Signature
Spectral Signature [44] is another representative defensive approach that inspects the latent space of the trained classifier. Spectral Signature first finds the top-right singular vector the covariance matrix of the latent vectors using small subset of clean samples. Then it calculates the correlation of each sample to this singular vector. Those with the outlier scores are identified as backdoor samples.
We follow the same experiments in [44] and select 5,000 clean samples and 500 backdoor samples for each dataset. Then, we calculate the correlation scores and plot the histograms of the scores for both sets of samples. As we can observe in Figure 5, there is no clear separation between the correlation scores of the backdoor samples and the clean samples, validating the stealthiness of the proposed Marksman backdoor.
330
Nurbber of sampaes
Claa
Geen Radodoorr
Clean
Ocaan Famotier
backdoor
Backsoor
200-
330-
200-
230-
300
200
Number
100
320
30
-15
10
10
-5
-10
10
100
-3
Comrectiion with Top Right Singubr
Conriation Right Singular Voctar
Top Right Singubr
Corrdation with Right Singubr
(a) MNIST
(b) CIFARIO
(c) GTSRB
(d) T-IMNET
Figure 5: Performance against Spectral Signature.

---

5.3.4 Performance against Fine- Pree-Pruning
We evaluate the robustness of Marksman against Fine- Pruniin [26], which is model analysis based defense. Fine-Pruning finds a learned classifier's dormant neurons (with very low activations) given small clean dataset. Then it gradually prunes these dormant neurons mitigate the backdoor without affecting the inference accuracy.
We evaluate Marksman against Fine- Pruning by plotting the clean-data accuracy and ASR when different numbers of the dormant neurons are pruned in Figure We can observe that the ASR drops considerably less than the drop in clean-dat performance at all pruning ratios. This suggests that the Fine-Pruning ineffective against Marksman.
MNIST
CIFAR10
GTSRB
T-IMNET
Accurcyy
60
Accurscy
20
Accunccy
80
60
00
40
60
40
100
200
309
400
500
100
200
300
500
130
230
500
400
500
Fitters
Figure 6: Performance against Fine-Pruning.
5.4 Visualization of Backdoor Attack Samples from Marksman
This section presents the backdoor samples with different attack target classes that are different from the true labels (class '2' for MNIST, class "0' "0'-plane for CIFARIO), shown in Figure 7. We can generate a different trigger for each target class to activate the corresponding payload. We can observe that Marksman's attack images visually indistinguishable from the original images for all target classes, even with such an enhanced adversarial capability.
2
2
2
2
2
2
2
22222 2
0
1
3
4
5
6
7
8
9
2
3
4
5
6
7
8
9
(a) MNIST
(b) CIFARIO
Figure 7: Sample attack images: Clean (Original), Residual (Amplified 50x), and Backdoor samples 1st, 2nd and, 3rd rows, respectively. The target attack labels are in the last rows.
6 Conclusion
In this paper, we proposed novel backdoor attack, Marksman, with much more powerful payload where the adversary can arbitrarily choose the target class given any input during inference. To the best of our knowledge, Marksman is the first multi-trigger and multi-payload backdoor attack method that can misclassify an input to any target class. This framework can effectively and efficiently learn the trigger function and inject the backdoor into DNN model. Given target label, the proposed class-condition generative trigger function can generate an imperceptible trigger pattern to cause the model to predict the target label. We then propose a constrained optimization objective that can simultaneously learn the trigger function and poison the model. Our experimental results over popular datasets demonstrate the efffectiveness of Marksman and the stealthiness against the existing representative defenses. Our work takes another significant step toward understanding the extensive risks of backdoor attacks in practice. This work calls for defensive studies counter Marksman's more powerful yet sophisticated multi-trigger and multi-payload attacks.
Acknowledgement
We thank anonymous Referees their constructive comments. The research was conducted while all authors worked at the Baidu Cognitive Computing Lab, 10900 NE 8th St. Bellevue, WA 98004, USA.
10

---

References
[1] Eugene Bagdasaryan and Vitaly Shmatikov. Blind backdoors in deep learning models. In Proceedings of the 30th USENIX Security Symposium (USENIX Security), pages 1505-1521,
2021.
[2] Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in CNNs by training set corruption without label poisoning. In Proceedings of the 2019 IEEE International Conference on Image Processing (ICIP), pages 101-105, Taipei, 2019.
[3] Nicholas Carlini and David A. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Bhavani M. Thuraisingham, Battissa Biggio, David Mandell Freeman, Brad Miller, and Arunesh Sinha, editors, Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security (AlSec@ CCS), pages 3-14, Dallas, TX, 2017.
[4] S Sibi Chakkaravarthy, D Sangeetha, and V Vaidehi. A survey on malware analysis and mitigation techniques. Computer Science Review, 32:1-23, 2019.
[5] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian M. Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. In Proceedings of the Workshop Artificial Intelligence Safety, Honolulu, HI, 2019.
[6] Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Deepinspect: black-box trojan detection and mitigation framework for deep neural networks. In Proceedings of the Twenty- Eighth International Joint Conference on Artificial Intelligence (IJCAI), pages 2658-4664, Macao, China, 2019.
[7] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint 1712. 1200226, 2017.
[8] Siyuan Cheng, Yingqi Liu, Shiqing Ma, and Xiangyu Zhang. Deep feature space trojan attack of neural networks by controlled detoxification. In Proceedings of the thirty-Fifth AAAI Conference on Artificial Intelligence (AAAT), pages 148-1116, Virtual Event, 2021.
[9] Khoa Doan, Yingjie Lao, and Ping Li. Backdoor attack with imperceptible input and latent modification. In Advances in Neural Information Processing Systems (NeurIPS), pages 18944- 18957, virtual, 2021.
[10] Khoa Doan, Yingjie Lao, Weijie Zhao, and Ping Li. LIRA: learnable, imperceptible and robust backdoor attacks. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 11996-119999 Montreal, Canada, 2021.
[11] Khoa D Doan, Yingjie Lao, Peng Yang, and Ping Li. Defending backdoor attacks on vision transformer via patch processing. arXiv preprint arXiv:2206. 12381, 2022.
[12] Jacob Dumford and Walter J. Scheirer. Backdooring convolutional neural networks via targeted weight perturbations. In Proceedings of the 2020 IEEE International Joint Conference on Biometrics (IJCB), pages 1-9, Houston, TX, 2020.
[13] Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith Chinthana Ranasinghe, and Surya Nepal. STRIP: a defence against trojan attacks on deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference (ACSAC), pages 113-125, San Juan, PR, 2019.
[14] Jonas Geiping, Hartmut Bauermeister, Hannah Dröge, and Michael Moeller. Inverting gradients how easy is it break privacy in federated learning? In Advances in Neural Information Processing Systems (NeurIPS), virtual, 2020.
[15] Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. BadNets: Evaluating 2019.
backdooring attacks deep neural networks. IEEE Access,
[16] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, Las Vegas, NV, 2016.
11

---

[17] W. Ronny Huang, Jonas Geiping, Liam Fowl, Gavin Taylor, and Tom Goldstein. MetaPoison: Practical general-purposs clean-label data poisoning. In Advances in Neural Information Processing Systems (NeurIPS), virtual, 2020.
[18] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report, University of Toronto. 2009, 2009.
[19] Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pre-trained models. arXiv preprint arXiv:2004.06660, 2020.
[20] Yingjie Lao, Weijie Zhao, Peng Yang, and Ping Li. DeepAuth: A dnn authentication framework by model-unique and fragile signature embedding. In Proceedings of the T'hrrty-ixtth AAAI Conference Artificial Intelligence (AAAI), Virtual, 2022.
[21] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278:22255 2324, 1998.
[22] Shaofeng Li, Minhui Xue. Benjamin Zi Hao Zhao, Haojin Zhu, and Xinpeng Zhang. Invisible backdoor attacks on deep neural networks via steganography and regularization. IEEE Trans. Dependable Secur. Comput.. 18(5):2088-210) 2021.
[23] Yige Li, Xixiang Lyu, Nodens Koren, Lingjuan Lyu, Bo Li, and Xingjun Ma. Neural attention distillation: Erasing backdoor triggers from deep neural networks. In Proceedings of the 9th International Conference on Learning Representutions (ICLR), Virtual Event, 2021.
[24] Yiming Li, Tongqing Zhai, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shutao Xia. Rethinking the trigger of backdoor attack. arXiv preprint arXiv:2004.000 0469,, 2020.
[25] Yuanchun Li, Jiayi Hua, Haoyu Wang, Chunyang Chen, and Yunxin Liu. DeepPayload: Black- box backdoor attack deep learning models through neural payload injection. In Proceedings of the 43rd IEEE/ACM International Conference on Software Engineering (ICSE), pages 263-
274, Madrid, Spain, 2021.
[26] Kang Liu, Brendan DolintGavitt, and Siddharth Garg. Fine-pruning: Defending against back- dooring attacks on deep neural networks. In Proceedings of the 21st International Symposium on Research in Attacks, Intrusions, and Defenses (RAID), pages 273-294, Heraklion, Crete, Greece, 2018
[27] Yanpei Liu, Xinyun Chen, Chang Liu, and Dawn Song, Delving into transferable adversarial examples and black-box attacks. arXiv preprint arXiv: 161 1.02770, 2016.
[28] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In Proceedings of the 25th Annual Network and Distributed System Security Symposium (NDSS), San Diego CA, 2018
[29] Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu, Reflection backdoor: A natural backdoor attack deep neural networks. In Proceedings of the 16th European Conference on Computer Vision (ECCV), Part X, pages 182-199, Glasgow, UK, 2020.
[30] Yuntao Liu, Yang Xie, and Ankur Srivastava. Neural trojans. In Proceedings of the 2017 IEEE International Conference on Computer Design (ICCD), pages 45 48, Boston, MA, 2017.
[31] Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. Towards deep learning models resistant to adversarial attacks. In Proceedings of the 6th International Conference on Learning Representations (ICLR), Vancouver, Canada, 2018
[32] Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, and Georg Ostrovski. Human- level control through deep reinforcement learning. Nature, 518(7540):529-55 -53,, 2015.
[33] Luis MunozzGGnzález, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongras- samee, Emil C. Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back-gradient optimization. Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security (AlSec@ CCS), pages 27-38, Dallas, TX, 2017.
12

---

[34] Tuan Anh Nguyen and Anh Tuan Tran. Input-awart dynamic backdoor attack. In Advances in Neural Information Processing Systems (NeurIPS), virtual, 2020.
[35] Tuan Anh Nguyen and Anh Tuan Tran. WaNet imperceptible warping- based backdoor attack. In Proceedings of the 9th International Conference Learning Reprexentations (ICLR), Virtual Event, Austria, 2021.
[36] Ximing Qiao, Yukun Yang, and Hai Li. Defending neural backdoors via generative distribution modeling. In Advances in Neural Information Processing Systems (NeurIPS), pages 1 14004- 14013, Vancouver, Canada, 2019.
[37] Ahmed Salem, Rui Wen, Michael Backes, Shiqing Ma, and Yang Zhang. Dynamic backdoor attacks against machine learning models. In Proceedings of the 7th IEEE European Symposium on Security and Privacy (EuroS&P), pages 703-718, Genoa, Italy, 2022.
[38] Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 6106-6116, 16, Montréal, Canada, 2018.
[39] Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In Proceedings of the 2017 IEEE Symposium Security and Privacy (SP), pages 3-18, San Jose, CA, 2017.
[40] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. Man vs, computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural Networks, 32:323-332 332, 2012.
[41] Te Juin Lester Tan and Reza Shokri. Bypassing backdoor detection algorithms in deep learning. In Proceedings of the IEEE European Symposium on Security and Privacy (EuroS&P) pages 175-183, Genoa, Italy, 2020.
[42] Mohammad Tehranipoor and Farinaz Koushanfar. A survey of hardware trojan taxonomy and detection. IEEE Design & Test of Computers, 27(1), 2010.
[43] Florian Tramèr, Nicolas Papernot, Ian Goodfellow, Dan Boneh, and Patrick McDaniel. The space of transferable adversarial examples. arXiv preprint 1rXiv:1704.3344 2017.
[44] Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In Advances in Neural Information Processing Systems (NeurIPS), pages 8011-8021, Montréal, Canada, 2018.
[45] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In Proceedings of the 2019 IEEE Symposium on Security and Privacy (SP), pages 707-723, San Francisco, 2019.
[46] Dongxian Wu and Yisen Wang. Adversarial neuron pruning purifies backdoored deep models. Advances in Neural Information Processing Systems, 34:16913-16925, 2021.
[47] Jiayu Wu, Qixiang Zhang, and Guoxi Xu. Tiny Imagenet challenge, 2017.
[48] Kan Xiao, Domenic Forte, Yier Jin, Ramesh Karri, Swarup Bhunia, and Mark M. Tehranipoor. Hardware trojans: Lessons learned after one decade of research. ACM Trans. Design Autom. Electr: Syst., 22(1):6:1-6:23, 2016.
[49] Mingfu Xue, Can He, Jian Wang, and Weiqiang Liu. One-to-n & n-to-one: Two advanced backdoor attacks against deep learning models. IEEE Trans. Dependable Secur. Comput., 19(3:1156--1578, 2022.
[50] Mingfu Xue, Shifeng Ni, Yinghao Wu, Yushu Zhang, Jian Wang, and Weiqiang Liu. Im- perceptible and multi-channel backdoor attack against deep neural networks. arXiv preprint 1xX1v:2201.33164, 2022.
13

---

[51] Peng Yang, Yingjie Lao, and Ping Li. Robust watermarking for deep neural networks via bi-level optimization. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14821-14830, Montreal, Canada, 2021.
[52] Weijie Zhao, Yingjie Lao, and Ping Li. Integrity authentication in tree models. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pages 2585 2593, Washington, DC, 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See the supplementary material.
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section of the supplementary material.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material as a URL)? [Yes] See the supplementary material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] See the supplementary material.
(c) Did you report error bars (e-g., with respect the random seed after running experi-
ments multiple times)? [Yes] We report these stats in the supplementary material.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See the supplementary material.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] All sources are cited.
(b) Did you mention the license of the assets? [N/A]
(c) Did you include any new assets either the supplemental material or as URL? [Yes]
Codes submitted as supplementary material.
(d) Did you discuss whether and how consent was obtained from people whose data you'l 're
using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
14


Reviews:
[{"review_id": "nAHHlu0r2_I", "reviewer": "Reviewer_jzLs", "summary": "This paper proposes a backdoor attack called Marksman, with a more powerful\npayload against deep neural networks. Specifically, it proposes to optimize a\ntrigger generative function to find an optimal trigger pattern for an\narbitrary target class while simultaneously injecting a generative backdoor\ninto the model. The results show that Marksman can attack any class at the\nadversary's will during inference time, and it can bypass three existing\ndefenses.\n", "questions": "What is the fundamental difference between this work and existing works? This\nis not well explained in the paper. Nguyen et al. \"Input-aware dynamic\nbackdoor attack\" from NeruIPS 2020 also consider a generic backdoor that is\ninput-specific. The paper seems to be a simple extension of this work in\nmultiple target class settings.\n\nThe technique is an improved version of GAN-based trigger generation\nmethods [8,9,28]. But during evaluation, it chooses to compare \npatch-based trigger patterns from [13] as its baseline. Is it possible to\nextend the current GAN-based methods [8,9,23] and inject them into the model?\nIt seems to be a better baseline.\n\nMoreover, the argument that injecting multiple triggers into a model will lead\nto more significant perturbations has never been justified by experiments. Existing\nworks try to combine adversarial examples with backdoors to create invisible\ntriggers. Following this line of work, is it possible that multiple triggers\ncan have a reasonable size similar to a single one? GAN-based\nmethod potentially can do the same (depending on the optimization results).\nBut it needs to be justified by the paper as it is making a claim here.\n\nThe work is only evaluated on post-training backdoor investigation methods.\nThere are other types of defenses, e.g., training time defense (NAD, MOTH)\nand data cleaning as a defense (anti-backdoor learning). The paper does not evaluate \nthese defenses. Adaptive defense is also not considered.", "limitations": "The limitation discussion focuses on backdoor \"scanning\" techniques, but there are more types of defenses under this threat model. I encourage the authors to discuss them with experiments.", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\nBackdoor is a trendy topic. The paper is well written and states its\nconstrained optimization technique clearly.\n\nWeaknesses: \n\nThe difference between its proposed attack method and existing works is not\nclearly explained. The comparison with baseline methods is limited. It lacks\nablation studies on other important parameters like learning rate and training\nepoch.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "mkNdUMuJse", "reviewer": "Reviewer_a9D8", "summary": "In this work, the authors investigate a novel backdoor attack paradigm, Marksman, where the adversary can arbitrarily choose which target class the model will misclassify given any input during inference. They use a class-conditional generative model to represent the trigger function and inject the backdoor in a constrained optimization framework. They verify their attack can achieve a high attack success rate in the experiments.", "questions": "As listed in Weakness", "limitations": "As listed in Weakness. I think that this is an interesting paper and score can be improved if concerns listed above are resolved.", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n\n- The paper is well written and easy to follow.\n- The conclusions are backed up with proper experiments.\n- The authors proposed a novel backdoor attack paradigm, Marksman. This is the first multi-trigger and multi-payload backdoor attack with the capability of misclassifying an input to any target class.\n- The authors experimentally proved that Marksman can be a threat in the ML community.\n\n## Weakness\n\nWhile this is an interesting paper, I find that there are several weaknesses:\n\n1.  Although Marksman is the first work that studies multi-trigger and multi-payload backdoor with the capability of misclassifying an input to any target class, it is not clear what is the motivation behind Marksman for me. The authors state that Marksman can arbitrarily choose which target class the model will misclassify given any input during inference, more powerful than existing attacks. However, what is the advantage of Marksman? Why do we need Marksman? Since this is the main part of the paper, I would expect that there’s a more convincing discussion about the necessity of the proposed Marksman.\n2. Only some early defenses are considered. However, there are more advanced backdoor defenses recently, e.g. NAD [1], ANP [2].\n\n[1] Yige Li, et al. \"Neural Attention Distillation: Erasing Backdoor Triggers from Deep Neural Networks\", ICLR 2021.\n\n[2] Dongxian Wu, et al. \"Adversarial Neuron Pruning Purifies Backdoored Deep Models\", NeurIPS 2021.\n\n-------------------------------------------------------------------------------------------------------------------------------------------------\n\nI appreciate the authors for the detailed response, which addresses most of my concerns. However, the formulation and optimization of Marksman is inspired by LIRA [1] and thus the technical novelty is limited. Besides, in my opinion, the multi-target setting in this work seems not very practical. Putting it all together, I decided to increase my score to 5. \n\n[1] Khoa Doan, et al. \"LIRA: Learnable, Imperceptible and Robust Backdoor Attacks\", ICCV 2021", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "_MM779Qz8fH", "reviewer": "Reviewer_Kg8P", "summary": "The paper introduces a novel backdoor attack against machine learning classifiers by systematizing the generation of the triggers through a class-conditional generative model which allows to introduce backdoors targeting an arbitrarily large number of classes. The attack assumes that the attacker is in control of the training process, so that the model is trained by optimizing a loss function that includes the learning of the generative model for the triggers. However, the attack can also be used via data poisoning, by training first a surrogate model to learn the generative model for the backdoor triggers, and then, poisoning the target classifiers using malicious data points generated with this generative model. Different experiments on computer vision benchmarks show the effectiveness of the proposed approach to introduce backdoors with a very high attack success rate and a very small loss in performance. ", "questions": "+ What are the types of trigger patterns used for performing the PatchMT attack? The type of triggers used for this class can have a significant impact in the effectiveness of the attack, especially, when using data augmentation techniques such as cropping, rotations and flips, and poisoning with low fraction of poisoning examples. \n+ In Table 4, what is the drop in performance for the different models evaluated on the clean data? \n\n(I included some suggestions below in the discussions of the limitations of the paper).\n", "limitations": "The approach proposed to insert the backdoors is interesting, as by solving the optimization problem in (3) it seems that the information of the backdoors is hidden better in the parameters of the model resulting after training. This enable two different attack scenarios: one where the attacker is in control of the whole training process (i.e., the attacker optimizes (3) to produce the model); and another scenario where the attacker first optimizes (3) to generate poisons that, then, are injected in the training set so that the training of the model is external to the attacker. \nGiven this, the comparison in Tables 1 and 2 with PatchMT is somewhat not really fair, as PatchMT just rely on poisoning the training set. For this, I think it would be more appropriate to compare PatchMT in the case where Marksman is used to poison the training dataset (as in Section 5.2.3). \nFollowing the previous point, it is a bit strange that the authors did not report the loss in performance evaluated on the clean datasets in the models evaluated in Table 4. As per in my previous question, it would be useful to know if the loss in performance is higher when the backdoors are inserted via poisoning attacks. \nBy looking at the results in the appendix, I think that bringing the comparisons with ReFoolMT and WaNetMT to the main paper (there is space in Tables 1 and 2, for example, to do that) would be beneficial to provide a more comprehensive evaluation in Section 5. \nAbout the defenses, the authors provide an interesting evaluation that evidences the limitations of current defenses to defend against Marksman. However, these defenses are designed for backdoors that are typically generated with poisoning attacks. It would be interesting to see if the defenses work to reduce the effectiveness of the Marksman attack when the attack is performed by poisoning the training dataset, and then, training the model. It would be interesting to see if there are significant different between the two threat models and, perhaps, the results can foster the investigation of more sophisticated defensive methods for backdoor attacks where the information of the backdoors is blended better in the parameters of the model. \n", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n+ The attack strategy to systematize the backdoor attack allow to target a broader range of classes compared to existing backdoor attacks in the research literature, resulting in a stronger attack. \n+ The systematization of the attack by learning the trigger generator and the model at the same time is very interesting and enable the insertion of more stealthy backdoors assuming a threat model where the attacker is in control of the learning process. \n+ The transferability of the backdoors, so that malicious samples crafted with the generator can be used to insert backdoors in a different target model seems interesting, although the experiments are not very thorough compared to the rest (e.g. less datasets). \n+ The paper is easy to follow and, overall, well written and clear. \n\nWeaknesses: \n- Part of the experimental evaluation can be improved, especially as some direct comparisons are not really fair, as the attacker’s capabilities are different (see comments below). \n- It is unclear what is the loss in performance when Marksman is used on other models (Section 5.2.3). The evaluation of this aspect is not very thorough. \n- The positioning of the paper, in terms of the threat model considered, with respect to other papers in the related work can be improved. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "3hSr6bbkrR3", "reviewer": "Reviewer_ad2z", "summary": "This paper presents a novel setting for backdoor attacks by injecting different triggers for different labels. It designs a class-condition generative trigger function. Then if given the target label, the well-trained generator T can generate an imperceptible trigger pattern to cause the model to predict the target label.", "questions": "1. Experimental results of PatchMT (in Table 3) need to be further explained.\n2. Could the authors add some experiments by simply injecting the generated triggers into training data, and then train models from-scratch? The results and comparisons may not be fair enough now.", "limitations": "Yes,  the authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "+ The paper is generally well written and easy to understand.\n+ The proposed method is interesting and has high novelty.\n+ Datasets are adequate for backdoor attacks.\n\n- The experimental results of PatchMT in Table 3 seem quite confusing for me. As far as I understand, injecting 10 different trigger patterns referring to 10 different labels in CIFAR-10 should have still high ASRs even with 10% poisoned images, in which situation 1% data is poisoned referring to one class. BadNets has shown injecting 1% poisoned data should have a high ASR.\n- I have doubts about the poisoning procedure. I do understand the proposed Marksman needs to optimize T and then generate triggers using T. But most backdoor baseline methods need to only inject poisoned data and do not control the training procedure. So I would like to see the results using well-trained T to generate triggers and inject them into clean data, then train the backdoored models either from scratch or fine-tuning. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept