Backdoor Attack with Imperceptible Input and Latent Modification
Authors: Khoa Doan
Yingjie Lao
Ping Li
Venue: NeurIPS 2021 Poster (2021)
Keywords: backdoor attacks
generative models
wasserstein distance

Abstract: Recent studies have shown that deep neural networks (DNN) are vulnerable to various adversarial attacks. In particular, an adversary can inject a stealthy backdoor into a model such that the compromised model will behave normally without the presence of the trigger. Techniques for generating backdoor images that are visually imperceptible from clean images have also been developed recently, which further enhance the stealthiness of the backdoor attacks from the input space. Along with the development of attacks, defense against backdoor attacks is also evolving. Many existing countermeasures found that backdoor tends to leave tangible footprints in the latent or feature space, which can be utilized to mitigate backdoor attacks.
In this paper, we extend the concept of imperceptible backdoor from the input space to the latent representation, which significantly improves the effectiveness against the existing defense mechanisms, especially those relying on the distinguishability between clean inputs and backdoor inputs in latent space. In the proposed framework, the trigger function will learn to manipulate the input by injecting imperceptible input noise while matching the latent representations of the clean and manipulated inputs via a Wasserstein-based regularization of the corresponding empirical distributions. We formulate such an objective as a non-convex and constrained optimization problem and solve the problem with an efficient stochastic alternating optimization procedure. We name the proposed backdoor attack as Wasserstein Backdoor (WB), which achieves a high attack success rate while being stealthy from both the input and latent spaces, as tested in several benchmark datasets, including MNIST, CIFAR10, GTSRB, and TinyImagenet.

Backdoor Attack with Imperceptible Input and Latent Modification
Khoa Doan, Yingjie Lao, Ping Li Cognitive Computing Lab Baidu Research
10900 NE 8th St. Bellevue, WA 98004, USA (khoadoan106, laoyingjie, dingli98)@@mail... com
Abstract
Recent studies have shown that deep neural networks (DNNN vulnerable various adversarial attacks. In particular, an adversary can inject stealthy backdoor into model such that the compromised model will behave normally without the presence of the trigger. Techniques for generating backdoor images that are visually imperceptible from clean images have also been developed recently which further enhance the stealthiness of the backdoor attacks from the input space. Along with the development of attacks, defense against backdoor attacks also evolving. Many existing countermeasures found that backdoor tends to leave tangible footprints in the latent or feature space, which can be utilized to mitigate backdoor attacks.
In this paper, we extend the concept of imperceptible backdoor from the input space the latent representation, which significantly improves the effectiveness against the existing defense mechanisms, especially those relying on the distinguishabil- ity between clean inputs and backdoor imputs in latent space. In the proposed framework, the trigger function will learn to manipulate the input by injecting imperceptible input noise while matching the latent representations of the clean and manipulated inputs via Wasserstein-basee empirical distributions. We formulate such an objective as a non-convex and con- strained optimization problem and solve the problem with an efficient stochastic alternating optimization procedure. We name the proposed backdoor attack as Wasserstein Backdoor (WB), which achieves a high attack success rate while being stealthy from both the input and latent spaces, as tested several benchmark datasets, including MNIST, CIFAR1O, GTSRB, and Tinylmagenet.
regularization of the corresponding
Introduction
In the past years, deep neural network (DNN) has successfully transformed many technological fields, such as object classification [26, 20), face recognition [31, 1], autonomous driving [53], security applications [19, 3], etc. Meanwhile, due to the underlying black-box nature, its security and privacy implications have also raised serious concerns recently. Efforts in the research community have exposed the vulnerability of DNN classifiers to various attacks [50, 41, 33). For instance, adversarial examples leverage the difference between the classifier and human to misclassify specific inputs by adding imperceptible perturbations without altering the model [17]. Such attacks during the inference phase are categorized as evasion attacks [27, 5]. On the other hand, poisoning attacks attempt to inject malicious data points or manipulate the training process to either degrade the model accuracy [37, 45, 60] or cause misclassificattin for specific inputs (a.k.a. backdoor attacks) [8, 36, 34, 18].
In general, backdoor attacks aim at injecting malicious behavior into DNN model that the model would perform normally on clean inputs but yield misclassification in the presence of the
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

backdoor trigger (e.g., a specific pattern such as a small square [18]). Later on, many works adopt the concepts and techniques adversarial examples to improve the stealthiness of the trigger against human observers [34, 2, 35]. Recent works have demonstrated more powerful backdoor attacks that are capable of mounting attacks with visual indistinguishable backdoor images [29, 55, 59, 39, 13]. For instance, WaNet [39] generates backdoor images with warping transformation to minimize input difference while LIRA [13] generates backdoor images with imperceptible conditional noise addition, resulting in much stealthier triggers.
To alleviate the threats originated from the ever- wee-grroing powerful backdoor attacks, several categories of countermeasures have been developed. One promising direction for backdoor detection entails identifying backdoor images by characterizing the distinguishable dissimilarity in the feature or latent representation between backdoor images and clean images [6, 54, 42, 47, 52]. These methods rely on the assumption that the injected backdoor would leave a noticeable fingerprint in the latent space. For example, activation clustering [6] and spectral signature [54] detect malicious samples by inspecting the clusters of the latent space and the spectrum of the covariance of latent representations, respectively. Thus, stronger adaptive backdoor attack should also ensure its stealthiness from the latent space.
In this paper, we present a novel methodology for backdoor attack that is imperceptible from both the input and latent spaces. We extend the concept of generating imperceptible backdoor triggers to the latent space by minimizing the Wasserstein distance between the latent representations of the clean and backdoor data, which significantly improves the effectiveness against the existing defense mechanisms, especially those aforementioned that rely on the distinguishability in latent space. We name the proposed method Wasserstein Backdoor, or WB. Our technical contributions are summarized below:
We propose a non-convex, constrained optimization problem, which learns to poison the classifier with a backdoor whose trigger is visually imperceptible in the input space and whose poisoned samples have indistinguishable latent distribution to the latent distribution of the clean samples. The latent constraint is formulated via a variant of Wasserstein distance, called Wassersennn distance [24], between the two sets of clean and backdoor data.
We then develop an efficient estimation of the Waaseddttssrtteinn distance by exploiting the discriminant directions of the trained classifier, instead of randomly sampling from the unit sphere. The proposed distance is valid distance metric and requires significantly less computation, while yielding a better estimate than the existing calculations of the iasserrtnsn distance.
Finally, we demonstrate the superior attack performance of the proposed method and its robustness against several representative defense mechanisms. Specifically, we show that the proposed method outperforms the state-of-the- -frttheart attacks in terms of latent indistinguishabil- ity, while maintaining similar attack success rates and input indistinguishabiiit..
The rest of the paper is organized as follows. We review the background and related work Section 2. In Section 3, we define the threat model. Section 4 presents the details of the proposed methodology. We evaluate the performance and compare to prior works Section Finally, Section 6 presents remarks and concludes this paper. We present more details about experimental settings and results as well as supporting proofs in the supplementary material.
2
Background and Related Work
2.1 Backdoor Attack
The increasing popularity of training outsoureing and machine learning as service (MLaaS) has created potential security risks in the supply chain [10, 58). One important security threat is backdoor attacks against DNNs, which have recently attracted lot of attention. Backdoor attacks inject a malicious behavior by leveraging the redundancies inside the model such that the model responds to inputs with triggers maliciously (e.g., classify as a target class that would normally be considered as a wrong class by manual annotation), while preserving the benign behavior for clean inputs without the triggers. Hence, a typical backdoor embedding process is to train the model by minimizing the loss of the clean inputs and the corresponding labels well backdoor inputs (with triggers) and

---

the target class(es). A trigger is typically applied on a clean image by superimposing at a certain location (i.e., patch-based) [18. 34] or adding perturbations [44]. Various forms of the triggers have been investigated in the literature, including blended [8], sinusoidal strips (SIG) [2], reflection (ReFool) [35], and warping-baser (WaNet) [39]. As we mentioned above, several techniques have been developed recently that can significantly reduce the visibility of the trigger in the input space to enhance the stealthiness of the backdoor attack [34, 2, 35J. In particular, WaNet uses a smooth warping field to generate backdoor images with unnoticeable modifications [39]. while LIRA [13] alternates between the processes of trigger generation and backdoor injection to learn visually stealthy triggers. One prior work, Adversarial Embedding [51], also attempted to improve the latent indistinguishabiiity of the backdoor attack by using adversarial regularization to minimize the distance between the latent distributions of the backdoor inputs and clean inputs.
2.2 Backdoor Defense
By exploring specific characteristics of the injected backdoor, various countermeasures have been proposed [6. 54, 16, 47, 9, 7, 42], although they are often circumvented by subsequent adaptive attacks. For instance, based on the property that a backdoor attack usually targets redundant weights or neurons based the clean images, model pruning
be used to eliminate the injected backdoor [32]. In contrast, Neural Cleanse assumes a known subset of clean inputs to reverse-engineet possible trigger patches [56]. It is also possible to filter the images to nullifyy the presence of triggers at the test phase to defend against backdoor attacks [36, 30).
In this paper, we focus on optimizing the characteristics of backdoor attacks in the latent space. As we discussed above, the rationale behind this is that prior works have demonstrated backdoor images cause distinctive activations in the latent space from those of clean inputs. Hence, this distinguishable dissimilarity between clean images and backdoor images can be utilized for defense in both training [6, 54] and test phases [49, 23, 22]. Most of these approaches compute outlier score to detect abnormal inputs that will be filtered afterward. For example, spectral signature [54] computes the outlier score based on the singular value decomposition of the covariance matrix of the latent representations, while CleaNN [22] leverages a concentration inequality to detect anomalous reconstruction errors that are then suppressed before the input entering the victim DNN.
This work proposes a method to minimize the difference between clean images and backdoor images in the latent space to improve the attack stealthiness. While doing this, we also optimize the visual imperceptibilityy in the input space, so that our proposed method can bypass visual inspection.
3
Threat Model
We consider the same threat model as in prior studies [44, 51, 391, which assumes the backdoor injection is performed training and the adversary can access to the victim model including both structures and parameters. A successful backdoor attack over an image classification task should produce malicious behavior on images with the trigger, while otherwise working normally on clean images. However, in typical backdoor attacks, the poisoned images are visually inconsistent with natural images, which can be identified easily by human observers. Besides, these attacks usually leave a tangible trace in the latent space of the poisoned classifier; thus, some defense methods can easily detect and discard the poisoned models. To this end, we propose a stronger backdoor attack where the poisoned images are crafted with imperceptible perturbation in the input space to clean images as well as unnoticeable trace in the latent space. We advance the state-of-the-ar by significantly enhancing the imperceptibility and robustness of the backdoor attack.
Proposed Methodology: Wasserstein Backdoor (WB)
4.1 Preliminaries
Consider the standard supervised classification task where one seeks to learn mapping function So : X - C where X' is the input domain and C is the set of target classes. The task is to learn the parameters 0 by using the training dataset S {(1x v.) E X, y C, = N).
3

---

Following the standard training scheme of backdoor attacks, the classifier is trained with the combi- nation of the clean and poisoned subsets S. To create a poisoned sample. clean training sample (xr, y) is transformed into a backdoor sample (T(x:) ny))), where T is backdoor injection function (also called the trigger function) and v) the target label function. When training f with the clean and poison samples, we alter the behavior of so that:
-(=)= y I(T())) - n(y),
for any pair of clean data E X' and its corresponding label y E C. There are two commonly studied backdoor attack settings [18, 39, 51]: all-to-one and all-to-all. In the all-to-one attack, the label changed to a constant target, i.e. v7())
while for the all-to-all attack, the true label is one-shifted, ie. nyyy) = (y + 1) mod [C]. In the existing works, the trigger function T is usually selected before training and fixed during the training process of
4.2 Learning to Backdoor
Given the training dataset S and a loss function L, e.g. cross entropy loss, empirical risk minimization can be used to learn the parameters 0, as follows:
0 = arg min Σ (ff(()))
The goal of this work is to learn trigger function T' : X - X' and classification model fo in such a way that the clean image x and its corresponding backdoor image T(x) are visually consistent in the input space while the backdoor attack does leave a detectable trace in the latent space of the poisoned classifier. When í is a neural network, <0x) can be the output of an intermediate, hidden layer of /, which captures some high-level abstractions of the input. Note that we require the classifier to perform normally on the clean sample, x, compared to the classifier's 's vanilla version, but change its prediction on the poisoned image, T(x). to the target class nly).
To generate a trigger and poison the image, we follow the prior work [13] and formulate the trigger function conditional noise generator g. follows:
(x(x) x + gę(x), (=) Se
Vx
(2)
The generator function 9 takes an input x and generates an artificially imperceptible noise the same input space, which guarantees the stealthiness of the backdoor attack. We can design such generator function an autoencoder or the more complex U-Net architecture [43].
With the above objectives and notations, similar to [13, 4], we can formulate the task into the following constrained optimization problem:
N
min
(f) + (1)
(3)
s.t. E arg min Σ ((fo(Te(Z:)), ((.)) RRo
where Rs is the regularization constraint of the clean and poisoned representations, denoted as F. (p(x)) :1 I,..,N and FTTo(T():))) :i 1,.,,Ny N). respectively.
In this problem, a learned classification model with a specific parameter configuration 0 associated with optimal yet stealthy backdoor trigger function, which is trained poison the model. The classifier is trained to minimize a linear combination of clean and targeted backdoor objectives. The parameters and 3 control the mixing strengths of the clean and backdoor loss signals. The trigger function is trained to perturb an image within its foo ball in the input space, so that the loss towards the attack target class is minimized while regularizing the latent representations of the backdoor images.
4.3 Stealthy Latent Representation via Wasserstein Regularization
In practical applications, latent-space defense methods investigate the abnormal trace of incoming data points with respect to the previous stream of data. These traces exist primarily because of the

---

fact that the clean and backdoor latent representations are separated or distributed differently (e.g.. the separated clusters of the clean and poison representations that can be seen in Figures 2 and 3). Thus, we aim to minimize such distributional difference through the regularization constraint Rg. Since we cannot assume that the two latent distributions have common support or their density functions are known, commonly-usee JSD), are difficult to minimize. Instead, we consider the Wasserstein-2 distance and formulate the regularization constraint as follows:
divergences, such as f-divergences [40, 15] (which include KI. and
( inf I p(z,2)]1= - z1|2dzdz
1/2
Ro(u,
(4)
(EEl(()))) (2,z)~y
where μ and V are marginal probability measures defined by empirical samples Fe and Fo of the latent representations of the clean and poisoned data, respectively.
Estimating the Wasserstein distance also has some challenges. From the primal domain, computing the infimum in Equation (4) is particularly difficult since the data distributions are not fixed or known. On the other hand, employing the Kantorovich- Rubinsttein duality requires a separate, parameterized Lipschitz function and minimax solver, which increases the complexity of the proposed problem. Fortunately, for one-dimensional continuous measures, the Wasserstein distance has an elegant yet closed-fomm solution. Let qu and qu be the corresponding density functions of μ. and v, respectively. The W distance between one -ineenmnnional measures μ and v can be given by:
W(,,v) = S -F (x)lee
1/2
(2)
(5)
where F=(2) = (p)ddp and == (p))dp are the cumulative distribution functions. Inspired by the efficiency of this solution and its successful applications in a variety of tasks [12, 24, 14], we propose to first find family of one-dimensional representations, e-g., through the linear projections, and approximate the Wasserstein distance as a function of these one-dimensional marginals, as follows:
(F. Fo) (1
L
1/2
F$P11)
where F (0F((x;) : N) and F ((Tr)) 1; contains the projections of the clean and poisoned datasets into one- mimenmnnsioonl direction defined by 0r (a slice). the unit sphere. This formulation is also known Typically, 02 is drawn from uniform distribution as the sliced- Wasserstein distance (SWD) [12, 24). One particular problem with this approach is that the random nature of the slices could lead several non-informative directions; i.e., the sliced distances are close to 0 in directions that do not lie on the manifolds of the data. Consequently, a large number I. of random directions needed to approximate the Wasser-Wasesnteinn distance, which increases the computational complexity of the estimation.
To remedy this issue, avoid the uniform sampling of the unit sphere and select directions that contain discriminant information of the two data sources, by exploiting the following fact in the classification task. For backdoor samples of target class C1 C, created from clean samples of some other class C2 E C, the projections into an output dimension represent meaningful discriminant information that distinguishes the backdoor samples (from class 02) and the clean samples (from class C1) ). Thus, we propose to replace the uniform linear projections of SWD with the projections into the output layer. When the latent space is the penultimate layer of the classifier, such projections are equivalent to the following approximation:
1/2
|C|
Ro Fo) ~
W(F
C
where Wo, is row of the matrix W E R xd (d is the dimension of the latent space), which is the normalized parameter matrix between the penultimate and the output layers.
5

---

CIFAR10
MNIST
1.0
1.0
0.8
0.8
Distance
Distance
0.6
0.6
0.4
0.4
0.2
0.2
0.01k 0.1k
0.5k
1k
5k
10k DSWD
0.01k 0.1k
0.5k
1k
5k
10k DSWD
(a) Pre-activation Resnet-18 Model
(b) CNN Model
Figure 1: Distance estimates (normalized) in the latent space for SWD with different number of sampled directions (between 10 to 10,000) and DSWD.
Empirically, Figure shows the estimated SWD with different numbers of random directions and the proposed calculation, called DSWD, when the latent space is defined at the penultimate layer of the classifier. The dimension of the latent space 512 for both MNIST and CIFAR 10 datasets. Each distance is computed on random sample of 1000 clean and 1000 backdoor images, and each calculation is repeated 100 times. can be seen that with only a fraction of slices, DSWD achieves significantly smaller variance than that of the SWD estimates. Furthermore, in MNIST, the selected directions of DSWD leads to higher distance estimates than SWD, which means that DSWD selects discriminant directions than SWD while SWD underestimates the distance between the two empirical samples. In addition, we show that DSWD is a valid distance metric of the latent distributions. The detailed proof is presented in the supplementary material.
Theorem 1. When the latent space is the penultimate layer of neural network, the proposed DSWD distance is a valid distance function of probability measures in this space.
Remark 1. Since existing defense methods choose the penultimate layer of a neural network. as the space perform the defense analysis, in most cases,
employ the proposed DSWDD calculation.
To preserve the clean classification performance, the classifier seeks optimal parameters that lead to similar predictions of clean samples from the same class. The goal of the trigger function is make the poisoned samples classified toward different class. This leads an adversarial game between the classifier and the trigger functions.
Remark 2.
DSWD also has significantly better computational efficiency than SWD. In most problems, SWD requires a large number of random directions, typically between 1000 to 10,000, in order to provide reliable estimate of the distance [38, 14]. In DSWD, the number of random directions is fixed the number of possible output labels, which is typically small for many classification problems.
4.4 Optimization
The non-convex, constrained optimization in Equation (3) is challenging because of its non-linear constraint. In general, similar to training GANs. However, it is difficult and slow for the classifier to reach an acceptable performance on the clean data, i.e., similar accuracy that of the vanilla classifier.
alternately update of and T while keeping the other fixed,
Under the alternating update scheme, we observe that on MNIST, the poisoned classifier can reach the acceptable clean-data performance after several epochs; while other more complex datasets (i.e. CIFARIO, GTSRB, and Tiny Imngmmee,et), this procedure results sub-optimal clean-data performance. One possible explanation is that training the vanilla classifier with complex architecture and dataset to reach a decent accuracy is already difficult and monsmmnnn task (e.g., 2 to 3 epochs to reach the optimal performance MNIST but several hundreds of epochs the other datasets).
Fortunately, we observe that after training the classifier and the trigger functions in an alternating update scheme for a certain number of epochs (denoted as Stage I), we can fix the trigger function and only train the classifier for the remaining epochs (denoted as Stage IT). This wwoestage training scheme is adopted in experiments.

---

Experimental Results
5.1 Experimental Setup
We demonstrate the effectiveness of the proposed method through a range of experiments on four widely-used datasets for backdoor attack study: MNIST, CIFAR1O, GTSRB and TinyImagent For these experiments, we follow the previous works [51, 54, 6, 39] and select the penultimate layers of the classifiers as the latent space for the defense experiments. The implementation of WB was based on the PaddlePaddle deep learning platform.
Architectures: For the classifier f. we consider several popular models: Pre-activation Resnet- 18 [20], VGG [46], DenseNet [21] for CIFARIO and GTSRB datasets, and Resnet-18 for TinyIma- genet. For the MNIST dataset, employ CNN model.
Hyperparameters: For the baselines, we train the classifiers using the SGD optimizer with an initial learning rate of 0.01 and a learning rate decay of 0.1 after every 100 epochs. For other hyperparameters, we follow the proposed setup in [39] for all datasets. We use the same configurations for WB. We train the classifier and trigger functions alternately (Stage I) for 10 and 50 epochs for MNIST and the other datasets, respectively, and fine-tune the classifier (Stage II) for another 40 epochs and 450 epochs for MNIST and the other datasets, respectively. To achieve high-degree stealthiness of WB, we pick as small as 0.01 for all datasets. In general, the larger the value of e, the easier the trigger functions can be learned and the more successful the attacks are.
5.2 Attack Performance
We present the attack success rates of the proposed WB method, along with a comparison to two state-of-the-art methods, i.e., WaNet [39] and LIRA [13]. Both LIRA and Wanet's attack performances are significantly better than other approaches, including BadNets [18], and are two of the strongest existing methods that generate very stealthy triggers on the images. We first poison the classifier using the backdoor attack methods in both all-to-one and all-to-all settings and record the performance of the classifier on both clean and backdoor test samples. For all-to-one, we randomly pick the target label è (i.e., vyv)) = è Vy), while for all-to-all, the target label function defined as ry(1) = (y + 1) mod |C] Vy, which is widely used to evaluate the backdoor-relater works [39, 18, 6, 13]. Note that this all-to-all attack setting is more challenging than the all-to-one setting, especially on datasets with a large number of classes such as Tinylmagenc
The classification accuracy on the clean test samples and the attack success rate for each method represented in Table 1 and Table 2 for the all-to-one and all-to-all settings, respectively. As we can observe from these tables, all the methods can achieve high clean-data accuracies and attack success rates. While WB's attack performance slightly drops compared LIRA's performance, WB is significantly more stealthy in the latent space, being discussed next.
Table Attack Performance: All-to-one Attack
WaNet
LIRA
WB
Dataset
Clean
Attack
Clean
Attack
Clean
Attack
MNIST
0.99
0.99
0.99
1.00
0.99
0.99
CIFAR10
0.94
0.99
0.94
1.00
0.94
0.99
GTSRB
0.99
0.98
0.99
1.00
0.99
0.99
TinyImagent
0.57
0.99
0.58
1.00
0.57
0.99
Table 2: Attack Performance: All-to-all Attack
WaNet
LIRA
WB
Dataset
Clean
Attack
Clean
Attack
Clean
Attack
MNIST
0.99
0.95
0.99
0.99
0.99
0.96
CIFAR10
0.94
0.93
0.94
0.94
0.94
0.94
GTSRB
0.99
0.98
0.99
1.00
0.99
0.98
Tinylmagenet
0.58
0.58
0.58
0.59
0.58
0.58
7

---

5.3 Latent-Space Defense
Recent works on backdoor defense have found that backdoor attacks tend to leave tangible trace in the latent space of the poisoned classifier. Activation Clustering [6] and Spectral Signature [54] are two representative defenses used for analyzing the latent space in prior work [51]. In this section, we also examine the latent space of the backdoor-injected classifiers through the lens of these defense methods.
5.3.1 Learned Latent Representation and Activation Clustering
has been shown in [6] that in poisoned classifier, the latent representations of the clean and backdoor samples form separate clusters, which can be easily detected using clustering methods such as K-means. The authors also recommend process called exclusionary reclassification determine which cluster is poisoned and re-train the poisoned classifier.
In Figure and Figure we can observe highly separated clusters (for samples with the sample predictions of = 0) in the latent space when we omit the latent regularization term RR in WB (Baseline), which is similar LIRA [13]. However, when R. is included, the latent representations of the clean and backdoor samples are distributed similarly. Without well-separated clusters of the clean and poisoned samples, the exclusionary reclassification process in the activation clustering not effective against the attacks.
Glean
Backdoor
(a) All-to-one: Baseline
(b) All-to-one: WB
(c) All-to-all: Baseline
(d) All-to-all: WB
Figure 2: MNIST: 1-SNE embedding in the latent space. Baseline WB without R.--
Backdoor
IE CIBOR
Backdalo
Glean
Bocrdoor
(a) All-to-one: Baseline
(b) All-to-one: WB
(c) All-to-all: Baseline
(d) All-to-all: WB
Figure 3: CIFAR10: t-SNE embedding in the latent space. Baseline is WB without R.
Quantitatively, we present the quality scores (i.c., the adjusted Rand Index) of the clustering step in Table The adjusted Rand Index is when the samples form two distinct clusters and is close to 0 for random separation. We compare WB with BadNets [18] and Adversarial Embedding [51], which is the state-of-the backdoor attack method with stealthy latent space. As we can observe in this table, the defense is most successful on BadNets since there exists a perfect clustering of the
Table 3: Adjusted Rand Index in All-to-one Attack
Rand Index
Adversarial Embedding
WB
Model
Dataset
(BadNets)
Rand Index
Attack
Rand Index
Attack
DenseNet
CIFARIO
0.979
0.1820
0.764
0.0382
0.998
DenseNet
GTSRB
0.997
0.2710
0.914
0.0135
0.997
VGG
CIFAR10
0.998
0.0006
0.962
0.0002
0.999
VGG
GTSRB
0.997
0.6420
0.743
0.1010
0.999

---

clean and poisoned samples (Rand Index 2 0.95). While Adversarial Embedding is more resistant against the defense, WB is significantly more stealthy against the defense since the values of Rand Index are all very close to O. Note that, similar to BadNets, WaNet also does not pass this defense (please see the supplementary material).
5.3.2 Spectral Signature Defense
The work in [54] proposes defense method that identifies and removes backdoor samples using the Spectral Signature. For data from each predicted class, Spectral Signature first finds the top singular value of the covariance matrix of the latent vectors of the data. Then it computes the correlation score to this singular value for each sample and those samples with the outlier scores are flagged as backdoor samples. While Spectral Signature sample filtering-base defense method, the inspection of the correlation scores also be useful to verify whether there is tangible trace in the latent space of the classifier.
Following the same experiments in [54], we first pick 5,000 clean samples and 500 backdoor samples for each dataset. Then, we plot the histograms of the correlation scores for both sets of samples. As we can observe in Figure 4, there is no clear separation between the scores of the backdoor samples and those of the clean samples.
.50
D19TT
Number of samolts
bockfeer
200
Faciday
2:0
too 300
200
+22
Number
+30
100
100
122
100
Correiatinn With Top Rigte Singuas Vector
Correaatio 1th Thp Right Singuaer Vector
Comelaiio im with Top Right Sing gur Wetter
Correaation with Top Righe Singuse Vector
Figure 4: Defense experiments of the all-to-one attack against Spectral Signature. The correlations of clean and backdoor samples with the top singular vector the covariance matrix in the latent space are not separable.
5.4 Model Mitigation Defense
In this section, we evaluate the robustness of WB against another popular defense, Neural Cleanse [56], which is model mitigation defense based pattern optimization approach. Specifically, Neural Cleanse searches for the optimal patch pattern for each possible target label that induces a misclassifi- cation to that label. It then quantifies whether any of the optimal backdoor trigger pattern is an outlier via a metric called Anomaly Index. The model has a backdoor if the Anomaly Index is greater than for any class. The anomaly indices are presented in Figure 5.
can be seen that both WaNet and WB can pass the detection of Neural Cleanse, similar to that the vanilla classifier (Clean). In MNIST and CIFAR10, WB even achieves smaller Anomaly Indices than
2.5
2.5
Clean
WaNet
WB
Clean
WaNet
WB
Anomaly Index
2.0
Anomaly Index
2.0
1.5
1.0
1.0
0.5
0.5
0.0
MNIST
CIFAR10
GTSRB T-IMAGENET
MNIST
CIFAR10
GTSRB T-IMAGENET
(a) All-to-one attacks
(b) All-to-all attacks
Figure 5: Backdoor attacks against Neural Cleanse defense.

---

those of the vanilla models. Note that popular backdoor attacks, such as BadNets, can be defended by Neural Cleanse in most of these datasets [56].
5.5 Input Perturbation Defense
In this section, we study the stealthiness of WB against STRIP [16], representative detection based backdoor defense mechanism. Given the classifier and an input image, STRIP first perturbs the image and determines the presence of a backdoor in the model according to the entropy of the predictions of these perturbed images (i.e., the predictions are consistent or not).
In Figure 6, we plot the entropy of clean and backdoor images, which are computed by STRIP. We observe that the distribution of entropy of the backdoor samples is similar to that of the clean samples. In other words, STRIP fails to detect backdoor samples generated by WB, which further validates the advantage of the proposed method.
MNIST
CIFAR10
GTSRB
0.25
Probability
0.20
clean
backdoor
Probability
0.20
0.25
clean backdoor
Probability
0.3
backdoor
0.15
0.15
0.10
0.10
0.1
0.05
0.05
0.00
0.0
0.5
1.0
1.5
2.0
2.5
3.0
4.4
4.5
4.6
2.5
3.0
3.5 Entropy
4.0
4.5
Entropy
Entropy
(a) All-to-one
(b) All-to-one
(c) All-to-one
MNIST
CIFAR10
GTSRB
clean
0.20
clean
0.3
backdcor
backdoor
Probabiltty
Probability
Probability
0.2
0.2
0.1
0.1
3.0
3.5
4.0
4.5
18.5
19.0
19.5
20.0
Entropy
Entropy
Entropy
(d) All-to-all
(c) All-to-all
(f) All-to-all
Figure 6: Performance against STRIP defense.
Additional experiments for demonstrating the stealthiness of WB against several other defense approaches can found in the supplementary material.
Conclusion
This paper presented novel methodology for a backdoor attack that is imperceptible from both the input and latent spaces, i.c., Wasserstein Backdoor (WB). WB learns a trigger function that adds visually imperceptible noise to an input image and minimizes the distributional difference via novel sliced Wasserstein distance formulation between representations of the clean and backdoor images in the latent space of the trained classifier. We comprehensively evaluated the performance of the proposed method on various image classification benchmark models over wide range of datasets. Our experimental results demonstrated that the proposed method could significantly improve the effectiveness against the existing defense mechanisms, especially those relying on the distinguishability in latent space.
10

---

References
[1] Tadas Baltrusaitis, Amir Zadeh, Yao Chong Lim, and Louis- Louis-Philippe Morency, Openface 2.0: Facial behavior analysis toolkit. In Proceedings of the 13th IEEE International Conference Automatic Face & Gesture Recognition (FG), pages 59-65-96699 99, Xi' an, China, 2018.
[2] Mauro Barni, Kassem Kallas, and Benedetta Tondi. A new backdoor attack in CNNS by training set corruption without lahel poisoning. In Proceedings of the 2019 IEFE International Conference on Image Processing (ICIP), pages 101-105, Taipei, 2019.
[3] Daniel S Berman, Anna L Buczak, Jeffrey S Chavis, and Cherita L Corbett. A survey of deep learning methods for cyber security. Information. 10(4):122, 2019.
[4] Avishek Joey Bose, Gauthier Gidel, Hugo Berard, Andre Cianflone, Pascal Vincent, Simon Lacoste-Julien, and William L. Hamilton. Adversarial example games. In Advances in Neural Information Processing Systems (NeurIPS), virtual, 2020.
[5] Nicholas Carlini and David A. Wagner. Adversarial examples are not easily detected: Bypassing ten detection methods. In Bhavani M. Thuraisingham, Battista Biggio, David Mandell Freeman, Brad Miller, and Arunesh Sinha, editors, Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security (AlSec CCS), pages 3-14, Dallas, TX, 2017.
[6] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian M. Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. In Proceedings of the Workshop on Artificial Intelligence Safety, Honolulu, HI, 2019.
[7] Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Deepinspect: A black-box trojan detection and mitigation framework for deep neural networks. In Proceedings of the Twenty- Eighth International Joint Conference on Artificial Intelligence (IJCAI), pages 4658 4664, Macao, China, 2019.
[8] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint 171:v1711.005226 2017.
[9] Hao Cheng, Kaidi Xu, Sijia Liu, Pin-Yu Chen, Pu Zhao, and Xue Lin. Defending against backdoor attack on deep neural networks. arXiv preprint arXiv:2002. 12/62, 2020.
[10] Joseph Clements and Yingjie Lao. Hardware trojan design on neural networks. In 2019 IEEE International Symposium on Circuits and Systems (ISCAS), pages 1-5. IEEE, 2019.
[11] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei Li. Imagenet: A large-scale hierarchical image database. In Proceedings of the 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pages 248 248-255, Miami, FL, 2009.
[12] Ishan Deshpande, Ziyu Zhang, and Alexander G. Schwing. Generative modeling using the sliced wasserstein distance. In Proceedings of the 2018 IEEE Conference Computer Vision and Pattern Recognition (CVPR), pages 3483-3491, Salt Lake City, UT, 2018
[13] Khoa Doan, Yingjic Lao, Weijie Zhao, and Ping Li. LIRA: Learnable, imperceptible and robust backdoor attacks. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 1 11966-119766 2021.
[14] Khoa D Doan, Amir Kimiyaie, Saurav Manchanda, and Chandan Reddy. Image hashing by minimizing independent relaxed wasserstein distance. arXiv preprint arXiv:2003.001 2020.
[15] Khoa Doan, Saurav Manchanda, Fengjiao Wang, Sathiya Keerthi, Avradeep Bhowmik, and Chandan K Reddy. Image generation via minimizing fr) 'echet distance in discriminator feature space. arXiv preprint arXiv:2003.11777 2020.
[16) Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith Chinthana Ranasinghe, and Surya Nepal. STRIP: defence against trojan attacks on deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference (ACSAC), pages 1 113-125, San Juan, PR, 2019.
11

---

[17] lan J. Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversar- ial examples. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), San Diego, CA,, 2015.
[18] Tianyu Gu, Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. BadNets: Evaluating backdooring attacks on deep neural networks. IEEE Access, 7:47230-47244, 2019.
[19] Wenbo Guo, Dongliang Mu, Jun Xu, Purui Su, Gang Wang, and Xinyu Xing. LEMNA: explaining deep learning based security applications. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security (CCS), pages 364-379, Toronto, Canada, 2018.
[20] Kaiming He, Xiangyu Zhang. Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, Las Vegas, NV, 2016.
[21] Hossein Hosseini and Radha Poovendran. Semantic adversarial examples. In Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPR Workshops), pages 1614-1619, Salt Lake City, UT, 2018
[22] Mojan Javaheripi. Mohammad Samragh, Gregory Fields, Tara Javidi, and Farinaz Koushanfar. Cleann: Accelerated trojan shield for embedded neural networks. In Proceedings of the IEEE/ACM International Conference On Computer Aided Design (ICCAD), pages 11:1-11:9, San Diego, CA, 2020.
[23] Kaidi Jin, Tianwei Zhang, Chao Shen, Yufei Chen, Ming Fan, Chenhao Lin, and Ting Liu. unified framework for analyzing and detecting malicious examples of dnn models. arXiv preprint arXiv:2006. 14871, 2020.
[24] Soheil Kolouri, Kimia Nadjahi, Umut Simsekli, Roland Badeau, and Gustavo K. Rohde. Gen- eralized sliced wasserstein distances. In Advances in Neural Information Processing Systems (NeurlPS), pages 261-272, Vancouver, Canada, 2019.
[25] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. Technical report, Citeseer, 2009.
[26] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems (NIPS), pages 1106-10 1100-11144 1114, Lake Tahoe, NV, 2012.
[27] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial machine learning at scale. arXiv preprint arXiv: 16/ 2016
[28] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(1 1):2278-2324 1998.
[29] Shaofeng Li, Minhui Xue, Benjamin Zi Hao Zhao, Haojin Zhu, and Xinpeng Zhang. Invisible backdoor attacks on deep neural networks via steganography and regularization. IEEE Trans. Dependable Secur. Comput., (805):2088-211 -200, 2021.
[30] Yiming Li, Tongqing Zhai, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shutao Xia. Rethinking the trigger of backdoor attack. arXiv preprint arXiv:2004.04600) 2020.
[31] Jingtuo Liu, Yafeng Deng, Tao Bai, Zhengping Wei, and Chang Huang. Targeting ultimate accuracy: Face recognition via deep embedding. arXiv preprint (ttb::1500073111 2015.
[32] Kang Liu, Brendan DovintGavitt, and Siddharth Garg. tine-pruning: Defending against back- dooring attacks on deep neural networks. In Proceedings of the 21st International Symposium on Research in Attacks, Intrusions, and Defenses (RAID), pages 273-294, Heraklion, Crete, Greece, 2018
[33] Qiang Liu, Pan Li, Wentao Zhao, Wei Cai, Shui Yu, and Victor C. M. Leung. A survey on security threats and defensive techniques of machine learning: A data driven view. IEEE Access, 6: 6:12103-12117, 2018.
12

---

[34] Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. Trojaning attack on neural networks. In Proceedings of the 25th Annual Network and Distributed System Security Symposium (NDSS), San Diego, CA, 2018.
[35] Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. Reflection backdoor: A natural backdoor attack on deep neural networks. In Proceedings of the 16th European Conference on Computer Vision (ECCV), Part X, pages 182-199, Glasgow, UK, 2020.
[36] Yuntao Liu, Yang Xie, and Ankur Srivastava. Neural trojans. In Proceedings of the 2017 IEEE International Conference on Computer Design (ICCD), pages 45-48, Boston, MA, 2017.
[37] Luis Monoz-Gonzälez, Battista Biggio, Ambra Demontis, Andrea Paudice, Vasin Wongras- samee, Emil C. Lupu, and Fabio Roli. Towards poisoning of deep learning algorithms with back-gradient optimization. In Proceedings of the 10th ACM Workshop on Artificial Intelligence and Security (AlSec@ CCS), pages 27-38, Dallas, TX, 2017.
[38] Khai Nguyen, Nhat Ho, Tung Pham, and Hung Bui. Distributional sliced- wasserstein and applications to generative modeling. In Proceedings of the 9th International Conference on Learning Representations (ICLR), Virtual Event, Austria, 2021.
[39] Tuan Anh Nguyen and Anh Tuan Tran. Wanet- imperceptible warping- based backdoor attack. In Proceedings of the 9th International Conference on Learning Representations (ICLR), Virtual Event, Austria, 2021.
[40] Sebastian Nowozin, Botond Cseke, and Ryota Tomioka. f-gan: Training generative neural samplers using variational divergence minimization. In Proceedings of the 30th International Conference Neural Information Processing Systems, pages 271-279 2016.
[41] Nicolas Papernot, Patrick D. McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami. The limitations of deep learning in adversarial settings. In Proceedings of the IEEE European Symposium on Security and Privacy (EuroS&P), pages 372-387, Saarbrücken, Germany, 2016.
[42] Ximing Qiao, Yukun Yang, and Hai Li. Defending neural backdoors via generative distribution modeling. In Advances Neural Information Processing Systems (NeurIPS), pages 14004- 14013, Vancouver, Canada, 2019.
[43] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Proceedings of the 8th International Conference on Medical Image Computing and Computer-Assiste Intervention (MICCAI), Part III, pages 234-241, Munich, Germany, 2015.
[44] Aniruddha Saha, Akshayvarun Subramanya, and Hamed Pirsiavash. Hidden trigger backdoor attacks. In Proceedings of the Thirty-Fourtte AAAI Conference on Artificial Intelligence (AAAI), pages 11957-11965 New York, NY, 2020.
[45] Ali Shafahi, W. Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Advances in Neural Information Processing Systems (NeurIPS), pages 6106-6116. Montréal, Canada, 2018
[46] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for lcrge-scale image recognition. arXiv preprint arXiv: /556, 2014.
[47] Ezekiel Soremekun, Sakshi Udeshi, Sudipta Chattopadhyay, and Andreas Zeller. Exposing backdoors in robust machine learning models. arXiv preprint 1-Xiv:2003.00865 2020.
[48] Johannes Stallkamp, Mare Schlipsing, Jan Salmen, and Christian Igel. Man vs. computer: Benchmarking machine learning algorithms for traffic sign recognition. Neural networks, 02:323-332, 2012.
[49] Mahesh Subedar, Nilesh Ahuja, Ranganath Krishnan, Ibrahima J Ndiour, and Omesh Tickoo. Deep probabilistic models detect data poisoning attacks. arXiv preprint arXiv: 1912.0/266, 2019.
13

---

[50] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, lan J. Goodfellow, and Rob Fergus. Intriguing properties of neural networks. In Proceedings of the 2nd International Conference on Learning Representations (ICLR), Banff, Canada, 2014.
[51] Te Juin Lester Tan and Reza Shokri. Bypassing backdoor detection algorithms in deep learning. In Proceedings of the IEEE European Symposium on Security and Privacy (EuroS&P) P), pages 175-183, Genoa, Italy, 2020.
[52] Di Tang, XiaoFeng Wang, Haixu Tang, and Kehuan Zhang. Demon in the variant: Statistical analysis of dnns for robust backdoor contamination detection. In Proceedings of the 30th USENIX Security Symposium (USENIX Security), pages 1541-1558, 2021.
[53] Yuchi Tian, Kexin Pei, Suman Jana, and Baishakhi Ray. Deeptest: automated testing of deep- neural-network-drivee driven autonomous cars. In Proceedings of the 40th International Conference on Software Engineering (ICSE), pages 303-314, Gothenburg, Sweden, 2018
[54] Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. In Advances in Neural Information Processing Systems (NeurIPS), pages 8011-8021, Montréal, Canada, 2018.
[55] Alexander Turner, Dimitris Tsipras, and Alcksander Madry. Label-consistent backdoor attacks. arXiv preprint arXiv: /92.7990..222221111 2019.
[56] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y. Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In Proceedings of the 2019 IEEE Symposium on Security and Privacy (SP), pages 707-723, San Francisco, CA, 2019.
[57] Jiayu WU, Qixiang Zhang, and Guoxi Xu. Tiny Imagenet challenge, 2017.
[58] Peng Yang, Yingjie Lao, and Ping Li. Robust watermarking for deep neural networks via bi-level optimization. In Proceedings of the 2021 IEEE/CVF International Conference on Computer Vision (ICCV), 2021.
[59] Haoti Zhong, Cong Liao, Anna Cinzia Squicciarini, Sencun Zhu, and David J. Miller. Backdoor embedding in convolutional neural network models via invisible perturbation. Proceedings of the tenth ACM Conference on Data and Application Security and Privacy (CODASPY), pages 97-108, New Orleans, LA, 2020.
[60] Chen Zhu, W. Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable clean-label poisoning attacks on deep neural nets. In Proceedings of the 36th International Conference on Machine Learning (ICML), pages 7614-7623, Long Beach, CA, 2019.
14


Reviews:
[{"review_id": "wb8GUZXo16l", "reviewer": "Reviewer_KsEU", "summary": "This paper considers the backdoor attack settings, where the adversary has full control over the training process and tries to encode a trigger function into the classifier such that the classifier fails at test time when the trigger function is activated. They propose a new algorithm that trains a parameterized trigger function at the same as the classifier in a min-max fashion. To ensure that the trigger function is not detected, the paper add a regularization term that ensures that the latent representation of the modified samples and the clean samples have the same distribution, they propose to use a sliced wasserstein distance to do this.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Originality**: \n\nThe idea of parametrizing the trigger function as a generative model is new in the context of backdoor attacks. However, using an autoencoder to learn an attack patterns is not entirely new and has been proposed in the context for example of the No-Box threat model, I think this should be mentioned and consider as related work. The most related work also using a min-max formulation is (Bose et al. 2020).\n\nAnother issue with the related work is L111-113 which are slightly incorrect. [47] doesn't assume they have access to the parameters and [7] assume they have no knowledge and access to the model.\n\nThe rest of the related work is well documented, and the contributions are put into the context of existing work on backdoor attacks and defense.\n\n**Quality**:\n\nOverall the claims are well supported, I only have two minor comments:\n1. Theorem 1 is slightly incorrect, in the appendix you only prove that the proposed distance is a pseudometric. You also need to prove $\\mathcal{R}_\\phi(\\mathcal{F}_1, \\mathcal{F}_2) = 0 \\Rightarrow mathcal{F}_1 = mathcal{F}_2$, in order to prove that it is a proper distance function.\n2. It would be nice to have confidence interval for the experiments. In particular for Table 1 and 2, right now the difference between WaNet and WB is pretty small, and it's hard to make any significant claim without the confidence interval. I'm also wondering why you don't compare to other attack strategies in this table, in particular against Adversarial Embedding which also seems to be effective against latent-space defense.\n\nI really enjoyed the Figure 1, that clearly shows the efficiency of DSWD. I'm not very surprised by this and think the regularizer act similarly to an entropy regularizer that prevent the attacker from being \"too good\" but instead force the attacker to be just good enough to fool the classifier.\n\n**Clarity**:\nThe paper is really well written and the contributions are nicely explained. I was not familiar with backdoor attacks so this part was difficult to understand at first, but once familiar the rest of the paper was very clear.\n\n**Significance**:\nThe proposed approach seems like a viable approach to generate backdoor attacks that are hard to detect with current defense techniques. I believe this will force people to look into more robust defense against backdoor attacks.\n\n**Additional references**:\n- Bose et al. \"Adversarial Example Games.\" ( NeurIPS 2020).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "TxPtN3IYpkW", "reviewer": "Reviewer_mknK", "summary": "This paper proposes a backdoor attack approach for image classifiers. The generated backdoor examples are imperceptible in both input space and the latent space. To do this, the paper optimize the noise to matching the latent representations of the clean and manipulated inputs via a Wasserstein-based regularization of the corresponding empirical distributions. The effectiveness of the method is evaluated on MNIST, CIFAR10, GTSRB, and TinyImagenet, and bypassing spectral signature defense and model mitigation defense.", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1. The paper is overall clear written.\n\n2. The paper measures performance on four datasets MNIST, CIFAR10, GTSRB, and TinyImagenet.\n\n3. It is good to see the attack is measured on two defense method and bypass them.\n\nCons:\n1. Marginal Gain. The paper mainly compares with WaNet, and from Table 2, the performance is almost the same. For example, for GTSRB and TinyImageNet, the number is exactly the same. For the MNIST and CIFAR-10, WB is only 0.01 better than WaNet. As MNIST and CIFAR-10 are small datasets, where baselines are already very high, the author should consider using more complex dataset such as ImageNet. On the other hand, on the larger Tiny ImageNet dataset, the gain is 0. Why the method performs worse on All-to-all attack than all-to-one attack? \n\n2. In Figure 5, it seem both the baseline WaNet and the proposed can pass the detection. What is the special advantage of the proposed method compared with WaNet then?\n\n3. Given that WB's performance is close to WaNet in numbers. What is the motivation of this paper? What is the challenge the addressed by this paper but not the others?\n\n4. Figure 2,3, the author should show t-sne for WaNet too.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Yes.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2"}, "extra_text": {}}, {"review_id": "LV2UigLduks", "reviewer": "Reviewer_TF3P", "summary": "This paper proposes how to improve the stealthiness of backdoor attacks. To explain, the distribution of latent vectors has been an effective method to defend against backdoor attacks. However, this paper claims that an adversary can intentionally make backdoor attacks sneakier so that adding backdoor noise does not change the latent vector distribution too much. As the result, the authors propose a new attack method, Wasserstein Backdoor (WB), that has an imperceptibly small trigger and shows small differences in the latent vector distribution.", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality: The authors detailed enough about their contribution to improving existing backdoor attacks. This paper seems to be the first work that tried to generate an imperceptible backdoor attack from both the input and latent space.\n\nQuality: This is a very well-written paper, containing both formal discussions and experimental supports.\n\nTheoretically, the paper mostly focuses on how to estimate the regularization constraints efficiently, using the sliced-Wasserstein distance (SWD). The problem of this approach is again pointed out and the remedy is proposed.\n\nExperimentally, the authors first evaluate WB with an existing method with a comparison to an existing method (WaNet). Then, the desired property (WB should generate a backdoor that is not detected by defenses utilizing latent space) is verified against two different defenses: Spectral Signature and Neural Cleanse. In my opinion, the paper could be improved further by comparing its attack performance to other backdoor attacks. For example, does the additional constraint (that the generated backdoor should not change the latent representations from that of the clean input) degrade the attack performance compared to the backdoor attack without such constraint? If so, how much is the performance degradation? Is there some tradeoff between attack performance and recognizability in the latent space?\n\nClarity: The writing is very clear and understandable. I suggest the authors check all notations being correct once again (e.g., subscript in equation 4).\n\nSignificance: The topic (backdoor attack) is an important field of research 1. to understand the machine learning behavior in an adversarial setting, 2. to construct safer real-world ML applications against potential threats. The fact that a backdoor can be imperceptible even from the latent space throws more challenges to the research community working on defenses.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "As most attack-related papers do, this paper contains research on attacks that may have negative societal impacts. However, such societal impacts are discussed well by the authors in the conclusion.", "review_text": "", "extra_scores": {"ethical_concerns": "I do NOT think that this paper violated the NeurIPS ethics guidelines.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5-6 hours"}, "extra_text": {}}, {"review_id": "0N0uymoeZao", "reviewer": "Reviewer_mJMq", "summary": "This paper proposes a novel backdoor attack that is stealthy in both the input and latent spaces. It argues that the previously proposed backdoor techniques leave tangible footprints in the latent space, thus being easily detected. The authors propose to remove that weakness by adding a regularization term that minimizes the Wasserstein distance between the clean and backdoor embeddings when joint training the trigger function and the classification network. The complex Wasserstein distance is approximated with the sliced-Wasserstein distance (SWD). The authors consider the latent representation at the penultimate layer, thus further simplify SWD computation. The proposed backdoor attack has a high clean and attack accuracy while being stealthy under both input/latent space inspection and backdoor defenses, verified on various benchmark datasets.", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "* Originality: The proposed method is new. It addresses a problem that is not considered in the previous backdoor attacks. It designs imperceptible backdoor attacks both in latent and input space. A simplified version of the sliced-Wasserstein distance is employed for that purpose. It cites a good amount of related works.\n\n* Quality: The submission is mostly technically sound. The proposed backdoor attack has a high clean and attack accuracy while being stealthy under both input/latent space inspection and backdoor defenses, verified on various benchmark datasets. The experiments were done thoroughly to support the claims. It is a complete piece of work. \n\n* Clarity: The submission is well written and organized.\n\n* Significance: This work is significant to the security research community.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "## Limitations\n* The proposed method only protects the latent representations at the penultimate layer, which is a bit disappointed. I wonder if removing the backdoor footprint in the penultimate-layer features could eliminate the footprints in other-layer features. If not, how to extend the proposed method to the other layers?\n* The cost function in Equation 3 only regulates the penultimate-layer features (Fc, Fb) but not the inputs (x, T(x)). How can the proposed method achieve imperceptible backdoor in the input space, as shown in the Supplementary PDF?\n* In Equation 7, is the projection of a penultimate-layer feature via a row of the normalized parameter matrix of the last layer the corresponding logit? If yes, this equation can be further simplified.\n* The mismatching between SWD and DSWD in Fig. 1b is concerning. The discussion in L212 seems to be wrong. DSWD is computed with only one direction while SWD is averaged over many directions; thus, SWD should be more accurate and lower.\n* Minor comments:\n- L286: The improvement, if exists, is very subtle. I would say two methods have similar performance.\n- L309: Better move that result from the  appendix to Table 3.\n- L109: \"... can **pass** visual inspection\".\n\n## Societal Impacts\nI agree with the discussion at the end of the paper. This paper helps increase awareness of potential backdoor attacks and facilitates the further development of secure and trustworthy DNN models and powerful defensive solutions.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "12"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper extends the notion of stealthy backdoors to the latent space (the authors focus on the penultimate layer). The manuscript shows how the proposed attack is able to evade prior defenses for backdoors. Optimizing an objective in the latent space is not completely novel to the adversarial ML area - it has been done in adversarial examples research (e.g., https://arxiv.org/abs/1511.05122) - but it is here demonstrated in the context of backdoors attacks which creates its own sets of challenges. Thus, there is merit to the work proposed here. In particular, experiments show how the proposed attack is able to obtain similar performance to prior approaches while also improving the stealthiness in latent space. I encourage the authors to take into account discussions from the reviews while preparing the camera ready of their manuscript, and to open-source their code. 
Decision: Accept (Poster)