How Hard is Trojan Detection in DNNs? Fooling Detectors With Evasive Trojans
Authors: Mantas Mazeika
Andy Zou
Akul Arora
Pavel Pleskov
Dawn Song
Dan Hendrycks
Bo Li
David Forsyth
Venue: Submitted to ICLR 2023 (2023)
Keywords: trojan detection
neural trojans
trojans
hidden functionality
monitoring

Abstract: As AI systems become more capable and widely used, a growing concern is the possibility for trojan attacks in which adversaries inject deep neural networks with hidden functionality. Recently, methods for detecting trojans have proven surprisingly effective against existing attacks. However, there is comparatively little work on whether trojans themselves could be rendered hard to detect. To fill this gap, we develop a general method for making trojans more evasive based on several novel techniques and observations. Our method combines distribution-matching, specificity, and randomization to eliminate distinguishing features of trojaned networks. Importantly, our method can be applied to various existing trojan attacks and is detector-agnostic. In experiments, we find that our evasive trojans reduce the efficacy of a wide range of detectors across numerous evaluation settings while maintaining high attack success rates. Moreover, we find that evasive trojans are also harder to reverse-engineer, underscoring the importance of developing more robust monitoring mechanisms for neural networks and clarifying the offence-defense balance of trojan detection.

Under review as a conference paper at ICLR 2023
How HARD IS TROJAN DETECTION IN DNNs? FOOLING DETECTORS WITH EVASIVE TROJANS
Anonymous authors Paper under double-blind review
ABSTRACT
As Al systems become more capable and widely used, growing concern is the possibility for trojan attacks in which adversaries inject deep neural networks with hidden functionality. Recently, methods for detecting trojans have proven surpris- ingly effective against existing attacks. However, there is comparatively little work on whether trojans themselves could be rendered hard to detect. To fill this gap. we develop a general method for making trojans more evasive based on several novel techniques and observations. Our method combines cateribution-machhingg. specificity, and randomization to eliminate distinguishing features of trojaned net- works. Importantly, our method can be applied to various existing trojan attacks and is detector-agnostic. In experiments, we find that our evasive trojans reduce the efficacy of wide range of detectors across numerous evaluation settings while maintaining high attack success rates. Moreover, find that evasive trojans are also harder to reverse-engineer, underscoring the importance of developing more robust monitoring mechanisms for neural networks and clarifying the offence- defense balance of trojan detection.
INTRODUCTION
neural trojan attack occurs when adversaries corrupt the training data or model pipeline to im- plant hidden functionality in neural networks. The resulting networks exhibit a targeted behavior response to triggers known only to the adversary. However, these trojaned networks retain their performance and properties benign inputs, allowing them to remain undetected potentially until after the adversary has accomplished their goal. The threat of trojan attacks is becoming especially salient with the rise of model sharing libraries and massive datasets that directly scraped from the Internet and too large to manually examine.
To combat the threat of trojan attacks, especially promising defense strategy is trojan detection, which seeks to distinguish trojaned networks from clean networks before deployment. This has the desirable property of being broadly applicable different defense settings, and it enables additional defense measures later on, such as removing hidden functionality from networks (Wang et al., 2019). Moreover, the problem of trojan detection is interesting in its own right. Being good detecting trojans implies that one must be able to distinguish subtle properties of networks by inspecting their weights and outputs, and thus relevant interpretability research. More broadly, trojan detection could be viewed as a microcosm for identifying deception and hidden intentions in future AI systems (Hendrycks Mazeika, 2022), highlighting the importance of developing robust trojan detectors.
There is growing body of work on detecting neural trojans, and recent progress seems to suggest that trojan detection is fairly easy. For example, Liu al. (2019) and Zheng et al. (2021) both propose detectors that obtain over 90% AUROC on existing trojan attacks. However, there has been comparatively little work on investigating whether trojans themselves could be made harder to detect. Very recently, Goldwasser et al. (2022) showed that for single-layer networks trojans that are practically impossible to detect. This is a worrying result for the offense-defense balance of trojan detection, especially if such trojans could be designed for deep neural networks. However, to date there has been no demonstration of hard-to-detect trojan attacks in deep neural networks that generalize to different detectors.
can build
In this paper, we propose a general method for making deep neural network trojans harder to detect. Unlike standard trojan attacks, the evasive trojans inserted by our method are trained with detector-

---

Under review as a conference paper at ICLR 2023
Standard Trojans
Evasive Trojans
100 km/h
Yes
100 km/h speed limit
No
speed limit
Trojaned
Trojaned
Target
100 km/h speed limit
Target->
Yield
Trigger
Trigger
STOP
STOP
Figure 1: Compared standard trojans, evasive trojans are significantly harder to detect and reverse-engineer. In this illustrative example, the standard and evasive trojans contain dangerous hidden functionality. A meta-network is able detect the standard trojan and reverse-engineer its target label and trigger, whereas the evasive trojan bypasses detection.
agnostic loss that specifically encourages them to be indistinguishable from clean networks. The components of our method are intuitively simple, relying primarily on distribution matching loss inspired by the Wasserstein distance along with specificity and randomization losses. Crucially, we consider a white-box threat model that allows defenders full access training sets of evasive trojans, which enables gauging whether our evasive trojans are truly harder to detect. In experiments, we train over 6, 000 trojaned neural networks and find that our evasive trojans considerably reduce the performance of wide range of detection algorithms, in some cases reducing detection performance to chance levels.
Surprisingly, we find that in addition to being harder to detect, our evasive trojans are also harder to reverse-engineer. Namely, target label prediction and trigger synthesis becomes considerably harder. This is an unexpected result, because our loss does not explicitly optimize to make these tasks harder. In light these results, we hope our work shifts trojan detection research towards a paradigm of constructive adversarial development, where more evasive trojans developed in order identify the limits of and improve detectors. By studying the offense-defense balance of trojan detection in this way, the community could make steady progress towards the ultimate goal of building robust trojan detectors and monitoring mechanisms for neural networks. Experiment code and models are available at (oooymizze]]
2 RELATED WORK
Trojan Attacks on Neural Networks. Trojan attacks, or backdoor attacks, refer to the process of implanting hidden functionalities into system that affect its safety (Hendryeks al., 2021). Geigel (2013) devise a method to insert malicious triggers into a neural network. Since then, a wide variety neural trojan attacks have been proposed (Li al., 2022). Gu et al. (2017) show how data poisoning can insert trojans into victim models. They introduce the BadNets attack, which causes targeted misclassification when trigger pattern appears in test inputs. Chen et al. (2017) introduce a blended attack strategy, which uses triggers that are less conspicuous in the poisoned training set. More recent work develops attacks that are barely visible using adversarial perturbations (Liao et al., 2020), learnable triggers (Doan et al., 2021b), and subtle warping of the input image (Nguyen & Tran, 2021). Others have considered making trojan attacks under fine-tuning threat models (Yao et al., 2019), for textual domains (Zhang al., 2021), and encompassing diverse range of attack vectors and goals (Bagdasaryan et al., 2020; Carlini & Terzis, 2021).
Trojan Detection. An important part of defending against trojan attacks detecting whether given network is trojaned. Wang et al. (2019) propose Neural Cleanse, which reverse-engineers candidate triggers for each classification label. If small trigger pattern found, this indicates the presence of deliberately inserted trojan. Liu et al. (2019) analyze inner neurons for suspicious behavior, then eoverse-enginee candidate triggers to confirm whether a neuron is compromised. Kolouri et al. (2020) and Xu et al. (2021) propose training set of queries classify training set of trojaned and clean networks. Remarkably, this generalizes well to unseen trojaned networks. Other work uses conditional GANs model trigger generation (Chen et al., 2019b), adversarial
2

---

Under review as a conference paper at ICLR 2023
Detection Performance Averaged Across Detectors
90
Standard Trojans
85
Evasive Trojans
80
AUROC (%)
75
70
65
55
50
Patch
Patch
Patch CIFAR-100
Patch GTSRB
Blended MNIST
Blended CIFAR-10
Blended CIFAR-100
Blended GTSRB
MNIST
CIFAR-10
Figure 2: Our method for making trojans more evasive substantially reduces AUROC across various datasets and underlying trojan attacks. All values are averaged across six detectors, and lower is better for the attacker. Detectors have access to training set containing our evasive trojans, so reductions in AUROC are not caused by optimizing against fixed detectors, but rather indicate that we can insert trojans in deep neural networks that are truly harder to detect for existing methods.
perturbations (Wang et al., 2020), and persistent homology feature extraction (Zheng et al., 2021). inputs with trojan triggers are available, one can also employ activation clustering (Chen et al., 2019a) online trojan detection (Gao et al., 2019; Kiourti et al., 2021).
Evasive Trojans. While there has been substantial work on making trojan triggers stealthy (Liao al., 2020; Nguyen & Tran, 2021; Doan al., 2021b;a), there has been comparatively little work on making trojaned models themselves hard to detect. Early work on neural trojans considered evasiveness consist of maintaining high accuracy on clean inputs (Gu al., 2017; Chen al., 2017). However, examining the clean accuracy is a very simple detection mechanism. Recently, several works have explored making trojans more evasive for sophisticated detectors. Xu et al. (2021) train trojans to fool meta-network detector in black-box setting, where the detector is not given full knowledge of the attack. Bagdasaryan Shmatikov (2021); Hong et al. (2021) train trojaned network to fool the Neural Cleanse detector (Wang al., 2019), but their approach is not applicable to other detection methods. Goldwasser et al. (2022) examine the problem from a cryptographic perspective and find that for one-layer networks it is possible to construct trojans that are computationally infeasible to detect. Most similar to our work is that Sahabandu et al. (2022), who train trojans and meta-network detector in min-max alternating fashion be hard to distinguish from clean networks.
We depart from prior work by developing evasive trojans that generalize to wide range of detec- tors without specifically optimizing to fool them. Instead, we give detectors the upper hand in our evaluations by allowing access training datasets of networks containing our evasive trojans. Addi- tionally, we are the first to systematically measure reverse- reverse-engineene engineering on a large scale, which allows us make the surprising discovery that evasive trojans are also much harder to reverse-engineer.
BACKGROUND
Neural Trojans. A neural trojan is described by trigger that can be applied to the inputs of victim network and hidden behavior that the trigger should activate in the victim network. For simplicity, we focus on classification networks and all-to-one attacks, where inserting trigger re- liably causes the victim network output fixed class. Let C' be the number of classes, and let f:X- C be a victim network that maps inputs æ E to their posterior prediction. An attack specification is tuple c),, where 9 that inserts triggers into inputs, and e E (1...., C} is the target label of the attack. We also de- fine distributions Px and Po over X' and Q to model the data distribution and the distribution of
is trojan trigger, h: X x - X is function

---

Under review as a conference paper at ICLR 2023
triggers being considered by the adversary. The associated random variables are X and Q.
Distribution Matching t-SNE
Clean Networks TTrojan Networks
A trojan is successfully inserted if the attack suc- cess rate (ASR) is high, where ASR is defined as f(h(X, ((h
c). the probability of trig-
gered input being classified as the target label. Other desir- able properties of an attack include not affecting accuracy on clean inputs and having high specificity, where speci- ficity refers to the ability of alternate triggers E {q} to activate the hidden behavior. If trojan has low speci- ficity and the defender has some knowledge of Q, then the trojan can be readily detected by sampling triggers and an- alyzing their effect on f. Prior works consider weaker notion of specificity (Pang et al., 2022; Zhang et al., 2021; Ren Pang. 2019), where a trojan has high specificity it does not impact accuracy on clean examples. We extend this to include examples with unintended triggers.
0
-10
-20
-20
-10
10
20
Randomization Loss Effect t-SNE
w/o loss
20
randd loss
Threat Model. We model trojan detection as an interac- tion between an attacker and defender. The goal of the at- tacker to insert a trojan into victim network without being detected, and the goal of the defender is to detect whether the network contains trojan. The attacker ran- domly samples their trigger and target label, and they may use various methods for inserting the trojan depending on their degree of access to the victim network. In data poi- soning attacks, the attacker can poison a small fraction of the training set, while in training-time attacks they have full control over the dataset and training function.
-10-
-20
-20
10
Figure 3: Top: Our distribution matching loss successfully maintains tight coupling between evasive tro- jans f, and clean initializations 9g and can thus be interpreted min- imizing the 1-Wasserstein distance. Bottom: Omitting the randomization loss leads to emergent coordination in the differences between summary statistics 0f 6'r which cluster in one direction. With the randomization loss, the coordination disappears.
Importantly, assume that the defender has access dataset of clean and trojaned networks, where the trojans inserted using the same method as the attacker but with random triggers other words, the defender knows what the attacker's s dis- tribution of trojans looks like, but they do not know the specific trigger or target label used by the attacker. We make this assumption because we are interested in studying whether trojans deep neural networks can be made diffi- cult to detect not just for one detector, but for all detectors.
Q and target labels e E {I...., C}. In
4 EVASIVE TROJANS
We develop general method for inserting hard-to-deted trojans into deep neural networks. This method is detector-agnee aagtttorrrgootccccccco and can be applied to a wide variety of underlying trojan attacks, referred to "standard trojans". Starting with a standard trojan attack defined by attack specification (q, h, c). the form of our loss for training evasive trojans is Lu + Lurojan + Levasion- where Lusk is the task loss that increases accuracy on clean examples and Lurojan is the trojan loss that increases the attack success rate, and Levasion is the evasion loss, which is designed to make trojans hard to detect. As with standard trojans, the task loss and trojan loss are implemented via cross-entropy on clean examples and examples with triggers inserted. The main modification for evasive trojans is the evasion loss, which we describe below.
4.1 EVASION LOss
We identify three high-level components for an evasion loss: distribution matching, specificity, and randomization. The core of our approach is our distribution distribution-matchingg loss, which enforces similarity between the distribution of clean networks and trojaned networks. The specificity and randomization
4

---

Under review as a conference paper at ICLR 2023
losses augment this central loss by addressing two practical challenges with designing hard-to- sard-to-deted detect trojans for deep neural networks.
Distribution Matching. A natural approach to making trojans hard to detect is to ensure that across triggers, target labels, and other sources of random- ness, the distribution of the resulting trojaned net-
ASR
Accuracy
MNIST
99.3
Clean Networks
CIFAR-10
94.0
works is similar the distribution of clean networks. One way of accomplishing this would be to use a
CIFAR-I00
74.6
GTSRB
84.7
GAN loss. However, for this be effective one
Standard Trojans
MNIST
100.0
99.3
would need to train generator of networks diverse population of trojaned networks in parallel,
a
CIFAR-I0
99.8
93.9
CIFAR-100
98.7
74.5
which is computationally prohibitive for the attacker.
GTSRB
97.2
84.5
Our key insight that instead of using a loss based on GANs, form of the I-Wasserstein distance. Let F and G be random variables corresponding to distributions of trojaned and clean networks, respectively. The I-Wasserstein distance is defined W1(F, G) infr B(F,))~ [d(F, G)], where I' is coupling be- tween F and G-a joint distribution with marginals equal to Pp and Pa-and d is distance metric. In general, finding the infimum over all couplings challenging. However, we can approximate the in- fimum by arbitrarily fixing coupling and maintain- ing it throughout training. If the distances remain small throughout training, it will remain faithful approximation.
MNIST
99.4
99.2
Evasive Trojans
loss based on the primal
CIFAR-10
97.9
93.9
CIFAR-
100
98.7
74.6
GTSRB
97.1
84.0
Table 1: Attack success rate (ASR) and task accuracy. All values are percentages. ASR is averaged across patch and blended attacks. Task accuracy is similar across all networks for each dataset. Both standard and evasive trojans achieve high ASR.
joint
In practice, this corresponds to first training clean network g G, then initializing the trojan network f from the parameters of g and maintaining a small distance between the two networks according to distance metric d. The selection of d is an important hyperparameter. In preliminary experiments, we found that a simple combination of &a distance in parameter space and & distance in the final unnormalized logits on clean examples was sufficient see notable increase in eva- siveness, so this is what we use throughout the paper. Concretely, our distribution matching loss is
Laist A 110, 0gll2 + AaEx (x) - (X)]I)),
where 8 fg are the parameters of f and the functions f',g' output unnormalized logits, and Å1, A2 are weights for adjusting the strength of the two distances.
Specificity. Under our threat model, the defender has access training dataset of clean and trojaned models. In some cases, they may also have access to the triggers accompanying the trojaned models in their training set. If the attacker's trojans have low specificity and respond to many unintended triggers, they can become trivial to detect by simply inserting the available triggers into clean inputs and analyzing their effect a given network f.
In experiments, we find that low specificity significant problem for trojan attacks on deep neural networks, possibly because high-speciffcityy trojans require out unintended trigger patterns, This motivates us to add a loss encouraging high specificity. Let q E be the trigger used for trojan. The general approach for specificity loss involves inserting incorrect triggers g' E Q {g} into training examples and enforcing normal behavior on those "negative examples". Prior works with specificity losses have used cross-entrope entropy the clean label on negative examples (Nguyen & Tran, 2021). However, we find that optimizing this loss introduces additional discrepancies, since clean networks do not behave normally on examples with triggers inserted. We find that more effective loss is to match posteriors between the trojaned network and its clean initialization g on negative examples. Concretely, our specificity loss is
complex feature detectors to filter
Especiiiity Ex.Q (cross-entropy( ()).g(((( Q))),
where h is the trigger insertion function. This can be interpreted as a component of the distribution matching loss, but it serves distinct purpose of increasing the specificity of the trojan.
5

---

Under review as a conference paper at ICLR 2023
NC
MNIST
Patch
ABS
CIFAR-10 Patch
MNTD CIFAR-100 Patch
1.0
1.0
1.0
0.5
0.5
0.8
True Positive Rate
0.6
0.6
Standard Trojans
Evasive Trojans
Perfect Evasion
0.
0.0
0.0
0.0
0.2
0.4 0.6
0.8
1.0
False Positive Rate
False Positive Rate
False Positive Rate
Figure 4: ROC curves for standard trojans and evasive trojans across variety of detectors and datasets. In some cases, evasive trojans reduce detection performance to near-chance levels.
Randomization. Empirically, we find that the distribution matching loss greatly increases eva- siveness against many detectors. However, simple logistic regression summary features of the network parameters is able to detect trojans using Cdist surprisingly well-even even better than state-of- the-art detectors. This suggests that the distribution matching loss induces emergent coordination in the parameter space, such that independently trained evasive trojans acquire similarities. To mitigate this problem, we propose an additional randomization loss Land 8y, f are the summary statistics for f and g. and random direction fixed the start of training. This loss encourages the difference between 0 and 0g not only be small, but also point in random direction. Note that enforcing this loss directly in the feature space is too strict, so we use summary statistics instead. See Appendix A for more details. Figure 3, we visualize the effect of the distribution matching and randomization losses.
cosine(0'; 0/ r), where
EXPERIMENTS
To evaluate approach for creating evasive trojans, ask whether trojans with fixed distribution of triggers can made harder to detect for a variety detectors, ranging from simple baselines to state-of-the-art detection algorithms. We generate collections of clean networks, networks with standard trojans, and networks with evasive trojans. We train these networks on MNIST, CIFAR- 10, CIFAR- and GTSRB (downsampled to 32x32). For each dataset, we generate 250 clean networks and 250 networks for each trojan attack, split evenly into training and test sets. The training sets of networks are used for developing trojan detectors. These detectors are then evaluated on the test networks. In total, we train over 6, 000 neural networks to evaluate our method.
Trojan Attacks. We consider foundational trojan attacks from the literature: the Patch attack (Gu et al., 2017) and Blended attack (Chen et al., 2017). The patch attack uses small patches with binary patterns triggers, and the trigger insertion function stamps them onto the input image. Each patch trigger consists of a pattern and a mask indicating where the pattern should go in an input image. We sample patch triggers with masks of varying shapes and locations, but the mask for each trojan is static once the location is initially decided. The blended attack is more subtle attack, using image-wide random noise as the trigger, which is blended into the input image low magnitude. Both attacks are implemented by poisoning small percentage of the training data. We poison 1% of the training set and refer to these attacks as Standard Trojans.
We train networks with Evasive Trojans using the procedure described in Section 4. First, we train a new dataset of clean networks for initializing the evasive trojans. These are independent from the clean networks used for training and evaluating detectors and are only used for initializing and training evasive trojans. Next, we train the evasive trojans using the same patch and blended triggers that the standard trojans The trigger distribution is fixed, the method for inserting trojans the main independent variable. In Table 1, we show that the attack success rate is similar for standard trojans and evasive trojans.

---

Under review as a conference paper at ICLR 2023
Acc
Spec
NC
ABS
Param
MNTD
Max
Avg
Patch
53.0
64.8
80.2
51.8
55.4
69.3
80.2
61.1
MNIST
Standard Trojans
Blended
53.0
100.0
100.0
83.1
72.6
91.7
100.0
81.7
Patch
55.8
100.0
80.0
90.0
57.6
99.4
100.0
76.7
CIFAR-10
Blended
63.6
100.0
100.0
82.0
83.0
100.0
100.0
85.7
Patch
57.9
99.9
84.9
70.8
61.8
96.5
99.9
75.0
CIFAR-100
Blended
61.3
100.0
100.0
72.0
85.2
99.8
100.0
83.7
Patch
50.3
71.0
64.0
56.2
48.5
63.3
71.0
58.0
GTSRB
Blended
51,4
78.5
100.0
60.9
99.9
96.8
100.0
78.2
Average
55.8
89.3
88.6
70.8
70.5
89.6
93.9
75.0
Patch
55.6
54.3
66.5
51.1
70.6
53.0
70.6
58.5
MNIST
Evasive Trojans
Blended
60.2
67.8
99.2
54.9
84.8
67.2
99.2
72.4
Patch
61.3
67.4
58.1
60.0
52.2
62.3
67.4
60.2
CIFAR-10
Blended
53.5
67.2
100.0
84.0
79.7
93.3
100.0
79.6
Patch
54.9
50.4
61.1
50.7
61.6
55.0
61.6
55.6
CIFAR-100
Blended
54.4
65.1
100.0
64.6
91.7
76.1
100.0
75.3
Patch
50.8
73.7
56.6
54.8
77.1
48.7
77.1
60.3
GTSRB
Blended
55.0
72.3
100.0
81.3
85.5
62.0
100.0
76.0
Average
55.7
64.8
80.2
62.7
75.4
64.7
84.5
67.2
Detection results. Our evasive trojans are harder to detect across a wide range of detectors, datasets, and attack specifications. All values are percent AUROC, and lower is better for the at- tacker. For each detector, we bold the better value in the "Average" row.
Table 2:
Network Architectures and Hyperparameters. For CIFAR-I0 and CIFAR-100, we use 40-2 Wide ResNets (Zagoruyko& & Komodakis, 2016) with a dropout rate of 0.3 (Srivastava al., 2014). For GTSRB, we use the ViT Vision Transformer (Beyer et al., 2022) implemented by lucidrains. For MNIST, we use a simple 5-layer convnet with batch norm.
We train all CIFAR-10, CIFAR-100, and GTSRB networks for 50 epochs with batch size of 128. We train all MNIST networks for 10 epochs with a batch size 256 except for evasive trojans, which we found benefited from 20 epochs of training after initializing from clean networks.
We train all CIFAR-10 10 and CIFAR- networks using SGD with learning rate 0.1, weight decay of 5 x 10 and Nesterov momentum of 0.9. We train all MNIST and GTSRB networks using Adam with a weight decay of 10-5 and other hyperparameters at default settings. All training hyperparameters chosen early in preliminary experiments and received minimal tuning.
Detectors. To measure the evasiveness of different trojans, we use variety of detectors. The accuracy-basee detector (Acc) simply uses the clean accuracy of a network as a score for detection. The specificity-basee detector (Spec) computes the entropy of average posteriors on inputs with random triggers inserted. If trojan has low specificity, these random triggers will activate it and reveal presence. Neural Cleanse (NC) iterates over possible target labels for an attack and directly searches for candidate triggers using gradient-based optimization (Wang et al., 2019). ABS monitors the behavior of neurons while perturbing the input to identify candidate compromised neurons. These candidate neurons used to generate trojan trigger similarly to Neural Cleanse, and the maximum attack success rate across candidates used as a detection score (Liu et al., 2019). MNTD trains set of queries and shallow classifier on the training set of clean and trojaned models (Xu et al., 2021). The Param detector a logistic regressor with single linear layer that takes summary statistics of the network parameters its input. Finally, the Max and Avg baselines the maximum and average AUROC obtained by the six detectors on given set of trojaned networks.
5.1 DETECTION
To measure the effectiveness of detectors, of clean and trojaned networks. AUROC is a threshold -nnheenddddde metric that can be interpreted as the probability that a positive example has higher detection score than negative example (Fawcett, 2006), so 50% corresponds to random detection performance. For hand-crafted detectors that do not leverage the training set, the AUROC sometimes be below 50%. We find that this happens
area under the ROC curve (AUROC) test sets

---

Under review as a conference paper at ICLR 2023
NC
ABS
Param
MNTD
Max
Avg
Patch
60.8
16.8
8.0
40.0
60.8
31.4
Standard Trojans
MNIST
Blended
100.0
41.6
8.8
98.4
100.0
62.2
Patch
52.0
94.4
11.2
99.2
99.2
64.2
CIFAR-I0
Blended
98.4
84.8
11.2
100.0
100.0
73.6
Patch
38.4
70.4
0.0
28.8
70.4
34.4
CIFAR-100
Blended
100.0
48.0
0.0
14.4
100.0
40.6
Patch
35.2
19.2
3.2
9.6
35.2
16.8
GTSRB
Blended
100.0
32.0
3.2
46.4
100.0
45.4
Average
73.1
50.9
5.7
54.6
83.2
46.1
Patch
28.8
13.6
8.0
17.6
28.8
19.4
MNIST
Evasive Trojans
Blended
92.0
28.0
9.6
68.8
92.0
58.1
Patch
9.6
40.0
11.2
12.8
40.0
22.7
CIFAR-10
Blended
7.2
80.8
9.6
88.8
88.8
55.0
Patch
1.6
2.4
0.0
0.8
2.4
1.4
CIFAR-100
Blended
2.4
34.4
1.6
8.8
34.4
16.3
Patch
1.6
20.0
1.6
3.2
20.0
9.3
GTSRB
Blended
3.2
76.0
1.6
19.2
76.0
35.2
Average
18.2
36.9
5.6
27.3
47.8
27.2
Table 3: Target label prediction results. Although we do not specifically design evasive trojans to be hard to reverse-engineer, we find that predicting their target labels is much harder. All values are percent accuracy, and lower is better for the attacker. These unexpected and concerning results that highlight the need for more robust trojan detection and reverse-engineeriig methods.
small degree in some experiments. In these cases, we negate the detection score before computing AUROC on the test set.
Main Results. Detection results are in Table and sample ROC curves are in Figure 4. We train standard and evasive trojans in eight settings and evaluate them on six detectors. Average AUROC across all cight settings is lower for evasive trojans in five out of the six detectors. In some cases, evasiveness substantially improves. For example, average AUROC for the MNTD detector drops by 25%. When looking at the most effective detector in each setting, evasiveness also improves on average, with a 9.4 percent drop AUROC. This shows that our evasive trojans are harder detect not just for a specific detector, but for diverse range of detectors that use different mechanisms.
Surprisingly, the blended attacks are detected very easily by Neural Cleanse, and our evasion loss is unable to reduce the efficacy of Neural Cleanse in these settings. This is unexpected, because Neural Cleanse is designed specifically to detect patch attacks. However, our evasion loss does make blended attacks harder detect for other methods, including MNTD and in some settings ABS, As shown in Figure 2, although blended attacks tend to be easier to detect than patch attacks, evasive trojans reduce the efficacy of the average detector across all four datasets. Additional results and experiment details are in Appendix B.
5.2
Once a trojan has been detected, one might want to know what the intended functionality of the trojan is or what causes it to activate. Reverse-engineering trojans is a nascent field with relatively little prior work. However, since evasive trojans make detection more challenging, a natural ques- tion to ask is whether they also make reverse-engin engiinerinnn harder. We operationalize these reverse- engineering tasks predicting the target label of trojan attack and predicting the segmentation mask of patch attacks. Since recovering trigger patterns is nontrivial (Guo et al., 2019), we focus on reverse-engineering the trigger mask.
Target Label Prediction. We use accuracy metric for predicting target labels. Neural Cleanse and ABS predict target labels as part their detection pipeline, so no modification is needed. For the MNTD and Param detectors, multiclass cross-entropy. Results are in Table 3. Surprisingly, we find that evasive trojans are not only harder to detect, but they also make predicting the target label considerably harder. For each of
replace their output layer and train them classifiers with
8

---

Under review as a conference paper at ICLR 2023
NC
ABS
Param
MNTD
Max
Avg
Standard Trojans
MNIST
4.9
4.5
4.6
3.8
4.9
4.4
CIFAR-10
6.0
4.6
5.5
7.6
7.6
5.9
CIFAR-I00
6.4
5.0
7.6
7.1
7.6
6.5
GTSRB
5.5
6.5
7.2
5.6
7.2
6.2
Average
5.7
5.2
6.2
6.0
6.8
5.8
MNIST
5.7
5.3
5.9
5.2
5.9
5.5
Evasive Trojans
CIFAR-I0 10
5.7
4.3
4.1
4.8
5.7
4.7
CIFAR- 100
5.9
5.6
4.8
5.2
5.9
5.4
GTSRB
5.6
6.0
7.2
4.0
7.2
5.7
Average
5.7
5.3
5.5
4.8
6.2
5,3
Table 4: Trigger synthesis results. All values are percent IoU, and lower is better for the attacker. Although IoU is low across the board, evasive trojans further reduce IoU for the most effective methods. This demonstrates the need to develop stronger and more robust trigger synthesis methods.
the four classifiers, accuracy evasive trojans is lower. Notably, the average accuracies for Neural Cleanse, ABS, and MNTD drop by 54.9, and 27.3 percentage points, respectively. The accuracy of the best classifier in each setting drops by 35.4% on average.
Accuracy on evasive trojans drops to chance levels several settings. For example, on CIFAR-10 standard trojans, MNTD reaches 99. accuracy, but for evasive trojans it drops to 11.2% accuracy (random chance would be 10%). As with detection, the classifiers are more effective on blended at- tacks, but Neural Cleanse is also reduced from wear-perfect prediction to random chance blended attacks. Our evasion loss was only intended to make trojans harder to detect, and there is no a pri- ori reason for it to make target labels hard to predict. Consequently, this is very unexpected and concerning result for defense methods.
Trigger Synthesis. We mean intersection over union (IoU) across trojaned networks a met- ric for predicting trigger masks. Neural Cleanse and ABS generate candidate trigger masks as part of their detection pipeline, put layer with a 4-dimensional output that regresses to the top-left and bottom-right coordinates of trigger masks in the training set. If predicted bounding box is invalid, the predicted mask defaults to the entire image. In all trigger synthesis experiments, only patch attacks are used. The trigger masks have varying shapes and locations, but they are fixed upon sampling for given trojan. Thus, the task is well-defined and is a standard binary segmentation task.
modification is needed. For MNTD and Param, replace the out-
Results are in Table In general, performance is quite poor across the trigger synthesis methods, with IoU never reaching above 8%. Additionally, average IoU is very close for standard trojans and evasive trojans on Neural Cleanse and ABS. However, average IoU for Param and MNTD is decreased by evasive trojans. For MNTD, IoU drops from 6% to 4.8%, which is a 20% relative reduction. The loU of the most effective trigger synthesis method drops from 6.8% to 6.2% on average. These results indicate that trigger synthesis is somewhat more difficult on evasive trojans.
However, IoU values are close to the floor in all cases, which demonstrates a need for more research on this important aspect of reverse-engineering trojans.
6 CONCLUSION
We introduced method for inserting evasive trojans in deep neural networks. Unlike standard trojan attacks, our evasive trojans are specifically designed to be hard to detect. To evaluate our method, we trained standard and evasive trojans on a large scale, creating training and test sets containing over 6. 000 neural networks. These networks were used to train and evaluate wide variety of trojan detectors, including state-of-the-art detection algorithms and simple yet effective baselines. We found that our evasive trojans are much harder to detect across wide range of evaluation settings, in some cases reducing detection performance to chance levels. Surprisingly, found that evasive trojans also make reverse-engineering the target label and trigger of trojan attack substantially harder. We hope these results demonstrate the need for further research into robust mechanisms for monitoring and detecting hidden functionality in deep neural networks.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Eugene Bagdasaryan and Vitaly Shmatikov. Blind backdoors in deep learning models. In 30th USENIX Security Symposium (USENIX Security 2/), pp- 1505-1521, 2021.
Eugene Bagdasaryan, Andreas Veit, Yiqing Hua, Deborah Estrin, and Vitaly Shmatikov. How to backdoor federated learning. In International Conference on Artificial Intelligence and Statistics, pp. 1938-2948, PMLR, 2020.
Lucas Beyer, Xiaohua Zhai, and Alexander Kolesnikov. Better plain vit baselines for imagenet- 1k. arXiv preprint arXiv:2205.0/580 2022.
Nicholas Carlini and Andreas Terzis. Poisoning and backdooring contrastive learning. arXiv preprint arXiv:2106.0960 2021.
Bryant Chen, Wilka Carvalho, Nathalie Baracaldo. Heiko Ludwig, Benjamin Edwards, Taesung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. In SafeA/ AAAI, 2019a.
Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Deepinspect: A black-box trojan detection and mitigation framework for deep neural networks. In IJCAI, volume 2, pp. 2019b.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv: 2017.
Khoa Doan, Yingjie Lao, and Ping Li. Backdoor attack with imperceptible input and latent modifi- cation. Advances in Neural Information Processing Systems, 34, 2021a.
Khoa Doan, Yingjie Lao, Weijie Zhao, and Ping Li. Lira: Learnable, imperceptible and robust backdoor attacks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 119966-119766 2021b.
Tom Fawcett. An introduction roc analysis. Puttern recognition letters, 27(8):861-874, 2006.
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference, pp. 113-125, 2019.
Arturo Geigel. Neural network trojan. J. Comput, Secur, 21:1911::3:1 2013.
Shafi Goldwasser, Michael P Kim, Vinod Vaikuntanathan, and Or Zamir. Planting undetectable
backdoors in machine learning models. arXiv preprint arXiv:2204.06974 2022.
Tianyu Gu, Brendan Dolan Davitt,, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprini 17Xi::700.0663:: 2017.
Wenbo Guo, Lun Wang, Xinyu Xing, Min Du, and Dawn Xiaodong Song. Tabor: A highly accurate approach to inspecting and restoring trojan backdoors in systems. ArXiv, abs/1908.01766 2019.
Dan Hendryeks and Mantas Mazeika. arXiv:2206.00 05862, 2022.
X-risk analysis for ai research. arXiv preprint
Dan Hendrycks, Nicholas Carlini, John Schulman, and Jacob Steinhardt. Unsolved problems in ml safety. arXiv preprint MxXiv:210.13911: 2021.
Sanghyun Hong, Nicholas Carlini, and Alexey Kurakin. Handcrafted backdoors deep neural networks. arXiv preprint urXiv:2106.04660 2021.
Panagiota Kiourti, Wenchao Li, Anirban Roy, Karan Sikka, and Susmit Jha. Misa: Online defense of trojaned models using misattributions. In Annual Computer Security Applications Conference, pp. 570-585, 2021.
Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash, and Heiko Hoffmann. Universal litmus patterns: Revealing backdoor attacks in cnns. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 301-310, 2020.
10

---

Under review as a conference paper at ICLR 2023
Yiming Li, Yong Jiang, Zhifeng Li, and Shu-Tao Xia. Backdoor learning: A survey. IEEE Transac- tions on Neural Networks and Learning Systems, 2022.
Cong Liao, Haoti Zhong, Anna Cinzia Squicciarini, Sencun Zhu, and David J. Miller. Backdoor embedding in convolutional neural network models via invisible perturbation. Proceedings of the Tenth ACM Conference on Data and Application Security and Privacy, 2020.
Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 1265-1282, 2019.
Tuan Anh Nguyen and Anh Tuan Tran. Wanet imperceptible warping- warping-based backdoor attack. In International Conference on Learning Representations, 2021.
Ren Pang, Zheng Zhang, Xiangshan Gao, Zhaohan Xi, Shouling Ji, Peng Cheng. Xiapu Luo, and Ting Wang. Trojanzoo: Towards unified, holistic, and practical evaluation of neural backdoors. In 2022 IEEE 7th European Symposium on Security and Privacy (EuroS&P), pp. 684-702. IEEE, 2022.
Xinyang Zhang Shouling Yevgeniy Vorobeychik Xiapu Luo Alex Liu Ting Wang Ren Pang, Hua Shen. A tale of evil twins: Adversarial inputs versus poisoned models. arXiv preprint
arXiv:1911.01559, 2019.
Dinuka Sahabandu, Arezoo Rajabi, Luyao Niu, Bo Li, Bhaskar Ramasubramanian, and Radha Poovendran. Game of trojans: A submodular byzantine approach. arXiv preprint arXiv:2207.05937, 2022.
Guangyu Shen, Yingqi Liu, Guanhong Tao, Shengwei An, Qiuling Xu, Siyuan Cheng, Shiqing Ma, and Xiangyu Zhang. Backdoor scanning for deep neural networks through k-arm optimization. In International Conference on Machine Learning, pp. 9525-9536. PMLR, 2021.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way to prevent neural networks from overfitting. The journal of machine learning research, 15())11229-198, 2014.
Di Tang, XiaoFeng Wang, Haixu Tang, and Kehuan Zhang. Demon the variant: Statistical analysis of {DNNs} for robust backdoor contamination detection. In 30th USENIX Security Symposium (USENIX Security 21), pp. 1541-1558, 2021.
Guanhong Tao, Guangyu Shen, Yingqi Liu, shengwei An, Qiuling Xu, Shiqing Ma, Pan Li, and Xiangyu Zhang. Better trigger inversion optimization in backdoor scanning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2022.
Bolun Wang. Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium Security and Privacy (SP), pp. 707-723. IEEE, 2019.
Ren Wang, Gaoyuan Zhang. Sijia Liu, Pin- Yu Chen, Jinjun Xiong, and Meng Wang. Practical detection of trojan neural networks: Data-limited and data-free cases. In European Conference on Computer Vision, pp. 222-238. Springer, 2020.
Xiaojun Xu, Qi Wang, Huichen Li. Nikita Borisov, Carl A Gunter, and Bo Li. Detecting ai trojans using meta neural analysis. In 2021 IEEE Symposium Security and Privacy (SP), pp. 103-120. IEEF, 2021.
Yuanshun Yao, Huiying Li, Haitao Zheng, and Ben Y. Zhao. Regula sub-rosa: Latent backdoor attacks on deep neural networks. ArXiv, abs//990..10444 2019.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Edwin R. Hancock Richard
C. Wilson and William A. P. Smith (eds.), Proceedings the British Machine Vision Conference (BMVC), pp. 87.1-87.12 12. BMVA Press, September 2016. ISBN 1-901725-59-6 doi: 10.5244/C. 30.87.
11

---

Under review as a conference paper at ICLR 2023
Xinyang Zhang. Zheng Zhang, Shouling Ji, and Ting Wang. Trojaning language models for fun and profit. In 2021 IEEE European Symposium on Security and Privacy (EuroS&P), pp. 179-197. IEEE Computer Society, 2021.
Songzhu Zheng, Yikai Zhang, Hubert Wagner, Mayank Goswami, and Chao Chen. Topological detection of trojaned neural networks. Advances in Neural Information Processing Systems, 34, 2021.
A EVASIVE TROJANS DETAILS
Randomization Loss. The randomization loss minimizes the cosine distance between the network parameters and random direction. However, using the randomization loss in the raw parameter space is far too restrictive and effectively amounts to adding noise to the parameters. Thus, we use space of differentiable summary features of the parameters instead. We concatenate the mean and standard deviation of each parameter vector obtained via the PyTorrch parameters enumerator, which forms& single vector summarizing the network parameters.
For MNIST networks, we found that even this loss was too restrictive and hard to satisfy, perhaps due the smaller size of the networks. Thus, we use alternate randomization loss for MNIST networks. Let 6y be summary statistics of the trojaned model parameters, let 6y be summary statis- ties of the clean initialization, and let be a binary random vector. The alternate randomization loss is
Gran min(O, 0g 0) r|h + minn(' min (1 -))l
where the min function outputs the elementwise minimum. This loss allows each summary statistic in the trojaned network to either be greater smaller than the corresponding summary statistic in the clean initialization, where the direction of allowed change is selected by the binary random vector, This can be considered softer, axis-aligned version of the original randomization loss.
Hyperparameters. Our evasive trojans have several hyperparameters. The two components of the distribution matching loss and the randomization loss each receive a separate weight. Let these weights be denoted by '₁ A2, and As, respectively. We use different loss weights for different experiments, and the weights are coarsely tuned primarily to ensure that ASR is high enough. Em- pirically, we find that increasing the weights for these losses smoothly decreases ASR and eventually the task accuracy. This is because the evasion loss is in direct conflict with the trojan loss, so there is a balancing act between satisfying the evasion loss and maintaining high ASR. Concretely, we use the following settings of the loss weights:
For blended attacks on CIFAR- 10, A = 0.02, X2 = 0.02, A3 = 0.00005
For blended attacks CIFAR-100, X1 0.01, X2 0.02, A 0.0001
For blended attacks on GTSRB, À 0.005, A2 0.01, A3 0.00001
For patch attacks on GTSRB, A = 0.02, A2 = 0.02, A3 = 0.0001
For patch attacks on CIFAR-10, 10, A = 0.02, A2 = 0.02, A3 = 0.00005
For all other experimental settings, \1 0.1, A = 0.05, A3 0.0001
We tune these weights by training small numbers of individual networks and monitoring the evasion loss components and ASR. Importantly, we do not tune them specifically to obtain better results in the main experiments except in preliminary experiments meant to identify appropriate ranges for the losses, which we performed in a small number of settings. In a few experimental settings, we observed that there was a long tail of networks with low ASR. We hypothesized that this was due to the randomization loss picking a challenging direction. Thus, we retrained all networks below a cutoff ASR using new random directions, which solved the problem. In general, we find that our evasion loss is fairly robust to selections of loss weights and easy to use once the appropriate ranges for the weights are identified. The specificity loss is implemented by inserting incorrect triggers into 16 examples for blended attacks and 10 examples for patch attacks. These numbers were selected early during preliminary experiments.
12

---

Under review as a conference paper at ICLR 2023
ASR Accuracy
Clean Networks
MNIST
99.3
CIFAR-10
94.0
CIFAR-I00
74.6
GTSRB
84.7
Average
88.1
Patch
100.0
99.3
Standard Trojans
MNIST
Blended
100.0
99.3
Patch
100.0
93.9
CIFAR-10
Blended
99.5
93.9
Patch
99.8
74.5
CIFAR-100
Blended
97.5
74.5
Patch
99.8
85.5
GTSRB
Blended
94.6
83.5
Average
98.9
88.0
Patch
99.5
99.3
MNIST
Evasive Trojans
Blended
99.2
99.2
Patch
100.0
93.9
CIFAR-10
Blended
95.8
94.0
Patch
99.9
74.6
CIFAR-100
Blended
97.4
74.7
Patch
96.4
84.4
GTSRB
Blended
97.8
83.5
Average
98.3
87.9
Table 5: Attack success rate (ASR) and task accuracy in all experimental settings. Each value is averaged across 125 neural networks in the validation set for the indicated experimental setting. All values are percentages.
Other Details. In preliminary experiments, found that several implementation details were im- portant for increasing the evasiveness of our trojans. Namely, we train all evasive trojans without dropout. Clean initializations turn dropout off. This is because dropout introduces uncorrelated randomness in the activations of the trojaned network and its clean initialization, which makes satisfying the logit matching compo- nent of Cdist challenging. For similar reasons, we also switch batch norm layers in clean initialization networks eval mode throughout the second stage training evasive trojans.
trained with dropout, but during the second stage of training
To improve performance blended attacks, we found that it was important to process the inputs for the clean, trojan, and specificity losses together in a single forward pass. This because networks that use batch norm are able to "cheat" by aggregating information across the batch. Empirically, this issue arose most prominently with blended attacks. Concatenating the inputs together fixes the problem.
B ADDITIONAL RESULTS
Additional Details Detectors.
The accuracy-based detector (Acc) simply uses the clean accuracy a network as a score for detection. If a trojan insertion method consistently decreases clean accuracy, it can become trivial to detect, so this is an important baseline detector.
The specificity- based detector (Spec) assumes that the defender has access to a small set of triggers sampled from the same distribution of triggers that are used by the trojaned networks in question. This detector inserts each of the triggers into images from the validation set and com- putes the entropy of the average posterior. The entropy values then averaged, the negative of
13

---

Under review as a conference paper at ICLR 2023
Acc
Spec
NC
ABS
Param
MNTD
Max
Avg
Patch
56.5
53.4
63.1
53.6
67.7
60.9
67.7
59.2
MNIST
Blended
58.4
54.1
97.3
61.4
93.6
74.4
97.3
73.2
Without Land
Patch
72.8
71.1
54.7
61.3
85.7
88.6
88.6
72.4
CIFAR-10
Blended
57.4
66.7
100.0
90.8
100.0
91.3
100.0
84.4
Patch
74.1
98.8
55.7
54.1
100.0
74.9
100.0
76.3
CIFAR-100
Blended
50.0
72.2
100.0
74.1
100.0
94.5
100.0
81.8
Patch
51.4
62.6
54.5
53.0
78.2
49.5
78.2
58.2
GTSRB
Blended
52.2
55.4
100.0
84.5
93.5
74.8
100.0
76.7
Average
59.1
66.8
78.2
66.6
89.8
76.1
91.5
72.8
Patch
55.6
54.3
66.5
51.1
70.6
53.0
70.6
58.5
MNIST
Blended
60.2
67.8
99.2
54.9
84.8
67.2
99.2
72.4
With Lrand
Patch
61.3
67.4
58.1
60.0
52.2
62.3
67.4
60.2
CIFAR-10
Blended
53.5
67.2
100.0
84.0
79.7
93.3
100.0
79.6
Patch
54.9
50.4
61.1
50.7
61.6
55.0
61.6
55.6
CIFAR-100
Blended
54.4
65.1
100.0
64.6
91.7
76.1
100.0
75.3
Patch
50.8
73.7
56.6
54.8
77.1
48.7
77.1
60.3
GTSRB
Blended
55.0
72.3
100.0
81.3
85.5
62.0
100.0
76.0
Average
55.7
64.8
80.2
62.7
75.4
64.7
84.5
67.2
Table 6: Randomization loss ablation. Without the randomization loss, the Param detector is espe- cially strong, leading to a high maximum AUROC across all detectors. Adding the randomization loss greatly reduces AUROC for MNTD and Param detectors. For the other detectors, average AU- ROC remains nearly unchanged. All values are percent AUROC, and lower better for the attacker.
which is used as the detection score. For trojans with low specificity, the entropy of the average posterior for triggered inputs will be lower than for clean networks, which enables detection.
Neural Cleanse (NC) iterates over possible target labels for an attack and directly searches for candidate triggers using gradient- based optimization (Wang et al., 2019). We use simplified version of Neural Cleanse that we found obtains stronger detection performance. Namely, in preliminary experiments we found that early stopping did not improve results, so we optimize for fixed number of gradient steps. Additionally, the original Neural Cleanse method uses an anomaly index based off the & norms of the optimized trigger masks for detection, which enables selecting principled threshold. However, results in significantly better detection, so we switch to this simpler score for Neural Cleanse. This is enabled by our large-scale evaluations on datasets of clean and trojaned networks, which allows using threshold-independent metrics and any real-valued detection score.
find that simply using the raw l norms
ABS monitors the behavior of neurons while perturbing the input to identify candidate compro- mised neurons. These candidate neurons are used generate trojan trigger similarly Neural Cleanse, and the maximum attack success rate across candidates is used as a detection score (Liu al., 2019).
MNTD consists of set of query inputs, which are passed through the network in question. The outputs on these queries are then concatenated and passed to shallow classifier, which outputs detection score. The queries and shallow classifier are optimized on the training set of clean and trojaned networks (Xu et al., 2021). MNTD is an example of broad class of techniques called meta-networks: neural networks trained interpret or monitor other neural networks.
The Param detector is logistic regressor with single linear layer that takes summary statistics of the raw network parameters as its input. For summary statistics, we concatenate the min, max, mean, median, and standard deviation of each parameter vector into a single feature vector summarizing the raw parameters of the network.
B. ABLATIONS.
Our evasive trojan training procedure has several distinct components. Here, we examine what happens when certain components removed.
14

---

Under review as a conference paper at ICLR 2023
NC
Param
MNTD
Patch
58.8
100
60.5
With Spenoltimate
Blended
91.6
100
70.9
Patch
66.5
70.6
53.0
Without Epenultimate
Blended
99.2
84.8
67.2
Table 7: Evaluation of using an & distance on the penultimate features as an additional component of the distance metric. Compared to the original distance metric, this improves evasiveness against Neural Cleanse (lower AUROC) but reduces evasiveness against MNTD and Param (higher AU- ROC). All values are percent AUROC, and lower better for the attacker.
Randomization Loss. We include the randomization loss to mitigate emergent coordination across independently trained evasive trojans. This coordination occurs when only using the coerbutiio-matching and specificity losses, and it enables strong detection performance with a sim- ple detector that performs logistic regression summary statistics of the parameters (Param).
In Table 6, we compare evasive trojans with and without the randomization loss. When the ran- domization loss is removed, the Param and MNTD detectors become much stronger, while average AUROC for the other detectors remains relatively unchanged. In several cases for trojans without the randomization loss, the Param detector obtains 100% AUROC. Consequently, including the domization loss substantially reduces the AUROC of the best detector from an average of 91.5% 84.5%. These results demonstrate that the randomization loss is an important component of our method for training evasive trojans.
Specificity Loss. We include the specificity loss to prevent the issue of low specificity, where un- intended triggers can activate the trojan. If a tro- jan has low specificity, then a defender with knowl- edge of the distribution of triggers the trojan by checking whether the known triggers cause unusual behavior. Our specificity-based de-
Distance Correlates With Detectability
100
easily detect
MNTD AUROC (%)
tector (Spec) is based on this intuition. To validate the importance of the specificity loss, retrain the
CIFAR- 10 blended evasive trojans without the speci- ficity loss. The specificity detector obtains 100% AUROC on these networks compared to 67. 67.2% AU- ROC when the specificity loss used. This indicates that the specificity loss has the desired effect and is an important component of our method for training evasive trojans.
Parameter-Spate Distance
Logit Matching Loss. The logit matching loss one the two components of our distribution matching loss. To isolate the impact of this loss, we train retrain the CIFAR-10 patch evasive trojans without the logit matching loss. The MNTD detector obtains 70.8% AUROC on these networks compared 62.3% with the logit matching loss and 99.4% for standard trojans. This shows that the logit matching loss an important component of our evasive tro- jans, but it only accounts for part of the increased evasiveness.
Figure 5: When training evasive trojans with different weights on the evasion loss, one can control the distance from the paired clean network logit-space and parameter-space. These distances correlate with the detection performance of MNTD. This suggests that further reducing our current distance metric could lead to even greater evasiveness.
Different Distance Metrics. Since the distance metric is an important component of our distribution-matchinn loss, interesting question is what happens when the metric is changed. Here, we explore adding an & distance on the penultimate features to the distance metric. Con- cretely, add Epenultimate xx [I]sx(X) 9p (X)]k); where 9p and fp are functions that output
15

---

Under review as a conference paper at ICLR 2023
the penultimate features of the respective networks and Ap is a scalar loss weight. We set Ap equal 0.1 and retrain the MNIST evasive trojans using the modified distance metric. As before, we train 500 models. split evenly into patch and blended triggers and divided into training and test sets. We evaluate these trojaned models against baseline detectors and show the results in Table 7. We find that that evasiveness against Neural Cleanse increases, but evasiveness against MNTD and Param decreases. This demonstrates that the distance metric has a large effect on evasiveness, and designing good distance metries that improve evasiveness across many diverse detectors is nontrivial.
Impact of Evasion Loss on Detector Performance. Two natural questions are (1) whether our evasion loss actually reduces the distance in parameter and logit space intended and (2) whether this correlates with improved evasiveness. To more precisely evaluate the impact of our evasion loss, retrain our evasive trojans with patch triggers on MNIST using different weights evasion loss. For each training run, we multiply all components of the evasion loss by a fixed scalar ranging from (original evasion loss) to 0 (no evasion loss, but still initializing from a clean network). In particular, the loss weights are 1, 0.01, 0.001, and 0. The corresponding distance values in parameter space are 0.7, 2.0, 6.5, and 8.8. In logit space. the distance values are 2.2, 2.5, 5.9, and 33.9, respectively. This shows that our evasion loss is optimized successfully. To see whether this translates into changes detectability, we compute the percent AUROC for MNTD at each of these loss weights. In Figure 5, we show the results of this experiment by plotting distance in parameter-space on the x-axis and MNTD AUROC on the y-axis. There is a clear correlation: larger parameter distances result in higher detection performance. This suggests that evasiveness could be further improved by developing approaches that allow one to reduce our current distance metric even further.
the
B.2 ADDITIONAL ATTACK COMPARISONS
Here, we compare other trojan attacks that are designed with evasiveness in mind. In each section, we clarify how these prior attacks differ from our own.
Blind Backdoors Neural Cleanse Evasion Method. Bagdasaryan & Shmatikov (2021) train tro- jans specifically to evade Neural Cleanse. Namely, a Neural Cleanse search process is carried out simultaneously with model training in two-phase update approach. We implement this method and train a dataset of 500 MNIST models to evaluate its evasiveness. As with the standard and evasive trojans, we evenly split these models into patch and blended triggers and divide them into training and test sets.
With MNTD, the AUROC for patch and blended trojans is 72.8% and 98.7%, respectively. With Neural Cleanse, the AUROC for patch and blended trojans is 77.3% and 98.9%, respectively. With the Param detector, the AUROC for patch and blended trojans is 100.0% in both cases. Compared to the performance of Neural Cleanse on standard trojans, this is slightly better in both cases. This shows that their evasion method does work. However, MNTD and Param still have high perfor- mances on their trojans (in the case of Param, this reaches perfect detection performance). This shows that training trojans to be evasive for a specific detector may not generalize to all detectors. By contrast, our evasive trojans do generalize to reducing the detection performance of a broad range of detectors without specifically training against them.
WaNet Warping Attack. As we note in the related work, there have been numerous prior works exploring how to make trojan triggers more stealthy, which we distinguish from making trojans themselves more evasive. These methods are specifically designed to evade dataset-level and input- level detectors like Activation Clustering, Spectral Signatures, STRIP, and SentiNet. They do so by making inputs with triggers appear more similar to inputs without triggers (either in the input-space or intermediate features). However, these methods are not designed to evade model-level detectors these detectors. An interesting question
like MNTD ABS and are almost never evaluated is whether the strong evasiveness of this class of trojans on dataset-level and input-level detectors transfers to evasiveness
model-level detectors. To investigate this, train 500 trojaned CIFAR- 10 models using the WaNet attack (Nguyen & Tran, 2021). This attack uses subtle spatial warping of the input trigger, which improves evasiveness against input-level detectors like STRIP.
16

---

Under review as a conference paper at ICLR 2023
We evaluate our baseline detectors against the trained WaNet models. The Neural Cleanse, MNTD, and Param detectors obtain AUROC scores of 99.5%, 100.0%, and 99.98%, respectively. Thus, they very easy to detect. The result on Neural Cleanse runs counter Neural Cleanse experiments in the WaNet paper. We are not certain what the cause for this discrepancy is, although there are several possible sources: (1) We use custom PyTorch implementation of Neural Cleanse that
a different detection score due to our evaluations being threshold-indeeeedder (2) Many papers that propose attacks with whole-image triggers observe low detection performance with Neural Cleanse [cite, cite, cite]. However, our implementation of Neural Cleanse obtains very high AUROC on blended triggers. This is itself unexpected, but it could explain why our Neural Cleanse implemen- tation also works for whole-image warping triggers. We tried out different hyperarrameters for the warping field to see this would affect evasiveness, but this did not help. These results indicate that methods designed for evasiveness against input-level detectors do not generalize to being evasive for model-level detectors. Thus, separate approaches are needed for evasiveness against model-level detectors.
Acc
Spec
NC
ABS
Param
MNTD
Patch
53.6
63.1
65.5
52.3
46.3
59.2
Standard
Blended
54.5
99.8
90.3
69.8
66.3
82.3
Patch
50.8
58.3
50.9
51.6
52.7
54.4
TaCT
Blended
50.6
78.8
68.4
61.7
64.6
94.5
Patch
52.8
55.4
57.2
51.7
58.2
50.9
Evasive
Blended
55.6
71.2
72.8
53.8
65.3
74.4
Patch 51.7
51.9
50.1
51.5
57.7
47.1
Tvasive+TaCT
Blended 55.7
69.3
66.0
51.0
64.5
69.6
Table 8: Results source-specific trojans. TaCT obtains highly general evasion, although our eva- sive trojans are slightly better on average. Combining the two methods yields even greater evasion, demonstrating that TaCT is complimentary with approach. All values are percent AUROC, and lower better for the attacker.
Targeted Contamination Attack (TaCT). In our main experiments, we focus on one-to-all at- tacks. However, one-to-one attacks, also known source-specific attacks, are an important setting as well. In these attacks, the hidden behavior is only trained to activate on one specific source class. The target class is selected from among the other classes. Tang et al. (2021) find that in this source- specific setting, one can greatly improve evasiveness against Neural Cleanse and MNTD detectors with simple modification to the standard dati-nois attack. Instead of just inserting poisoned examples in the source class, they also insert "cover examples", which contain the trigger but are labeled with their original clean label. These cover examples are inserted for all classes besides the source class, which can be considered form of specificity loss for the source-specific setting. They name this method the Targeted Contamination Attack (TaCT).
TaCT is method for training evasive trojans in the source-specific setting, and there is some evi- dence in the original paper that generalizes across various model-level detectors. To compare our evasive trojans to TaCT, we adapt our standard and evasive trojans for the source-specific setting. This involves only inserting triggers for examples from the source class. For TaCT, we insert cover examples as well. Since TaCT can be combined with our evasive trojans, we include an experiment for this as well. We train 500 trojaned MNIST models for each setting and show results in Table 8.
Interestingly, we find that standard trojans are far harder to detect in the source-specific setting than in the all-to-one setting. Additionally, TaCT greatly improves evasiveness compared to the standard trojans. In fact, it is comparable to our evasive trojans. However, when we combine TaCT with our evasion loss, we obtain the best results. Averaging across all settings, the percent AUROC values for standard trojans, TaCT, evasive trojans, and evasive trojans with TaCT are 66.9, 61.4, 59.9, and 57.2. This shows that TaCT and our evasion loss are complimentary, and in settings where TaCT is applicable strongly recommend evaluating detectors against it.
17

---

Under review as a conference paper at ICLR 2023
Standard
Evasive
Trojans
Trojans
Patch 94.6
80.0
MNIST
Blended
53.9
62.6
Patch 98.8
76.9
CIFAR-10
Blended
99.9
100.0
Patch
57.3
52.5
GTSRB
Blended 71.3
75.4
Average
79.3
74.6
Table 9: The state-of-the-art PixelBackdoor trojan detector evaluated against standard trojans and our proposed evasive trojans. All values are percent AUROC, and lower is better for the attacker.
B.3 ADDITIONAL DETECTOR EVALUATIONS
Our six baseline detectors are selected to be representative of various different approaches to model- level detection. Here, we evaluate several additional strong detectors from recent years.
PixelBackdoor Detector. Tao et al. (2022) propose improved optimization process for trigger inversion. Their method builds on the trigger synthesis paradigm introduced by Neural Cleanse, and they obtain stronger detection results when compared to Neural Cleanse. We evaluate their detector on MNIST, CIFAR-10, and GTSRB (due to the long runtime, we did not evaluate on CIFAR-100 for the rebuttal deadline). Results shown in Table 9, For all the patch attack settings, our evasive trojans are considerably harder to detect. In the blended attack settings our evasive trojans are easier detect. However, the average AUROC is lower for our evasive trojans. Moreover, when consider the other detectors in Table 2. the maximum AUROC on blended attacks is still obtained by Neural Cleanse, so the blended results here would not impact the central metric of Max AUROC.
K-Arm Detector. Shen et al. (2021) propose the K-Arm detector. This detector iteratively selects most promising labels for trigger optimizer and can thought of as an improved version of Neural Cleanse. On CIFAR-10, standard trojans with patch and blended triggers obtain 52.9% and 89.0% AUROC, respectively. Our evasive trojans obtain 51.1% and 69.5% AUROC, respectively.
18


Reviews:
[{"review_id": "kocjVEyHHUv", "reviewer": "Reviewer_BFyZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "A. Paper summary\n\n- This paper proposes a new trojan method by designing evasive trojans. The key idea is to fine-tune a clean model by leveraging Wasserstein distance to minimize the output logits (between clean output / poisoned outputs) and using l2 distance to regulate the parameter distance (between the clean model and trojan model). Other tricks are provided (e.g., adding random loss) to further reduce the detection rate.\n\n- The result shows it can reduce the detection performance to near-chance levels.\n", "summary_of_the_review": "C. Questions:\n- In your evaluation, you only show the final detection success rate from different detectors. However, your key motivation is to minimize the output logits/weight difference. Can you show the result that your method is effective in reducing those distances? It would be more interesting if you can show a plot between those distances v.s detection accuracy.\n\n- Following the above questions, why are you specifically choosing the output logits? What if you added the penultimate layer's output?\n\n- In the \"randomization\" section, you mentioned some empirical results that are not provided in the evaluation. I believe Figure 3 is your motivation; so how about its (i.e., the randomization loss) effectiveness in reducing the detection rate?\n\nI have read many detector papers that are using the distribution of the final layer to detect trojan models. This paper uses Wasserstein distance to minimize this distribution gap between poisoned output logits and clean output logits which is a new challenge to the existing detection methods.\n\nOverall, I think the paper is very clear and highly motivated. However, some ablation studies are missing and I don't think the analysis is exciting. There are a lot of ways to dive deep into the problem and to provide more insights. \n\nMinor: Figure 2 is redundant with Table 2. Your figures and tables are too big. \n", "strength_and_weaknesses": "\nB. Strength And Weaknesses\n\n* Strength: \n1. The paper is clear and well-written. \n2. The novelty is straightforward to understand.\n3. Motivation is strong.\n\n* Weaknesses:\n1. The analysis is not enough (and not exciting). Some ablation studies are not provided.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eCqqxc-c6Tj", "reviewer": "Reviewer_jfBX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a method to train backdoored models that are stealthier (i.e., they are close to the distribution of clean models). Standard backdoor detection algorithms have less success against these models and they are also more difficult to reverse engineer.", "summary_of_the_review": "My biggest concern about this paper is the evaluation. First of all, the proposed attack is a supply-chain attack where the attacker has full control over the resulting model. Evasive attacks within this threat model is extensively studied, some examples are [1,2,3]. None of these works are considered in evaluation as baselines. The baseline attacks (Patch - Blend) are vanilla attacks that are shown time and time again easy to detect. Moreover, these baseline attacks are poisoning attacks (not supply chain) which is a significantly more difficult attack to perform. All in all, it is not clear whether this attack brings anything new to the table and whether the claims are true. I would recommend the authors to spend more energy into surveying the SOTA in attacks and defenses, think more carefully about their threat model and perform a more comprehensive evaluation.\n\nSecond, there has been more advanced published backdoor defenses the paper has not evaluated. These defenses claim to perform better than old defenses such as NC or ABC. For example: [4,5]\n\nThat being said, I like the idea of specifically improving the specificity of the backdoor via loss function. I think this is also a good tool to understand the limits of specificity (e.g., how specific can we make our backdoor, considering the inevitable side effects [6]?)\n\n[1] https://arxiv.org/abs/2002.12200\n[2] https://www.usenix.org/conference/usenixsecurity21/presentation/tang-di\n[3] https://openaccess.thecvf.com/content/ICCV2021/papers/Li_Invisible_Backdoor_Attack_With_Sample-Specific_Triggers_ICCV_2021_paper.pdf\n[4] https://arxiv.org/abs/2102.05123\n[5] https://www.cs.purdue.edu/homes/taog/docs/CVPR22_Tao.pdf\n[6] https://arxiv.org/abs/2010.09080", "strength_and_weaknesses": "+ Specificity is understudied in backdoor attacks and the proposed attack considers to improve specificity.\n\n- Very problematic evaluation and baseline attacks,", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "C_7QKM6L04d", "reviewer": "Reviewer_zJUx", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper uses distribution matching and randomization to reduce Trojan specificity. The author proposes to use 1-Wasserstein distance for the design of the loss function. The authors conduct experiments on MNIST, CIFAR-10 and CIFAR-100, show that the method proposed by the authors can improve the difficulty of Trojans being detected by detectors, and can be applied to various types of Trojans. Experiments also show that the author's proposed method can make Trojans more difficult to reverse engineer.", "summary_of_the_review": "This work attempts to make Trojans in models harder to detect, which makes sense. I hope the author can elaborate on the design of the loss function in Section 4.1, the use of 1-Wasserstein distance. In addition, the authors need to make a more complete analysis of the experimental results.", "strength_and_weaknesses": "Strengths:\n1. The author takes the evasion of Trojans as one of the goals and designs a loss function with a clear structure.\n2. The author uses 1-Wasserstein distance as a metric, and experiments show that it has a good effect, effectively improving the difficulty of detecting Trojan attacks.\n3. The author designed a relatively sufficient experiment, including the performance of the evasion Trojan under different detectors. The results show that the evasion Trojan is harder to detect, its target label is harder to predict, and more difficult to reverse engineer.\n \nWeaknesses:\n1. As one of the innovations of this work, the use of 1-Wasserstein distance in the loss function is not elaborated. How the author approximates the infimum also has no further clarification.\n2. As the core of the paper, the three-stage loss function designed by the author does not give a theoretical analysis, and the heuristic design of the loss function makes the persuasiveness decrease.\n3. I didn't find any text mentioning Figure 1 and Table 2, besides, the experimental results in Table 2 show that the standard Trojan performs even better than the evasion Trojan under the Param detector, which the author does not explain. The figure layout of the entire paper needs to be further optimized.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "CFLay7r62tT", "reviewer": "Reviewer_Wgb5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes an evasive trojan (backdoor) attack on deep neural networks that makes existing defenses ineffective. The key idea of this attack is \"distribution matching.\" The paper first fine-tunes a network f to construct a backdoored model g that (1) behaves similarly on both clean samples and those with the trigger pattern in the logits and (2) has a minimal difference in their parameter distributions. To increase the specificity of this attack, they further include an objective that the logits of f and g are similar on the same samples with the trigger pattern. In evaluation, the paper demonstrates the attack's effectiveness with a high success rate and a low accuracy drop after backdooring. The attack also evades existing defenses, such as NC, ABS, and MNTD, and the paper shows that the trigger reconstruction was unsuccessful against their backdoored models.", "summary_of_the_review": "It is important to study the weaknesses of existing defenses from an offensive perspective. From this angle, the paper provides a new attack that the community can use. However, the fact that the prior work has explored similar ideas reduces the novelty of this attack. \n\nIn addition, the technical novelty this paper claims (formulating a new objective to produce such evasive trojans) is also studied by several existing works. Unfortunately, this paper misses those works, which gives me concern that the claims are not from a comprehensive review of backdoor attacks and defenses.\n\nMoreover, the idea of \"distribution matching\" only considers the distributional differences we observe in the output space (the logits), which also gives me concern that the detection can work in the feature space or the latent space. Or, if the attack matches all the distributions, the trojans can be easily removed by just fine-tuning a model.\n\nI like that the evaluation has been conducted with a large number of models (Kudos to the authors), but the evaluation misses several other backdoor defense mechanisms that we even don't need any detection.\n\nI further found that the trigger reconstruction mechanisms have not been evaluated with some advanced techniques or properly, which gives me concern that claiming that \"trojan detection is hard\" is a bit early.\n", "strength_and_weaknesses": "\nStrengths:\n\n1. The paper studies an evasive trojan attack that makes existing detection ineffective.\n2. The paper conducts experiments with a large number of neural networks.\n3. The paper is well-written and easy to read.\n\nWeaknesses:\n\n1. Several prior works explored this idea are missing.\n2. The \"distribution matching\" can introduce several trojan artifacts, leading to detection.\n3. The evaluation against backdoor removals is missing.\n4. The evaluation against reverse-engineering efforts seems incorrect.\n\n\nDetailed comments:\n\nThis paper studies an evasive trojan attack against existing detection mechanisms. I like the research problem this paper tackles, as most detection techniques proposed so far rely on a specific artifact of backdoors affected by methodologies that the attackers choose.\n\n[Prior work on evasive trojans]\n\nOpposite to the paper's claims (stated in the Introduction), there has also been a vast literature on testing the effectiveness of either detection mechanisms or backdoor removal. \n\nOne example is the work by Tan et al. [1]. This work evades existing detection by making the latent representations of clean samples and the samples with the trojan trigger(s) similar. A minor difference between the work by Tan et al. and this paper is whether the loss makes the \"latent representations\" or the \"logits\" similar. \n\nAnother example is the work by Bagdasaryan et al. [2]. This work formulated the backdoor injection (which is fine-tuning of a clean model, the same as this paper) as multi-task learning for conflicting objectives (which is the superset of the idea proposed by the paper). The work also showed that the attack could evade many existing defenses by varying the objective the attacker adds to the original training objective (L_{task}). Unfortunately, the paper only discusses this work as a proposal for another backdoor attack but misses 60% of the entire paper about evading defenses shown by the authors, which is more valuable.\n\n[1] Tan et al., Bypassing Backdoor Detection Algorithms in Deep Learning, Euro S&P 2020.\n[2] Bagdasaryan et al., Blind Backdoors in Deep Learning Models, USENIX Security 2021.\n\n\n[Concerns about distribution matching]\n\nDistribution matching this paper proposes defines the \"distribution\" at the input and output spaces of a trojan model. However, the \"distribution\" can be defined in the latent representation space or feature spaces a network produces while forwarding an input.\n\n(1) This implies that one can detect the trojaned models this attack produces by comparing the latent representations and features. I wonder whether a defender can detect the models in Table 2 with the latent representations. With a fixed set of input examples and several potential trigger patterns, the defender can collect the latent representations from those examples and their versions with the triggers. The defender can run clustering to separate trojaned networks and benign ones.\n\n(2) If evaluation (1) does not offer sufficient detection scores (low AUCs), then I think a defender can easily remove the backdoors from those networks by fine-tuning on a small subset of clean samples for a few epochs (with the same learning rate that we used for training the benign models).\n\n\n[Weak evaluation]\n\nA line of backdoor defenses that this paper does not consider is backdoor removal. Prior work proposed fine-tuning or fine-pruning as a defense mechanism. They don't need trigger reconstruction or detection; just training (or pruning) a model can reduce the attack success rate. \n\nMy second concern is that the evaluation against trigger synthesis or reconstruction is under-studied. NC and ABS are not primarily designed for reconstructing triggers (even if the reconstruction process is important). Some mechanisms specifically study the reconstruction [3]. Due to a large number of parameters, conventional reconstruction leads to potential triggers with no semantic meaning; thus, [3] refines the process to reconstruct more semantically meaningful triggers. Using that advanced reconstruction, the attacker could find the trigger used by an adversary.\n\n[3] Sun et al., Poisoned Classifiers Are Not Only Backdoored, They Are Fundamentally Broken, 2020.\n\nMoreover, the success cases of trigger synthesis are measured by the IoU metric. It's a bit unconvincing that reconstructing the \"exact\" trigger is necessary to identify backdoored models. One can just extract approximate versions of the original trigger and then exploit them to see whether the classifications are biased or not, which could be sufficient.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject