Trigger Hunting with a Topological Prior for Trojan Detection
Authors: Xiaoling Hu
Xiao Lin
Michael Cogswell
Yi Yao
Susmit Jha
Chao Chen
Venue: ICLR 2022 Poster (2022)
Keywords: Trojan detection
diversity loss
topological prior

Abstract: Despite their success and popularity, deep neural networks (DNNs) are vulnerable when facing backdoor attacks. This impedes their wider adoption, especially in mission critical applications. This paper tackles the problem of Trojan detection, namely, identifying Trojaned models – models trained with poisoned data. One popular approach is reverse engineering, i.e., recovering the triggers on a clean image by manipulating the model’s prediction. One major challenge of reverse engineering approach is the enormous search space of triggers. To this end, we propose innovative priors such as diversity and topological simplicity to not only increase the chances of finding the appropriate triggers but also improve the quality of the found triggers. Moreover, by encouraging a diverse set of trigger candidates, our method can perform effectively in cases with unknown target labels. We demonstrate that these priors can significantly improve the quality of the recovered triggers, resulting in substantially improved Trojan detection accuracy as validated on both synthetic and publicly available TrojAI benchmarks.

Published as a conference paper at ICLR 2022
TRIGGER HUNTING WITH A TOPOLOGICAL PRIOR FOR TROJAN DETECTION
Xiaoling Hu Stony Brook University
Xiao Lin, Michael Cogswell, Yi Yao & Susmit Jha
SRI International
Chao Chen Stony Brook University
ABSTRACT
Despite their success and popularity, deep neural networks (DNNs) are vulnerable when facing backdoor attacks. This impedes their wider adoption, especially in mission critical applications. This paper tackles the problem of Trojan detection, namely, identifying Trojaned models models trained with poisoned data. One popular approach is reverse engineering, i.e., recovering the triggers on a clean image by manipulating the model's prediction. One major challenge of reverse engineering approach is the enormous search space of triggers. To this end, we propose innovative priors such as diversity and topological simplicity to not only increase the chances of finding the appropriate triggers but also improve the quality of the found triggers. Moreover, by encouraging a diverse set of trigger candidates, our method can perform effectively in cases with unknown target labels. We demonstrate that these priors can significantly improve the quality of the recovered triggers, resulting substantially improved Trojan detection accuracy as validated on both synthetic and publicly available TrojAI benchmarks.
INTRODUCTION
Deep learning has achieved superior performance in various computer vision tasks, such as image classification KKrzhevsky et al., 2012). image segmentation (Long et al. 2015). object detection (Gir) shick et al.] (014)) etc. However, the vulnerability of DNNs against backdoor attacks raises serious concerns. In this paper, we address the problem Trujan attucks, where during training, an attacker injects polluted samples. While resembling normal samples, these polluted samples contain specific type of perturbation (called triggers). These polluted samples are assigned with target labels, which usually different from the expected class labels. Training with this polluted dataset results in Trojaned model. At the inference stage, a Trojaned model behaves normally given clean samples. But when the trigger is present, it makes unexpected, yet consistently incorrect predictions.
One major constraint for Trojan detection is the limited access to the polluted training data. In practice, the end-users, who need detect the Trojaned models, often only have access to the weights and architectures of the trained DNNs. State-of-the-art (SOTA) Trojan detection methods generally adopt reverse engineering approach (Guo al.] 2019 Wang et al.) 2019 Huster & Ekwedike 2021 Wang et al. (020bb Chen et al. 2019c Liu et al. (219). They start with a few clean samples, using either gradient descent or careful stimuli crafting, to find potential trigger that alters model prediction. Characteristics of the recovered triggers along with the associated network activations are used as features to determine whether model is Trojaned or not.
Trojan triggers can be of arbitrary patterns (e.g., shape, color, texture) at arbitrary locations of an input image (e.g., Fig. D. As a result, one major challenge of the reverse engineering- approach is the enormous search space for potential triggers. Meanwhile, just like the trigger is unknown, the target label (i.e.. the class label to which a triggered model predicts) is also unknown in practice. Gradient descent may flip model's prediction to the closest alternative label, which may not necessarily be the target label. This makes it even more challenging to recover the true trigger. Note that many existing methods (Guo et al. 2019 Wang et al. 2019 2020b) require target label. These methods achieve
"Email: Xiaoling Hu (xiaolhu @cs..toobbbooo.cccii

---

Published as a conference paper at ICLR 2022
(b)
(d)
(e)
Figure I: Illustration of recovered triggers: (a) clean image, (b) poisoned image, (c) image with a trigger recovered without topological prior, (d)-(f) images with candidate triggers recovered with the proposed method. Topological prior contributes to improved compactness. We run the trigger reconstruction for multiple rounds with diversity prior to ensure a diverse set of trigger candidates.
target label independence by enumerating through all possible labels, which can be computationally prohibitive especially when the label space is huge.
We propose a novel langosicabelaagnosic reverse engineering method. First, to improve the quality of the recovered triggers, we need a prior that can localize the triggers, but in a flexible manner. We, therefore, propose to enforce a topological prior to the optimization process of reverse engineering. i... the recovered trigger should have fewer connected components. This prior is implemented through a topological loss based on the theory of persistent homology (Edelsbrunner & Harer 2010). It allows the recovered trigger to have arbitrary shape and size. Meanwhile, it ensures the trigger is not scattered and is reasonably localized. See Fig. for an example - comparing (d)-(f) vs. (c).
As second contribution, we propose to reverse engineer multiple diverse trigger candidates. In- stead of running gradient decent once, we run it for multiple rounds, each time producing one trigger candidate, e.g., Fig. [ (d)- Further- more, we propose a frigger diversity loss to en- sure the trigger candidates to be sufficiently dif- ferent from each other (see Fig. 2) Generating multiple diverse trigger candidates can increase the chance of finding the true trigger. It also mitigates the risk of unknown target labels. In the example of Fig. [2] the first trigger candidate flips the model prediction to label different from the target, while only the third candidate hits the true target label.
Clean
R3
Decision Boundary
Target Class
Figure 2: Illustration of generating a diverse set of trigger candidates to increase the chance of find- ing the true trigger, especially for scenarios with unknown target labels.
Generating multiple trigger candidates, however, adds difficulties in filtering out already-subtle cues for Trojan detection. We also note reverse engineering approaches often suffer from false positive triggers such as adversarial perturbations or direct modification of the crucial objects of the image") In practice, we systematically extract rich set of features to describe the characteristics of the reconstructed trigger candidates based on geometry, color, and topology, well network activations. A Trojan-detection network is then trained to detect Trojaned models based on these features. Our main contributions are summarized as follows:
We propose topological prior to regularize the optimization process of reverse engineering. The prior ensures the locality of the recovered triggers, while being sufficiently flexible regarding the appearance. It significantly improves the quality of the reconstructed triggers.
We propose a diversity loss generate multiple diverse trigger candidates. This increases the chance of recovering the true trigger, especially for cases with unknown target labels.
Combining the topological prior and diversity loss, we propose a novel Trojan detection framework. On both synthetic and public TrojAl benchmarks, our method demonstrates substantial improvement in both trigger recovery and Trojan detection.
Modiiication of crucial objects is usually not valid trigger strategy; it is too obvious to end-users and is against the principle of adversaries.
2

---

Published as a conference paper at ICLR 2022
2 RELATED WORK
Trojan detection. Many Trojan detection methods have been proposed recently. Some focus on [
detecting poisoned inputs via anomaly detection (Chou et al., 2020; Gao et al. 2019 Liu et al. [017 Ma & Liu 2019). For example, SentiNet (Chou et al. 2020) tries to identify adversarial inputs, and
uses the behaviors of these adversarial inputs detect Trojaned models. Others focus analyzing the behaviors of the trained models (Chen et al., 2019a Guo et al. 2019 [hhen et al. 2021 Sun et al. 2020). Specifically, Chen et al. (2019a) propose the Activation Clustering (AC) methodology to analyze the activations of neural networks to determine if a model has been poisoned or not.
While early works require all training data detect Trojans (Chen et al. 2019a Gao al. 2019
Tran et al. 2018), recent approaches have been focusing on a more realistic setting when one has limited access to the training data. particular promising direction reverse engincering approaches. which recover Trojan triggers with only a few clean samples. Neural cleanse (NC) (Wang et al. 2019) develops a Trojan detection method by identifying if there trigger that would produce misclassified results when added to an input. However, as pointed out by (Guo et al. 2019). NC
becomes futile when triggers vary in terms of size, shape, and location.
Since NC, different approaches have been proposed, extending the reverse engineering idea. Using conditional generative model, DeepInspect (Chen et al. | 2209cch learns the probability distribution of potential triggers from a model of interest. Kolouri et al. (2020) propose to learn universal patterss that change predictions of the model (called Universal Litmus Patterns (ULPs)). The method is efficient as it only involves forward passes through a CNN and avoids backpropagation. ABS (Liu et al. 2019) analyzes inner neuron behaviors by measuring how extra stimulation can change the network's prediction. [Wang et al. (2020b) propose data-limited TrojanNet detector (TND) by comparing the impact of per-sample attack and universal attack. [uuo et al. (2019) cast Trojan detection as a non-convex optimization problem and it is solved through optimizing an objective function. Huster & Ekwedike (202I) solve the problem by observing that, compared with clean models, adversarial perturbations transfer from image image more readily in poisoned models. Zheng et al. (2021) inspect neural network structure using persistent homology and identify structural cues differentiating Trojaned and clean models.
Existing methods are generally demanding on training data access, neural network architectures, types of triggers, target class, etc. This limits their deployment to real-world applications. As for reverse engineering approaches, it remains challenging, if not entirely infeasible, to recover the true triggers. We propose novel reverse engineering approach that can recover the triggers with high quality using the novel diversity and topological prior. Our method shares the common benefit of reverse engineering methods; it only needs a few clean input images per model. Mcanwhile, our approach is agnostic of model architectures, trigger types, and target labels.
Topological data analysis and persistent homology. Topological data analysis (TDA) is a field in which one analyzes datasets using topological tools such as persistent homology (Edelsbrunner &
Harer 2010; Edelsbrunner et al. 2000). The theory has been applied to different applications (Wu
al.
2017
Kwitt et al. 2015 Wong al. [2016] Chazal al.,
2013 Ni et al. 2017 Adams et al.
2017:
Bubenik 2015 Varshney & Ramamurthy 2015 Hu et al.
2019
2021
Zhao et al. 2020 Yan
al.
2021 Wu et al. 2020).
As the advent of deep learning, some works have tried to incorporate the topological information into deep neural networks, and the differentiable property of persistent homology make it possible. The main idea is that the persistence diagram/barcodess can capture all the topological changes, and it is differentiable to the original data. Hu et al. (2019) first propose a topological loss learn to segment images with correct topology, by matching persistence diagrams in supervised manner. Similarly, Clough et al.] (2020) use the persistence barcodes to enforce given topological prior of the target object. These methods achieve better results especially in structural accuracy. Persistent- hommoetntt--ooloy--- based losses have been applied to other imaging (Abousamra et al. 2021 Wang et al., 2020a) and
learning problems (Hofer et al. 2019 2020
Carrière al. 2020 Chen et al. 2019b).
The aforementioned methods use topological priors in supervised learning tasks (namely, segmenta- tion). Instead, in this work, we propose to leverage the topological prior in an unsupervised setting; we use a topological prior for the reverse engineering pipeline to reduce the search space of triggers and enforce the recovered triggers have fewer connected components.
3

---

Published as a conference paper at ICLR 2022
3 METHOD
Our reverse engineering framework is illustrated Fig. 3 Given a trained DNN model, either clean or Trojaned, and a few clean images, use gradient descent to reconstruct triggers that can flip the model's prediction. To increase the quality of reconstructed triggers, we introduce novel diversity loss and topological prior. They help recover multiple diverse triggers of high quality.
The common hypothesis of reverse engineering approaches is that the reconstructed triggers will appear different for Trojaned and clean models. To fully exploit the discriminative power of the reconstructed triggers for Trojan detection, we extract features based on trigger characteristics and associated network activations. These features are used train classifier, called the Trojan-dctection network, to classify a given model as Trojaned or clean.
We the discriminative power of the extracted trigger features, and thus the Trojan-detection network, are highly dependent on the quality of the reconstructed triggers. Empirical results will show the proposed diversity loss and topological prior are crucial in reconstructing high quality triggers, and ensures high quality Trojan-detection network. We will show that our method can learn to detect Trojaned models even when trained with a small amount of labeled DNN models.
For the rest of this section, we mainly focus on the reverse engineering module. We also add details of the Trigger feature extraction Sec. 3.3 and details of the Trojan-detection network Sec. A.1
Engineering
Trojan-detection Network
Clean Image
Reversed Image
Features
min L (m, 8; x.x,f m.8
Binary dassification: Trojaned
Trained models
Diversity Topo. Prior
Figure 3: Our Trojan detection framework.
3.1 REVERSE ENGINEERING OF MULTIPLE DIVERSE TRIGGER CANDIDATES
Our method is based on the existing reverse engineering pipeline first proposed by Neural Cleanse (Wang et al.) 2019). Given trained DNN model, f(-) the map- ping from an input clean image x 3 x to the output y E K with K classes, where M and N denote the height and width of the image, respectively. Denote by f-(-) the k-th output of f. The predicted label c* is given by c* arg max; (x), 1 k I K. We introduce parameters and m to convert x into an altered sample
(1 m)x
Φ m, 0)
mθ
6(x, m. 0) = (1 m) O x m O
(1)
Figure 4: m and convert an input im- age x into an altered one Φ(x, m, 0). The @ omitted here for simplification.
where the binary mask m (0, MNNN and the pattern M determine the trigger. 1 denotes all-one
matrix. The symbol "Q" denotes Hadamard product. See Fig. 4 for an illustration. We intend to find a triggered image & o(x, m, θ) so that the model prediction = arg max, fx()) is different from the prediction the original image e².
We find the triggered image, x, by minimizing a loss over the space of m and 0:
L(m, 0; x,
xxJ,, f, = (-..p(-..) + A L.tv(-...) + d A2 + R(m),
(2)
where Litip= Lave and Liopo denote the aabel-flipping loss, diversity loss, and topological loss, respectively. We temporarily dropped their arguments for convenience. Å1, X2 are the weights to
4

---

Published as a conference paper at ICLR 2022
balance the loss terms. R(m) is regularization term penalizing the size and range of the mask (more details will be provided in the Sec. bec.A..fff of Appendix).
To facilitate optimization, we relax the constraint on the mask m and allow it to be continuous- valued function, ranging between O and I and defined over the image domain, m E [o, 1|MxN Next, introduce the three loss terms one-by-one.
Label-fipping loss L fipp: The label-flipping loss fisp penalizes the prediction of the model regard- ing the ground truth label, formally:
L f m, x: f.))) = fe- (x(x, m, 0))
(3)
Minimizing Likp means minimizing the probability that the altered image 6(x, m, 0) is predicted as c". other words, we are pushing the input image out of its initial decision region.
Note that we do not specify which label we would like to flip the prediction to. This makes the optimization easier. Existing approaches often run optimization to flip the label to a target label and enumerate through all possible target labels (Wang et al. 2019 2020b). This can be rather expensive in computation, especially with large label space.
The downside of not specifying a target label during optimization is we will potentially miss the correct target label, i.e., the label which the Trojaned model predicts on a triggered image. To this end, we propose to reconstruct multiple candidate triggers with diversity constraints. This will increase the chance of hitting the correct target label. See Fig. 2 for an illustration.
Diversity loss Lute: With the label-flipping loss yup we flip the label to a different one from the original clean label and recover the corresponding triggers. The new label, however, may not be the same as the true target label. Also considering the huge trigger search space, it is difficult to recover the triggers with only one attempt. Instead, we propose to search for multiple trigger candidates to increase the chance of capturing the true trigger.
We run our algorithm for N1 rounds, each time reconstructing a different trigger candidate. To avoid finding similar trigger candidates, we introduce the diversity loss Lai to encourage different trigger patterns and locations. Let mj and 0, denote the trigger mask and pattern found in the j-th round. At the i-th round, we compare the current candidates with triggers from all previous founds in terms of L2 norm. Formally:
Laio (m, 0) = -- E m c 0 - m, O 0,112.
Minimizing Laie ensures the eventual trigger m, C 0, to be different from triggers from previous rounds. Fig. [f) demonstrates the multiple candidates recovered with sufficient diversity.
3.2 TOPOLOGICAL PRIOR
Quality control the trigger reconstruction remains a major challenge in reverse engineering methods, due to the huge search space of triggers. Even with the regularizor R(m), the recovered triggers can still be scattered and unrealistic. See Fig. (e) for an illustration. We propose a topological prior to improve the locality of the reconstructed trigger. We introduce a topological loss enforcing that the recovered trigger mask m to have as few number of connected components as possible. The loss is based on the theory of persistent homology (Edelsbrunner et 2000: Edelsbrunner & Harer 2010). which models the topological structures of a continuous signal in robust manner.
Persistent homology. We introduce persistent homology in the context of 2D images. A more comprehensive treatment of the topic can be found in (Edelsbrunner & Harer 2010 Dey & Wang 2021). Recall we relaxed the mask function m to a aattiuous-vvluue function defined over the image domain (denoted by Q). Given any threshold a. we can threshold the image domain with regard to m and obtain the superlevel set, Q == {o E 1pm(p) 2 ).. A superlevel set can have different topological structures, e.g., connected components and holes. If we continuously decrease the value a, we have a continuously growing superlevel set Ω2x This sequence of superlevel set is called a filtration. The topology of S continuously changes through the filtration. New connected components are born and later die (are merged with others). New holes are born and later die (are sealed up). For each topological structure, the threshold at which it is born is called its birth time. The threshold at which it dies is called its death time. The difference between birth and death time is called the persistence of the topoloiiaa structure.
5

---

Published as a conference paper at ICLR 2022
We record the lifespan of all topological structures over the filtration and encode them via 2D point set called persistence diagram, denoted by Dgm(m). Each topological structure is represented by 2D point within the diagram, p E Dgm(m), called persistent dot. We use the birth and death times of the topological structure to define the coordinates of the corresponding persistent dot. For each dot p (g)(m)), we abuse the notation and call the birth/death time of its corresponding topological structure as birth(p) and death(p). Then we have for an example function m (viewed terrain function) and its corresponding diagram. There are five dots in the diagram, corresponding to five peaks in the landscape view.
(death(p), birth(p)). See Fig. 5
To compute persistence diagram. we use the classic algorithm (Edelsbrunner Harer 2010 Edels brunner et al.) (2000) with an efficient implementation (Chen & Kerber 2011 Wagner et al. 2012). The image first discretized into a cubical complex consisting of vertices (pixels), edges and squares, A boundary matrix is then created to encode the adjacency relationship between these elements. The algorithm essentially carries out a matrix reduction algorithm over the boundary matrix, and the reduced matrix reads out the persistence diagram.
S
(a)
(b)
(C)
(d)
Death
Figure 5: From the left to right: (a) sample landscape for continuous function. The values the peaks Cg < 01 < 02 < 0g < 04 < og-- As we decrease the threshold, the topological structures of the superlevel set change, (b) and (c) correspond to topological structures captured by different thresholds, (d) highlighted region in (a), (e) the changes are captured by the persistence diagram (right figure). We focus on the O-dimensional topological structures (connected components). Each persistent dot in the persistence diagram denotes a specific connected component. The topological loss is introduced to reduce the connected components, which means pushing most of the persistent dots to the diagonal (along the green lines).
Topological loss Lxopo: We formulate our topological loss based on persistent homology described above. Minimizing our loss reduces the number of connected components of triggers. We will focus on vero-dimensional topological structure, i.c., connected components. Intuitively speaking, each dot in the diagram corresponds to a connected component. The ones far away from the diagonal line are considered salient as its birth and death times are far apart. And the ones close to the diagonal line are considered trivial. In Fig. 5 there is one salient dot far away from the diagonal line. It corresponds to the highest peak. The other four dots are closer to the diagonal line and correspond to the smaller peaks. The topological loss will reduce the number of connected components by penalizing the distance of all dots from the diagonal line, except for the most salient one. Formally, the loss Leopo is defined as:
Liopo (m) Σ peDgm(m)))pp (p*)
[birth(p) death(p)]3,
where p" denotes the persistent dot that is farthest away from the diagonal (with the highest persis- tence). Minimizing this loss will keep p" intact, while pushing all other dots the diagonal line, thus making their corresponding components either disappear or merged with the main component.
Differentiability and the gradient: The loss function (Eq. 5 is differentiable almost everywhere in the space of functions. To see this, we revisit the filtration, i.e., the growing superlevel set as we continuously decrease the threshold a. The topological structures change at specific locations of the image domain. A component is born at the corresponding local maximum. It dies merging with another component at the saddle point between the two peaks. In fact, these locations correspond to critical points of the function. And the function values at these critical points correspond to the birth and death times of these topological structures. For a persistent dot, p. we call the critical point corresponding its birth, cp(p), and the critical point corresponding to its death, ca(p). Then have by)th(p) = m(cs (p)) and death(p) = m(ca(p)). The loss function (Eq. 5) can be rewritten as a polynomial function of the function m different critical points.
Ltopo (m) = =Σ pe Dgm
[m(c((()) (m(ca)))) m(ca(p)))²
(6)

---

Published as a conference paper at ICLR 2022
The gradient can be computed naturally. Liopo is piecewise differentiable loss function over the space of all possible functions m. In a gradient decent step, for all dots except for p", we push up the function at the death critical point ca(p) (the saddle), and push down the function value the birth critical point (p(p) (the local maximum). This is illustrated by the arrows in Fig. Middlle-Righh This will kill the non-salient components and push them towards the diagonal.
3.3 TRIGGER FEATURE EXTRACTION AND TROJAN DETECTION NETWORK
Next we summarize the features we extract from recovered triggers. The recovered Trojan triggers can be characterized via their capability flipping model predictions (i.e., the fubel-fiippig loss). Moreover, they are different from adversarial noise as they tend to be more regularly shaped and are also distinct from actual objects which can be recognized by a trained model. We introduce appearance- features to differentiate triggers from adversarial noise and actual objects.
Specifically, for label flipping capability, we directly use the label-flipping loss ftip and diversity loss Laie features. For appearance features, we use trigger size and topological statistics as their features: 1) The number of foreground pixels divided by total number of pixels in mask m: 2) To capture the size of the triggers in the horizontal and vertical directions, we fit Gaussian distribution to the mask m and record mean and std in both directions; 3) The trigger we find may have multiple connected components. The final formulated topological descriptor includes the topological loss Trpoo the number of connected components, mean and std in terms of the size of each component.
After the features are extracted, build neural network for Trojan detection, which takes the bag of features of the generated triggers as inputs, and outputs a scalar score of whether the model is Trojaned or not. More details are provided in Sec. A.I of Appendix.
EXPERIMENTS
We evaluate our method on both synthetic datasets and publicly available TrojAl I benchmarks. We provide quantitative and qualitative results, followed by ablation studies, to demonstrate the efficacy of the proposed method. All clean/Trojaned models are DNNs trained for image classification.
Synthetic datasets (Trojaned-MNNIST and Frojaned-CIFARE No generate 200 DNNs (50% of them are Trojaned) trained to classify MNIST and CIFARIO data, respectively. The Trojaned models are trained with images poisoned by square triggers. The poison rate is set as 0.2.
We adopt the codes provided by
TrojAI benchmarks (TrojAl-Roundl, Round2, Round3 and Round4): These datasets are pro- vided by US IARPANIST who recently organized Trojan AI competition. Polygon triggers generated randomly with variations shape, size, and color. Filter-based triggers are generated by randomly choosing from five distinct filters. Trojan detection is more challenging on these TrojAl datasets as compared to Triggered-MNIST due to the use of deeper DNNs and larger variations in appearances of foreground/backgroumn objects, trigger patterns etc. Round1, Round2, Round3 and Round4 have 1000, 1 1104, 1008 and 1008 models, respectively. Descriptions of the difference among these rounds are provided in Sec. A.2 of Appendix.
Baselines: We choose recently pub- lished methods including NC (Neural Cleanse) (Wang et al.) 2019). ABS (Liu al. 2019), TABOR (Guo 2019), UL.P (Kolouri et al. (020)), and DLTND (Wang et al. (020bb) as basclines.
Table 1: Comparison on Thojaned-MMNIST/IIFARRR Method Metric Tiiinnd-A
NC
AUC
0.57 + 0.07
0.75 0.07
al.
ABS
AUC
0.63 + 0.04
0.67 + 0.06
TABOR AUC
0.65 + 0.07
0.71 0.05
ULP
AUC
0.59 + 0.03
0.55 1 0.03
DLTND AUC
0.62 1 0.05
0.52 0.08
Implementation details: We set X 1, A 10 and Ny = for all our experi- ments (i.e., generate 3 trigger candidates for each input image and each model). The parameters of Trojan detection network are learned using set of clean and Trojaned models with ground truth labeling. We train
Ours AUC
0.88 1 0.04
0.91 1 0.05
NC
ACC
0.60
0.73
ABS
ACC
0.65 1 0.02
0.69 1 0.04
TABOR
ACC
0.62 0.04
1
ULP
ACC
0.57
0.59
DLTND
ACC
0.64 0.07
0.55 0.07
Ours
ACC
0.89 0.02
0.92 0.04
7

---

Published as a conference paper at ICLR 2022
10%
10%
(a)
(b)
(c)
(d)
(c)
(f)
Figure 6: Examples of recovered triggers overlaid clean images. From left right: (a) clean image, (b) triggers recovered by (Wang et al. 2019), (c) triggers recovered by (Liu et al. 22199) (d)
triggers recovered by (Guo al., 2019), (e) triggers recovered by our method without topological
prior, and (f) triggers recovered by our method with topological prior.
the detection network by optimizing cross entropy loss using the Adam optimizer (Kingma & Ba 2014). The hidden state size, number of layers of MLP.. MLP,, as well as optimizer learning rate, weight decay and number of epochs are optimized using Bayesian hyperparameter searet for 500 rounds on 8-fold cross-validation.
We follow the settings in (Sikka et al. 2020p. We report the mean and standard deviation of two metrics: area under the ROC curve (AUC) and accuracy (ACC). Specifically, we evaluate our approach on the whole set by doing an 8-fold cross validation. For each fold, we use 80% of the models for training, 10% for validation, and the rest 10% for testing.
Evaluation metrics:
Results: Tables [ show the quantitative results on the Trojaned-MNIST/CFFARR WWWISNIIIT//IRIIIIII and TrojAI datasets, respectively. The reported performances of baselines are reproduced using source codes provided by the authors or quoted from related papers. The best performing numbers are highlighted in bold. From Tab. [ and we observe that method performs substantially better than the baselines. It is also worth noting that, compared with these baselines, our proposed method extracts fix-sized features for each model, independent of the number of classes, architectures, trigger types, etc. By using the extracted features, we are able to train a separate Trojan detection network, which salable and model-agnostic.
Table 2: Performance comparison on the TrojAl dataset.
Method
Metric
TrojAI-Round
TrojAl-Rounde TroojAl-Rund2
TrojAI-Round4
NC
AUC
0.50 0.03
0.63 0.04
0.61 1 0.06
0.58 0.05
ABS
AUC
0.68 - 0.05
0.61 = 0.06
0,57 + 0.04
0.53 0.06
TABOR
AUC
0.04
0.66 + 0.07
0.50 0.07
0.52 0.52:t0 0.04
ULP
AUC
0.06
0.48 = 0.02
0.53
0.54
DLTND
AUC
0.61 161::00 0.07
0.58 0.04
0.62 0.07
0.56 0.05
Ours
AUC
0.02 0.90
0.87 0.05
0.89 0.04
0.92 0.06
NC
ACC
0.53
0.49 = 0.02
0.59
0.60
ABS
ACC
+
0.59 + 0.05
0.56 0.03
0.51 0.05
TABOR
ACC
0.03
0.68 0.08
0.51
0.55 0.06 0.06
ULP
ACC
0.58 +
0.51 0.03
0.56 4.600.00 0.04
0.57 4: 0.04
DLTND
ACC
0.59 0.04 0.04
0.61 = 0.05
0.65 0.04
0.59 1 0.06
Ours
ACC
0.91 + 0.03
0.89 + 0.04
0.90 + 0.03
0.91 + 0.04
Fig. 6 shows few examples of recovered triggers. We observe that, compared with the baselines,
the triggers found by our method are more compact and of better quality. This is mainly due the introduction of topological constraints. The improved quality of recovered triggers directly results in improved performance of Trojan detection.
8

---

Published as a conference paper at ICLR 2022
Ablation study of loss weights: For the loss weights A1 and X2, we empirically choose the weights which make reverse engineering converge the fastest. This is a reasonable choice as in practice, time is one major concern for reverse engineering pipelines.
Despite the seemingly ad hoc choice, we have observed that our performances are quite robust to all these loss weights. As topological loss is major contribution of this paper, conduct ablation study in terms of its weight (A2) on AlojAl-R dataset. The results are reported in Fig. Z We observe that the proposed method is quite robust to X2, and when À 10, it achieves slightly better performance (AUC: 0.92 + 0.06) than other choices.
AUC vs A2
Ablation study of number of training model samples:
The trigger features and Trojan detection network are important in achieving SOTA performance. To further demonstrate the efficacy of the proposed diversity and topological loss terms, we conduct another ablation study to investigate the case with less training model samples, and thus a weaker Trojan detection network.
2.00
The ablation study in terms of number of training samples on TrojAl-Round4 data is illustrated in Tab. We observe Figure 7: Ablation study results for A2- that the proposed topological loss and diversity loss will boost the performance with/without fully trained Trojan- Trojan-detection network. These two losses improve the quality of the recovered trigger, in spite of how the trigger information is used. Thus even with limited number of training samples (e.g., 25), the proposed method could still achieve significantly better performance than the baselines.
22
Ablation study for loss terms: We investi- gate the individual contribution of different loss terms used to search for the latent triggers. Tab. 4 lists the corresponding performance on the Tr- dataset. We observe a de- crease in AUC (from 0.92 to 0.89) if the topo- logical loss removed. This drop is expected as the topological loss helps to find more compact triggers. Also, the performance drops signifi- cantly (from 0.92 to 0.85 in AUC) if the diversity loss is removed. We also report the performance by setting N7 = 2; when Nr = 2, the performance increases from 0.85 to 0.89 in AUC. The reason is that with diversity loss, we are able to generate multiple diverse trigger candidates, which increases the probability of recovering the true trigger when the target class is unknown. Our ablation study justifies the use of both diversity and topological losses.
Table 3: Ablation study for # of training samples.
# of samples
Ours
w/o topo w/o diversity
25
0.77 1 0.04 0.73 0.03 0.68 1 0.04
50
0.81 1 0.03 0.76 1: 0.05 0.73 1 0.02
100
0.84 + 0.05 0.78 1: 0.06
0.76 +: 0.03
200
0.86 0.04 0.82 0.04
0.79 + 0.05
400
0.90 = 0.05 0.85 0.03
0.82 = 0.04
800
0.92 = 0.06 0.89 + 0.04 0.85 + 0.02
In practice, we found that topological loss can im- prove the convergence of trigger search. Without topological loss, it takes R:50 iterations to find a rea- sonable trigger (Fig. ee)) In contrast, with the topo- logical loss, it takes only = iterations to converge better recovered trigger (Fig. (()). The rationale
Table 4: Ablation results of loss terms. Method
TrojAl-Roungh
w/o topological loss
0.89 1 0.04
w/o diversity loss (Nr = 1)
0.85 0.02
Nr
0.89 1 0.05
with all loss terms (Nr = 3) 0.92 + 0.06
is that, as the topological loss imposes strong constraints on the number of connected components, it
largely reduces the search space of triggers, consequently, making the convergence of trigger search much faster. This is worth further investigation.
Unsupervised vs supervised: Our technical contributions are agnostic of whether the detection is supervised (i.e., using annotated models) or unsupervised. The proposed diversity and topological losses are used to improve the quality of reconstructed trigger, using only a single model and input data. More discussions and results in terms of (un)supervised settings are included in Sec. A.S
5 CONCLUSION
In this paper, we propose a diversity loss and a topological prior to improve the quality of the trigger reverse engineering for Trojan detection. These loss terms help finding high quality triggers efficiently. They also avoid the dependant of the method to the target label. On both synthetic datasets and publicly available TrojAl benchmarks, our approach recovers high quality triggers and achieves SOTA Trojan detection performance.

---

Published as a conference paper at ICLR 2022
Ethics Statement: As we have developed better Trojan detection algorithm and introduce the method in details, the attackers may inversely create Trojaned models that are more difficult to detect based on the limitations of current method. Attack and defense will always coexist, which pushes researchers to keep developing more efficient algorithms.
Reproducibility Statement: The implementation details mentioned in Sec. The details of the data are provided Sec. Sec. of Appendix. The details of Trojan detection classifier are described in Sec. A.1 of Appendix. The used computation resources are specified in Sec. A.7 of Appendix.
Acknowledgement: The authors thank anonymous reviewers for their constructive feedback. This effort was partially supported by the Intelligence Advanced Research Projects Agency (IARPA) under the contract W911N00000 The content of this paper does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.
REFERENCES
Shahira Abousamra, Minh Hoai, Dimitris Samaras, and Chao Chen. Localization in the crowd with topological constraints. In Proceedings of AAAI Conference on Artificial Intelligence, 2021.
Henry Adams, Tegan Emerson, Michael Kirby, Rachel Neville, Chris Peterson, Patrick Shipman, Sofya Chepushtanova, Eric Hanson, Francis Motta, and Lori Ziegelmeier. Persistence images: stable vector representation of persistent homology. The Journal of Machine Learning Research, 18(1):218-255)) 2017.
Peter Bubenik. Statistical topological data analysis using persistence landscapes. The Journal of Machine Learning Research, 16(1):77-00,, 2015.
Mathieu Carrèère, Frédéric Chazal, Yuichi Ike, Théo Lacombe, Martin Royer, and Yuhei Umeda.
Perslay: A neural network layer for persistence diagrams and new graph topological signatures. In International Conference Artificial Intelligence and Statisties, pp. 2786-2796. PMLR, 2020.
Frédéric Chazal, Leonidas J Guibas, Steve Oudot, and Primoz Skraba. Persistence-based clustering in riemannian manifolds. Journal of the ACM (JACM), 60(6):41, 2013.
Bryant Chen, Wilka Carvalho, Nathalic Baracaldo, Heiko Ludwig. Benjamin Edwards, Tacsung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. In SafeAI@ AAAI, 2019a.
Chao Chen and Michael Kerber. Persistent homology computation with a twist. In Proceedings 27th
European Workshop on Computational Geometry, volume 11, pp. 197-200, 2011.
Chao Chen, Xiuyan Ni, Qinxun Bai, and Yusu Wang. A topological regularizer for classifiers via persistent homology. In The 22nd International Conference Artificial Intelligence and Statistics, pp. 2572-2582. PMLR, 2019b.
Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. Deepinspect: A black-box trojan detection and mitigation framework for deep neural networks. In IJCAI, pp. 4658-4664, 2019c.
Edward Chou, Florian Tramèr, and Giancarlo Pellegrino. Sentinet: Detecting localized universal attacks against deep learning systems. In 2020 IEEE Security and Privacy Workshops (SPW), pp. 48-5 54. IEEE, 2020.
James Clough, Nicholas Byrne Ilkay Oksuz, Veronika A Zimmer, Julia A Schnabel, and Andrew King. topological loss function for deep-learning based image segmentation using persistent homology. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020.
Tamal K. Dey and Yusu Wang. Computational Topology for Data Analysis. Cambridge University Press. 2021.
Herbert Edelsbrunner and John Harer, Computational topology: an introduction. American Mathe- matical Soc., 2010.
10

---

Published as a conference paper at ICLR 2022
Herbert Edelsbrunner, David Letscher, and Afra Zomorodian. Topological persistence and simplifica- tion. In Proceedings s1 Anmual Symposium on Foundations of Computer Science, pp. 454 463. IEEE, 2000.
Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: A defence against trojan attacks on deep neural networks. In Proceedings of the 35th Annual Computer Security Applications Conference, pp. 1 113 2019.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 580-587, 2014.
Wenbo Guo, Lun Wang, Xinyu Xing, Min Du, and Dawn Song. Tabor: A highly accurate approach to inspecting and restoring trojan backdoors systems. 20th IEEE International Conference on Data Mining, 2019.
Christoph Hofer, Roland Kwitt, Mare Niethammer, and Mandar Dixit. Comnectivity representation learning via persistent homology. In International Conference on Machine Leurning, pp. 2751-2766 PMLR, 2019.
Christoph Hofer, Florian Graf, Bastian Rieck, Mare Niethammer, and Roland Kwitt. Graph filtration learning. International Conference on Machine Learning, pp. 4314 4323. PMLR, 2020.
Xiaoling Hu, Fuxin Li, Dimitris Samaras, and Chao Chen. tation. Advances in neural information processing systems, 32, 2019.
deep image segmen-
Xiaoling Hu, Yusu Wang, Li Fuxin, Dimiuris Samaras, and Chao Chen. Topology-aware segmentation using discrete morse theory. International Conference on Learning Representations, 2021.
Todd Huster and Emmanuel Ekwedike. Top: Backdoor detection in neural networks via transferability of perturbation. arXiv preprint arXiv:2103. 2021.
Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv:1412.69880 2014.
Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash, and Heiko Hoffmann. Universal litmus patterns: Revealing backdoor attacks in cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 301-310, 2020.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep con- volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Roland Kwitt, Stefan Huber, Marc Niethammer, Weili Lin, and Ulrich Bauer. Statistical topological data analysis-a kernel perspective. In Advances neural information processing systems, pp. 3070-3078, 2015.
Yinggi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang, Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference Computer and Communications Security, pp. 1265-1282, 2019.
Yuntao Liu, Yang Xie, and Ankur Srivastava. Neural trojans. In 2017 IEEE International Conference on Computer Design (ICCD), pp. 48. IEEE, 2017.
Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3431-3440, 2015.
Shiqing Ma and Yingqi Liu. Nic: Detecting adversarial samples with neural network invariant checking. In Proceedings of the 26th Network and Distributed System Security Symposium (NDSS 20/9), 2019.
Xiuyan Ni, Novi Quadrianto, Yusu Wang, and Chao Chen. Composing tree graphical models with persistent homology features for clustering mixed-type data. In Proceedings of the 34th International Conference Machine Learning- -ooninggVVolume 70, pp. 2622-2631. JMLR. org, 2017.
11

---

Published as a conference paper at ICLR 2022
Guangyu Shen, Yingqi Liu, Guanhong Tao, Shengwei An, Qiuling Xu, Siyuan Cheng, Shiqing Ma, and Xiangyu Zhang. Backdoor scanning for deep neural networks through k-arm optimization. arXiv preprint arXhv:2102.05123, 2021.
Karan Sikka, Indranil Sur, Susmit Jha, Anirban Roy, and Ajay Divakaran. Detecting trojaned dnns using counterfactual attributions. arXiv preprint arXiv:2012.02225, 2020.
Mingjie Sun, Siddhant Agarwal, and J Zico Kolter. Poisoned classifiers are not only backdoored, they are fundamentally broken. arXiv preprint arXiv:2010.0908/ 2020.
Brandon Tran, Jerry Li, and Aleksander Madry. Spectral signatures in backdoor attacks. Advances in neural information processing systems, 2018.
Kush R Varshney and Karthikeyan Natesan Ramamurthy. Persistent topology of decision boundaries. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 3931-3935. IEEE, 2015.
Hubert Wagner, Chao Chen, and Erald Vuçini. Efficient computation of persistent homology for cubical data. In Topological methods in data analysis and visualization II, pp. 91 Springer, 2012.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pp. 707-723. IEFF, 2019.
Fan Wang, Huidong Liu, Dimitris Samaras, and Chao Chen. Topogan: A topology-aware generative adversarial network. In European Conference on Computer Vision, pp. 118-136, 1-6.. Springer, 2020a.
Ren Wang, Gaoyuan Zhang, Sijia Liu, Pin-Yo Yu Chen, Jinjun Xiong, and Meng Wang. Practical detection of trojan neural networks: Data-limited and data-free cases. European Conference Computer Vision (ECCV), 2020b.
Eleanor Wong, Sourabh Palande, Bei Wang, Brandon Zielinski, Jeffrey Anderson, and P Thomas Fletcher. Kernel partial least squares regression for relating functional brain network topology to clinical measures of behavior. In 2016 IEEE 13th International Symposium on Biomedical Imaging (ISBI), pp. 1303-1306. IEEE, 2016.
Eric Wong, Leslie Rice, and Zico Kolter. Fast better than free: Revisiting adversarial training. In International Conference on Learning Representations, 2019,
Pengxiang Wu, Chao Chen, Yusu Wang, Shaoting Zhang, Changhe Yuan, Zhen Qian, Dimitris Metaxas, and Leon Axel. Optimal topological cycles and their application in cardiac trabeculac restoration. In International Conference Information Processing Medical Imaging, pp. 80-92. Springer, 2017.
Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris Metaxas, and Chao Chen. A topological filter for learning with label noise. Advances in neural information processing systems, 33: 11382-21393, 2020.
Zuoyu Yan, Tengfei Ma, Liangcai Gao, Zhi Tang, and Chao Chen. Link prediction with persistent homology: An interactive view. In International Conference on Machine Learning, pp. 11659- 11669. PMLR, 2021.
Qi Zhao, Ze Ye, Chao Chen, and Yusu Wang. Persistence enhanced graph neural network. In International Conference on Artificial Intelligence and Statistics, pp. 2896-2906. PMLR, 2020.
Songzhu Zheng, Yikai Zhang, Hubert Wagner, Mayank Goswami, and Chao Chen. Topological detection of trojaned neural networks. Advances in Neural Information Processing Systems, 34, 2021.
12

---

Published as a conference paper at ICLR 2022
A APPENDIX
A.I LEARNING-BASED TROIAN-DETECTION NETWORK
While bottom-up trigger generation uses appearance heuristics to search for possible triggers, we cannot guarantee the recovered triggers are true triggers, even for Trojaned models. Many other perturbations can create label flipping effects, such as adversarial samples and modifications specific on the most semantically critical region. See Figureßfoo for illustrations. These examples can easily become false positives for Trojan detector; they can be good trigger candidate generated by the reverse engineer pipeline.
To fully address these issues, we propose top-down Trojan detector learned using clean and Trojaned models. It helps separate true Trojan triggers from other false positives. To this end, we propose to extract features from the reverse engineered Trojan triggers and train a separate shallow neural network for Trojan detection.
For each model, generate diverse set of Nr possible triggers for each of its K output classes to sean for possible triggers. As described above, we extract one feature for each generated trigger. As result, for each model we have NT K sets of features.
The regularizer R(m) consists of a mass term and a size term. For mass, we use má as the average value of m. For size, we normalize the mask m into a distribution p(æ, y) over and y. To capture the spatial extent of mask m, we compute the standard deviation of X ~ p(x) as 6x and Y ~ p(y) as ōy. As a result, R(m) m + 8x dy is the regularizer.
Regularizer:
Classification network: After the features are extracted, we build neural network for Trojan detection, which takes as input for given image classifier, the bag of features of its generated triggers and outputs scalar score of whether model is Trojaned or not.
Bottom up Reverse Engineering with Topological Prior
Top-down Trojan ClassiBiation
Changes Prediction
Localized Connected
Preserves Content
Object modirication
x
Since different models may have different num- bers of output classes K, the number of features varies for each model. Therefore, sequence of modeling architectures - such as bag-of- bag-of-words, Recurrent Neural Networks and Transformers could be employed to aggregate the Nr x K features into a 0/1 Trojan classification output.
Adversarial noise
X
?
X
Trojan trigger
Figure 8: Our Trojan detection method combines bottom-up Trigger reverse engineering under topo- logical constraints, with top-down classification. Such combination allows us to accurately isolate Trojan triggers from non-Trojan patterns such as adversarial noise and object modifications.
Given a set of annotated models, clean or in- fected, supervised learning can be applied to train the classification network. Empirically, found that simple bag-of-words technique achieved the best Trojan detection performance while also being fast to run and data efficient.
Specifically, let the bag of features be (u;}, i 1.... 1....,NyKK NyK. The features are first transformed individually using an MIP, followed by average pooling across the features and another MIP to output the Trojan classification:
h = MLP.(E)) h= NTK The =MLP((h), ..K.
A.2 DETAILS ABOUT THE TROIAI DATASETS
TrojAI-Round1, Round2, Round3, Round4 datasets: These datasets provided by US IARPA/NIST| and contain trained models for traffic sign classification (for each round, 50% of total models are Trojaned). All the models are trained synthetically ereated image data of non- real traffic signs superimposed on road background scenes. Trojan detection is more difficult on Round2/Round3/Roun444 compared Roundl due to following reasons:
13

---

Published as a conference paper at ICLR 2022
Round2/Round3 have number of classes: Round1 has 5 classes while Round2/Round3 have 5-25 classes.
Round2/Round3 have more trigger types: Round only has polygon triggers, while Round2/Round3 have both polygon and Instagram filter based triggers.
The number of source classes are different: all classes are poisoned in Roundl, while 1, 2, all classes are poisoned in Round2/Round3.
Round2/Round3 have more type of model architectures: Round1 has 3 architectures, while Round2/Round3 have 23 architectures.
Round3 experimental design identical Round2 with the addition of Adversarial Training. Two different Adversarial Training approaches: Projected Gradient Descent (PGD), Fast is Better than Free (FBF) (Wong et al. 2019) are used.
Unlike the previous rounds, Round4 can have multiple concurrent triggers. Additionally, triggers can have conditions attached to their firing. The differences are listed as follows:
All triggers in Round4 are one to one mappings, which means trigger flips a single source class to single target class.
Three possible conditionals, spatial, spectral and class are attached to triggers within this dataset.
Round4 has remove the very large model architectures to reduce the training time.
Round1, Round2, Round3 and Round4 have 1000, 1104, 1008 and 1008 models respectively.
A.3 SUPPORTING ADDITIONAL TRIGGER CLASSES
Different classes of Trojan triggers are being actively explored in Trojan detection benchmarks. For image classification, the TrojAI datasets include localized triggers which can be directly applied to objects along with global filter-based Trojans, where the idea is that color filter could be attached to the lens of camera and results in global image transformation.
Our top-down bottom-up Trojan detection framework is designed to support multiple classes of Trojan triggers, where each trigger class gets its dedicated reverse engineering approach and pathway the Trojan detection network. Adding support to a new class triggers, e.g. color filters, amounts adding a reverse engincering approach for color filters and adding appearance feature descriptor for Trojan classification.
Reverse engineering color filter triggers. The pipeline of reverse engineering color filter triggers is illustrated in Fig. Comparing to reverse engineering local triggers in Fig.4, the filter editor and the loss functions are adjusted to find color filter triggers.
We model a color filter trigger using per-pixel position-dependent color transformation. Let (rijs Diy, biy] be the color of a pixel input image x at location (i.j), and we model a color filter trigger using an MIP (1; gflter) with parameters gilter which performs position-dependent color mapping:
[rig Sig, big) E(rajs Mij, bij, i, j, Oflter.
Here [rly, big] is the color pixel (i.j) of the triggered image x. For TrojAI datasets we use 2-layer 16 hidden neurons MLP to model the color filters, it learns sufficiently complex color transforms while being fast to run:
L Rilter
Aller L flip
+ X filter y lilter + > pllleer (gilter)
(9)
The label flipping loss 7 fiter remains identical:
flip
I Glier
Úc-
(10)
14

---

Published as a conference paper at ICLR 2022
Diversity
Filter Editor
Trained models
min Thpp
fitter
+ füter + Rf
(x)
fx)
Figure 9: Reverse engineering of global color filter triggers.
The diversity loss Liller is designed to induce diverse color transforms. We use how a color filter transforms 16 random (r, 9, b, i,j) tuples to characterize a color filter E:: (fileer) We record the 16 output (f: g. b) tuples as a 48-dim descriptor of the color filter, denoted as Ugmar. The diversity loss is the L2 norm of the current candidate to previously found triggers:
Nr
dio
Lighter
Uggithr 1|2.
(11)
j=
The regularizer term Rfilter (ffite( is simply an 1.2 regularizer on the MLP parameters to reduce the complexity on the color transforms:
Rftltor (ffittr)
||0fther
(12)
reverse engineering to generate K classes by Nfftter diverse color filter triggers. For each color filter trigger, we combine an appearance descriptor with label flipping loss I, fiter and diversity loss L. ddi its combined feature flip
Feature extractor for color filter triggers. For each model
descriptor. For the appearance descriptor, we use the same descriptor discussed in diversity: how color filter E-; (fileer) transforms 16 random (r,g, b, i, tuples. We record the 16 output (f,9, tuples as a 48-dim appearance descriptor.
As result for each model we have filter K features for color filter triggers.
Trojan classifier with color filter triggers. A bag-of-words model is used to aggregate those features. The aggregated features across multiple trigger classes, e.g. color filters and local triggers, are concatenated and fed through an MLP for final Trojan classification as below:
R filter MLPfitter (ef (Tfilter),
Rfttter
Σ filter i= == N.....N.A.. filter K
(13)
N-fittrr K
Nloeat
M LL (Ffooll))
(Focal),
Mlocat
Σ
Thooat i= J1...Nyh
(14)
Nr
= M MLPP Rlocal)).
(15)
For the TrojAl datasets, color filter reverse engineering is conducted using Adam optimizer with learning rate x 10-2 for 10 iterations. Hyperparameters are set to Xftee 0.05 and Xfiler 10-4 We also set Ny 2 and N filter
8.
A.4 COMPARED WITH ADDITIONAL SOTA METHOO
We also compare proposed method with another parallel work (Shen et al. 2021 Here directly quote the numbers (Accuracy) from the paper for comparison. Note that the eval protocols are different, but only subtly. Additionally, provide the comparison of recovered triggers with (Shen) jet al., I) in Fig. to demonstrate that the proposed method could also improve the quality of recovered triggers.
15

---

Published as a conference paper at ICLR 2022
h
Table 5: Comparison with (Shen et al. 2021).
Method
Roundl
Round2
Round3
Round4
Shen al. 2021
0.90
0.89
0.91
0.89
Ours
0.91
0.89
0.90
0.91
(a) Original image
(b) Shen et al. 2021)
(c) The proposed method
Figure 10: Recovered triggers compared with (Shen et al. 2021)
A.5 UNSUPERVISED SETTING FOR TROJAN DETECTION
Indeed, our method outperforms existing methods in different settings: fully supervised setting with annotated models, and unsupervised setting. In the main text (Tab. we have already demonstrated that with a limited number of annotated models, method outperformed others. To make fair comparison, following Neural Cleanse and DLTND, we also use the simple technique based Median Absolute Deviation (MAD) for trojan detection. We report the performance of Round 4 data of TrojAl in Tab. 6
Table 6:
Unsupervised perfor-
mances Trojan.
Method
AUC
ACC
NC
0.58
0.60
ABS
0.53
0.51
TABOR
0.52
0.55
ULP
0.54
0.57
DLTND
0.56
0.59
Ours 0.63
0.65
Because of the better trigger quality, due to the proposed losses,
our method outperforms baselines such as Neural Cleanse. We also note that, in Tab. 6 all methods (including ours) perform unsatisfactorily in the unsupervised setting. This brings us back to the discussion as to whether supervised setting is justified in Trojan detection (although this is not directly relevant to method).
From the research point view, we believe that data-driven methods for Trojan detection are unavoidable as the attack techniques continue to develop. Like in many other security research problems, Trojan attack and defense are two sides of the same problem that are supposed to advance together. When the problem was first studied, classic unsupervised methods such Neural Cleanse are sufficient. In recent years, the attack techniques have continued to develop, exploiting the entire dataset and leveraging techniques like adversarial training. Meanwhile, detection methods
16

---

Published as a conference paper at ICLR 2022
confined with only the given model and a few sample data. For defense methods to move forward and to catch up with the attack methods, it seems only natural and necessary to exploit supervised approaches, e.g., learning pattems from public datasets such as the TrojAI benchmarks.
A.6 IMPLEMENTATION BASELINES
We carefully choose the methods with available codes from the authors our baselines and follow the instructions obtain the reported results. And we list all the available repositories here:
Neural Cleanse: https//ggiibbboom////luaaaaaaooo
ABS: http::/ggiiuub.ooommhaayytt////s
TABOR: http:///iiiabbccmmm//mmmaa//AAA
ULP: http::/ghhhhboooo////iciiiiiiiir
DLTND:
Part of the reason for the strong benefit of method, well the ablated version (without the new losses), is because of the availability of the annotated training models.
A.7 COMPUTATION RESOURCES
We implement our method on a server with an Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz and Tesla V100 GPUs (32GB Memory).
17


Reviews:
[{"review_id": "l01dYy0UxE4", "reviewer": "Reviewer_6VNg", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths and Weaknesses:\n\n+ The paper is very well-written.\nIt flows well and provides extensive explanations on abstract topics such as persistent homology to help the reader understand them easily.\n\n+ The proposed approach seems novel and promising.\nThe experimental results are comprehensive, and are done on synthetic (MNIST and CIFAR-10) as well as real-world (TrojAI) backdoored models.\nIt is shown that using the proposed approach one can get significant improvements over state-of-the-art baselines such as Neural Cleanse [[1](https://ieeexplore.ieee.org/abstract/document/8835365)] and DL-TND [[2](https://arxiv.org/abs/2007.15802)].\n\n+ Although the use of topological loss seems to be enhancing the detection of backdoor models, an explanation as to why this choice is valid is somehow missing.\nIt is pointed out that this way the topological loss would limit the search space of all possible triggers, which as a result would increase the chances of finding the true one.\nWhile this makes total sense, it also brings a natural question to mind: are all the possible triggers compact?\nIn other words, how does the introduced topological loss behave in case the adversary poisons the model with a scattered trigger?\nI think this question needs to be answered to help the readers understand why a topological loss needs to be employed.\n\n### Minor Comments/Questions:\n\n+ To minimize Eq. (2), a sum over all the validation samples is taken, or they are minimized separately? If it is the latter, please add a summation to indicate this.\n\n+ In Table 3, the number of samples means the number of models in the training data or the number of validation samples used for reverse engineering?\n\n+ In the assumptions of Section 3.1 $\\mathbf{x}$ needs to be in $\\mathbb{R}^{3\\times M \\times N}$ to accommodate the three RGB channels.", "summary_of_the_paper": "This paper presents a method for the detection of backdoored neural networks.\nIt is assumed that the user only will have access to the trained model and a few clean validation samples, not the training data.\nAs such, a reverse engineering approach is taken to solve the task: i.e. using the validation samples, the method aims to recover possible backdoor triggers.\nTo this end, a loss function with three terms is minimized:\n\n1. a _label flipping_ loss to extract triggers that can subvert the classifier's decision,\n2. a _diversity_ loss that aims to generate various triggers with different patterns, and\n3. a _topological_ loss that tries to encourage the compactness of the recovered triggers.\n\nBased on these loss terms, triggers are reverse engineered for a dataset containing malicious and benign models.\nThen, using these triggers a set of features are built for each model (Trojaned vs. benign).\nUsing these features, a binary classifier is trained to predict if the models are backdoored or not.\nThe performance of the proposed approach is tested under various experimental settings and ablation studies.", "summary_of_the_review": "This work introduces a novel approach for the detection of backdoor neural networks.\nThe performance of the approach seems to be promising.\nHowever, there are some speculations around the use of topological loss for detection as the compactness of the triggers is not a necessity in backdoor attacks.\nIf the authors can elaborate on this assumption, I would be happy to increase my score.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "YtXYhjd8kuW", "reviewer": "Reviewer_Tmo4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The introduction of the diversity loss and topological loss is interesting and seems to improve the quality of generated triggers. The evaluation is conducted on a large benchmark. I have the following concerns with the paper.\n\n1. The current approach seems to heavily rely on a large set of models with the same or closely related task. A reasonable detection result (>0.8 AUC) requires at least 50 models as shown in Table 3. This does not seem practical. How can a defender acquire such a large set of clean models? Even if the defender can obtain a set of clean models, the task for these model also need to be similar, e.g., traffic sign detection in TrojAI datasets. What if these models have different tasks? Can the proposed approach still work? The paper needs to be better motivated why having a large set of clean models with similar task is practical and realistic.\n\n2. Following up the first point, the comparison with several baselines does not seem fair. NC, ABS, and TABOR do not require any clean models for training. The detection is directly carried out on a given model (either benign or trojaned). The proposed approach relies on the assumption of having a set of clean models, which has a different threat model than these baselines. A fair comparison would be some modified version of NC, ABS or TABOR that also takes a set of clean models into consideration during detection. For instance, the defender could first run these baselines on the clean set and get an estimation of how clean models would behave. She then uses this information to guide the detection.\n\n3. A recent state-of-the-art approach (Shen et al.) is not compared. The paper only briefly mentions it in the related work but does not empirically compare with it. The results from the original paper (Shen et al.) are very similar with marginally differences. The experiment of the proposed approach is only evaluated on 10% of the TrojAI datasets as it requires training and validating on the remaining 90%, whereas Shen et al. evaluated on the whole set. The proposed approach can be better compared with the baseline by presenting the results from TrojAI test set that is evaluated remotely by the TrojAI organizer. This can rule out the possibility that the approach does not overfit on the local training set as this is a training based method.\n\n4. There is no discussion and evaluation on adaptive attacks. When knowing the existence of the proposed detection, the attacker can trojan a model by maximizing the mask using the losses in Equation (2). She can also add those reverse engineered triggers in the training set during poisoning to counter the effect of generating a set of small triggers. The trojan detection network can also be considered by incorporating in the loss function.\n\n5. The paper does not clearly separate the contribution of existing works and this paper. For instance, the trigger reverse engineering technique by using a mask and a pattern in Section 3.1 is proposed by NC. Utilizing continuous values instead of binary values for the mask is also proposed by NC. This should either be discussed in a section regarding existing works or explicitly credits the contribution to existing works.\n\n6. From the ablation results in Table 3, without the topological loss or the diversity loss, the results are already better than evaluated baselines. Why is this the case? What are other factors contributing to the final good detection results? Without the above two losses, it is more or less NC, which should have a very low detection result. Do the author care to explain in detail which component of the approach leads to the good results.\n\n7. The presentation needs improvement. Equation (2) introduces three losses, namely, the label-flipping loss, diversity loss, and topological loss. The first two losses are discussed in Section 3.1, while the topological loss is presented in a separate section, Section 3.2. These three losses should be at the same section level. The caption of Figure 5 is hard to be distinguished from the main text. There are no images with the injected triggers in Figure 6.", "summary_of_the_paper": "This paper proposes a trojan detection method using reverse engineering techniques. It improves existing approaches by incorporating a trigger diversity loss and a topological loss during trigger generation. The diversity loss aims to generate a set of triggers with different patterns and locations. The topological loss is to make triggers more connected with fewer perturbations spread out on the input. The paper builds a neural network to detect trojaned models by using the features of generated triggers such as the number of trigger pixels, the mean and standard deviation of triggers in horizontal and vertical directions, etc. This paper evaluates the proposed approach on two synthetic datasets and TrojAI benchmarks.", "summary_of_the_review": "1. The setup does not seem practical.\n2. The comparison with existing works is unfair.\n3. There is no comparison with a state-of-the-art approach.\n4. The paper is not evaluated on adaptive attacks.\n5. It is not clear about the contribution of different components.\n6. The presentation of the paper needs improvements.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "S3n34YHxucM", "reviewer": "Reviewer_momo", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The experiments show fantastic Trojan detection accuracy under the TrojAI benchmark settings.\n2. The idea of the diversity penalty is simple and straightforward. I like the consideration of unknown target labels and multiple triggers, which may be the closest to the real scenes.\n3. A very detailed ablation study for the contribution of each item.\n\nWeaknesses:\n\n1. It took me a while to understand the idea of topological simplicity and why such prior exists in the trojan attack settings. The motivation needs to be further clarified. For example, can the embedded trigger be a set of scattering points of pixels? If can, will the topological simplicity exist? This also leads to another question about the quality evaluation of recovered triggers. In fig 6, It is doubtful to consider the “more compact” as a good metric for triggers.\n2. The second limitation of the proposed method is the training of the detection network, which needs annotation of the trojan model. To my knowledge, a very unique characteristic of a trojan attack is its stealthiness. Once the type and method of attack are exposed publicly, the threat does not exist. Are there any possible ways to apply the proposed method in a more realistic scene without the annotations of the trojaned model? \n3. It will be better if the authors provide a more detailed implementation of baselines since the performance increase is very astonishing. Also, a clear analysis of the impact of trigger types is better to be exhibited.", "summary_of_the_paper": "This paper presents a novel method for trigger detection against trojan attacks (especially the TrojAI benchmark). The key idea is to introduce the diversity penalty and the topological simplicity to help find more different high-quality triggers in the image domain.", "summary_of_the_review": "This paper is well-written and proposes a pretty novel trigger hunting method. However, the limitation of this method is also very obvious, not good enough for real threats. I will increase my score if the authors can provide more insights under more general settings of trojan attacks except for only TrojAI benchmarks.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "QeS0QnckJEc", "reviewer": "Reviewer_3UGX", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper proposed a novel target-label-agnostic reverse engineering method for reverse engineering in recovering trojaned triggers on clean images. The proposed diversity loss and topological prior significantly improved the performance in finding trojaned triggers and the quality of the found triggers compared to other existing methods. The proposed method is novel, the experiments were well conducted, and analysis was well performed. ", "summary_of_the_paper": "This paper proposes a diversity loss and a topological prior to not only increase the chances of finding the appropriate triggers but also improve the quality of the found triggers. These loss terms significantly improve the efficiency in finding trojaned triggers. The experiments results show that the proposed method performs substantially better than the baselines on the Trojaned-MNIST/CIFAR10 and TrojAI\ndatasets, respectively.", "summary_of_the_review": "This paper proposed a novel target-label-agnostic reverse engineering method for reverse engineering in recovering trojaned triggers on clean images. he proposed diversity loss and topological prior significantly improved the performance in finding trojaned triggers and the quality of the found triggers compared to other existing methods. The proposed method is novel, the experiments were well conducted, and analysis was well performed. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a diversity loss and a topological prior to not only increase the chances of finding the appropriate triggers but also improve the quality of the found triggers. These loss terms significantly improve the efficiency in finding trojaned triggers. The experiments results show that the proposed method performs substantially better than the baselines on the Trojaned-MNIST/CIFAR10 and TrojAI datasets, respectively. This paper shows detailed ablation study with great empirical performance. Some reviewers have doubts about the experimental comparisons and some of the assumptions made in the algorithm. Overall, the thorough experimental investigation fo the proposed method makes this paper worthy of publication and widely being shared.
Decision: Accept (Poster)