Topological Detection of Trojaned Neural Networks
Authors: Songzhu Zheng
Yikai Zhang
Hubert Wagner
Mayank Goswami
Chao Chen
Venue: NeurIPS 2021 Poster (2021)
Keywords: Topology
Persistence Homology
Trojan Attack

Abstract: Deep neural networks are known to have security issues. One particular threat is the Trojan attack. It occurs when the attackers stealthily manipulate the model's behavior through Trojaned training samples, which can later be exploited. Guided by basic neuroscientific principles, we discover subtle -- yet critical -- structural deviation characterizing Trojaned models. In our analysis we use topological tools. They allow us to model high-order dependencies in the networks, robustly compare different networks, and localize structural abnormalities. One interesting observation is that Trojaned models develop short-cuts from shallow to deep layers. Inspired by these observations, we devise a strategy for robust detection of Trojaned models. Compared to standard baselines it displays better performance on multiple benchmarks.

Topological Detection of Trojaned Neural Networks
Songzhu Zheng', Yikai Zhang', Hubert Wagner', Mayank Goswami", Chao Chen
Stony Brook University, {zheng songzhu, chao chen.
wtw..........tttt...e..oeee..a.. 10 edu *Morgan Stanley, Yikai Zhangdmorggasttanleyy com com
University of Florida, hwagner@ufl eddu CCity University of New York, mayank. goswami@qc .
Abstract
Deep neural networks are known to have security issues. One particular threat is the Trojan attack. It occurs when the attackers stealthily manipulate the model's behavior through Trojaned training samples, which can later be exploited. Guided by basic neuroscientific principles, we discover subtle - yet critical structural deviation characterizing Trojaned models. In our analysis we use topological tools. They allow us model high-order dependencies in the networks, robustly compare different networks, and localize structural abnormalities. One interesting observation that Trojaned models develop short-cuts from shallow to deep layers. Inspired by these observations, we devise a strategy for robust detection of Trojaned models. Compared to standard baselines it displays better performance on multiple benchmarks.
Introduction
Recent years have witnessed rapid development of deep neural networks (DNNs) 3 25
58
16
However, due to their high complexity and lack of transparency, DNNs are vulnerable various malicious attacks 12. 56]. This paper focuses on one type of data poisoning attack called the Trojan attack [23]. In this scenario the attacker injects Trojaned samples into the training dataset for example by using incorrectly labeled images overlaid with a special trigger. At the inference stage, the model trained with such data, called Trojaned model, behaves normally on clean samples, but makes consistently incorrect predictions on the Trojaned samples.
The challenges in identifying such attacks stem from the confined setting: the user has access only to the DNN model and few clean samples. In such data-limited setting, methods requiring dense sampling
very practical. Instead, state-of-the-art methods often follow reverse engineering strategy [59 43 62). Starting with a clean sample, they try to reconstruct a Trojaned sample that can change the prediction. Network's 's response to such reverse engineered sample can help determine if the network was indeed Trojaned. However, in practice the search space for triggers huge, and efficient, reliable detection has proven challenging so far.
Previous approaches treat a neural network as black-box, only inspecting the dependency between its input and output. In this paper, open the box and look into the internal mechanisms of the model. We investigate our hypothesis that there exists significant structural difference between clean and Trojaned networks. To this end, we follow classic adage of neuroscience. "Neurons that fire together, wire together" [26]. We consider neurons with highly correlated activation as wired together even if they are not directly connected in the network. Unfortunately. direct inspection of such connectivity is not sufficient, presumably due to the high heterogeneity of models and data.
To overcome this issue, propose higher-order structural information of neural networks. Our method uses tools from topological data analysis, particularly persistent homology 18 With principled algebraictoopological foundations
more advanced tools which allow us model more subtle,
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

47]. these tools are perfectly suited for modelling higher-orde structural information. We use them to capture salient topological structures particularly the connected components and holes present in the aforementioned neuron connectivity graph.
Equipped with topological tools, compare clean and Trojaned neural networks. We observe significant discrepancy between their topology- - and quantify this difference by comparing topological descriptors called persistence diagrams. We can go step further, as the tools allow us to localize the topological aberration - revealing presence of highly salient loops spanning the Trojaned models, absent from the clean models!
Trying to understand the implications of our observations, we ask: What does the topologial abnormality reveal about Trojaned network? We claim that these loops reveal strong short cuts that connect neurons from shallow and deep layers bearing resemblance to the neuroscientific concept of reflex arc. This is sensible in Trojaned models, the classifier has to switch prediction once it sees a trigger. The deep layer neurons (closer to prediction) have to be highly dependent on some shallow layer neurons (closer to input).
Our empirical observations are substantiated a theoretical result. Theorem[ states that given sufficient samples, the topological descriptor is provably consistent. This result serves as sanity check, showing that what we observed was not a fluke. We conclude by proposing a sopology-basee Trojan detection algorithm. In realistic data-limited setting, experiments on symthetic and competition datasets show that our method is highly effective, outperforming existing approaches. The topological detector can help mitigate the security threat posed by Trojan attacks.
The code of this paper can be found at http::///tttbbboc///////l/////pr
1.1 Related Work
Trojan detection. Early works Trojan detection use both clean and Trojaned samples. Chen et al. [9] inspect the representation of all samples at the penultimate layer of the neural network. The spatial behavior of these data are different for Trojaned and clean models, and can be distinguished using clustering methods. Gao et al. (21) use the entropy of model prediction over all training data to decide whether model is Trojaned. These methods require all training data, including the Trojaned ones; this is not realistic at real world deployment.
For realistic lima-li setting, reverse engineering strategy has been widely adapted. Wang et al. [59] craft and recover the unknown triggers through optimization. Random initialized triggers are mixed with clean images and gradient descent is used find the trigger that can alter the prediction of the network. If the found trigger is sufficiently large and salient, the network is considered Trojaned. Other works largely follow similar strategy to recover triggers, but the recovered triggers in different ways [4 24 32 66). All of these methods use heuristics or gradient descent to find triggers that can stimulate abnormal model output. They focus heavily on dependency between input and output. Few methods investigate the information flow and exploit neuron interaction.
Topological analysis of neural networks. Persistent homology was introduced to measure topo- logical property of data in a robust and quantifiable manner [18] Since its introduction [19 70 a
great amount of theoretical progress has been made: in stability of persistence diagrams 13 7 in algorithms
46 14 [0]. and in proving various statistical properties 200 3. In machine learning,
topological information has been used for clustering 88 49]. In the supervised setting, classifiers
based on topological features have been proposed via direct vectorization [1, kernel machines [53] 38 36 6). and convolutional neural networks (37). Topological information has also been used
in the analysis of images (29 30
61
63
and graph-structured data (37 68
27
69
66, [].
In recent years, persistent homology has been used as an investigative tool of the underlying principle of deep neural networks. One hypothesis is that the topology of the data at deep layer representation can be correlated to the behavior of a neural network [48). It is shown that the topology of the
decision boundary can be indicative of the generalization power of a classifier [52] 41). With
the recent invention of differentiable topological loss, one may enforce priors such topological
simplicity to improve the performance of deep neural networks 11 28
Wu et al. 64 use the
topology of the data representation to filter noise in the data.
We remark that from a purely mathematical perspective this localization is straightforward operation but achieve this practical datasets we had to push the boundary of existing computational tools.

---

Trigger
Trajaned Dataset
Trajened
9
9
9
Training
9
7 7
"g" Trigger Sensitive
Trigger
(b). Trojan Attack
(a). Trojaned Examples
Figure 1: An illustration of Trojaned Examples and Trojan attack. (a). Trojaned examples from MNIST, CIFAR IO and NIST TrojAI competition dataset. (b). To inject backdoor, we add trigger (a white À pattern the upperleft corner) to images of digit 9, and assign label 0 to them. After training, the Trojaned model predicts a normal/clean digit 9 image to be class 9, but predicts class if it sees digit 9 image with the trigger. A normal (or clean) model will ignore the trigger and still predict a triggered digit image as class 9.
An alternative strategy to apply persistent homology is to treat the neural network architecture as the underlying topological space, i.e., treating all neurons as nodes and their connections edges [54 45 39]. Notably, such approach was used for the detection of adversarial examples [22]. These methods are restricted to the original network architecture, only focus O-dimensional topological feature, and thus cannot capture long range neuron interactions between shallow and deep layers.
Corneanu et al. [15] builds a filtration of neuron connectivity using the Pearson correlation matrix among neural activation. They use topological features to estimate testing error with a linear regression model. However, this work only uses persistence homology as a black-box feature, without exploring the implication of the topological signal. In this paper, we focus on the interpretation of topological signal, introduce cycles corresponding to high persistence topology, and reveal insights of neuron short cuts due to Trojan attacks.
Outline. In Sec. 2 we introduce Trojan detection problem. In Sec. we explain how to extract topological features from given neural network models. In Sec. we show that there does exist difference in topology between Trojaned and clean models. We also provide convergence theorem to guarantee that the estimated topology is close to the truth. In Sec. []] we extend the idea to a realist setting and propose an automatic Trojan detection algorithm. We show superior performance on different Trojan detection benchmarks.
Problem: Trojan Detection
Trojan attack (also called backdoor attack) of deep neural networks was first introduced by Gu et al. (23). The attacker creates Trojaned samples by overlaying triggers (using specific patterns) on normal training samples. These Trojaned samples are assigned specific targel class labels different from the labels of the original training samples. These Trojaned samples are mixed into clean samples. Training with such Trojaned dataset leaves a backdoor in DNN. The Trojaned model makes expected prediction on normal data. But when it sees trigger, it will behave abnormally and misclassify the data as the target class. See Figure I for an illustration.
Newer and more sophisticated Trojan attacks have been proposed to use less Trojaned data to achieve better trigger stealth 12
55] [57]. There are also Trojan attacks targeting domains These are beyond the scope of this paper.
beyond computer vision B35 65
We now formalize the above intuitions. Denote the set of clean samples as D (X,y) and the set of Trojaned samples as δ = (X, 7). The inputs of Trojaned samples are X = (w き - m.) C m 3 x E X3 with modified labels ÿ {W y/ f Vve, where m is the mask indicating the position of the trigger, ó is the content of the trigger and O is the Hadamard product. A Trojaned model f is trained with the union of the clean and Trojaned samples D and Π. When the model f is well trained, it will make abnormal prediction when it sees the triggered samples f()) V/ # ym. but will give identical prediction clean model does whenever a clean input given, i.e., f(x) = ff()) = y. baring some expected prediction error.
The task of Trojan detection is to determine whether given model is Trojaned or clean. We will start our investigation with a full-data setting: we have access to training samples both clean and Trojaned. In Sec. 4 focusing on such ideal setting, we validate our hypothesis and show that
3

---

Trojaned and clean models are significantly different in topology. In Sec. 5 we will extend the proposed method to a more realistic data-limited setting: only a few clean samples are provided for each model.
3 Method: Neuron Correlation, Persistent Homology, Cycle Representatives
Next, we present the main mathematical tools for this study, namely the Vietoris- construction and persistent homology. Due to space constraints, we only provide intuitive description, leaving technical details and a formal description to the supplemental material and a formal textbook [18].
We start by providing some intuitions, which are formalized later as necessary. The entry point for our considerations is the connectivity graph based on the correlation of neuron activation. In the next step, we consider the simplicial complex generated by the cliques of this graph and filter it with different thresholds. This is often called the Vietoris-Rij Rips filtration. As the threshold changes it captures various topological structures as they are born and die. We consider the lifespans of these structures as an essential characterization of the neural network. Further, we view the associated geometric structures crucial for interpreting the behaviour of the network- in particular the discrepancy between the clean and Trojaned networks.
We mention that this construction can be viewed way of approximating the topological behaviour of the underlying metric, or dissimilarity, space. More concretely, it approximates the patterns in which metric balls of increasing radii intersect both as pairs and in larger subsets. Formal explanation of this aspect of this construction is beyond the scope of the paper, however we believe that the intuitions we offer are sufficient to grasp the crux of our approach.
The neuron connectivity graph and its simplicial complex. We study a neural network with m neurons. Each neuron is denoted by e for E [mm] [mm] and may belong to any layer of the network. By feeding a set of n inputs X = { F.. - either clean or Trojaned- - through the neural network, we record n-dimensional activation vector for each neuron. This activation vector is denoted by (X) E R", for i E [m]] Each position, 8, (æz) E R, represents the activation value of i-th neuron given 1-th input. Now, for any pair of neurons, Us Uj we let V (X;,,, uy (X))
-either
EiEta) (2;)), (ax)) given a measurement (u, v) R x R + [-R, RJ, where R is a positive
constant. We calculate the generalized correlation according to Pt. (W(,,.)))W,,....) where
1/ ((v..,))
(1; 1z) is a shorthand for (v;(),, (x),)))))))))) (x))). We call the m x m correlation matrix M = _. We construct a weighted complete graph with m nodes, representing all the neurons, and m(m 1)/2 edges connecting all pairs of neurons. Let the edge weight be wij = 1 Pi.j- This provides a pairwise dissimilarity between neurons that is negatively proportional to their correlation We denote this graph by GM-
To model the underlying topological space, we extend the graph to higher order discretization called simplicial complex Fl The complex, denoted by S, is collection of discrete elements including nodes, edges, and triangles. These elements are called O-, 1-, and 2-simplices respectively. The nodes and edges are those of graph GM; the triangles are spanned by any three nodes of the graph, i.c., (i. 1 S < < k S m.
Vietoris-Rin Rips filtration. We assign a filter function to all elements of the complex, ΦM SS - IR For any node i, Φss(t)
O. For any edge (i.j), we use the weight function, Φss (i. = Wa.j- For any triangle (i,j,k) we take the maximum of its edge function values: ( max (ox (i. dw (i, k), (br(i, For the rest of the paper, we may drop M and simply use e when the context is clear. With the filter function, we may use any threshold / to filter elements of the complex, and keep the remaining as a sublevel set, S; = fa E S (()) S (). We start with t -oo continuously increase it until t co. As the threshold increases, the sublevel set grows from an empty set to the whole complex S. Formally, we have a filtration induced by o, 0 = St C S, C C S, S.
=
Lifespans of topological structures and persistence diagrams. Through the filtration, topological features such as connected components and holes can appear and disappear. A O-dimensional (OD)
2 Note that this is not a proper metric distance. However this does not affect our topological construction. "In this paper, we focus on 2-dimensional simplicial complexes. Please note that both the intrinsic and extrinsic dimension of the modelled space may be much higher.

---

topological structure is connected component. Its birth time is the smallest function value over all its nodes. The death time is when the component merged with another one born earlier. An I I-dimensional (1D) hole appears as closed loop. It disappears when it is sealed up by set of triangles. Figure in supplementary material shows a large 1D hole appearing during the filtration, as well as many small ones. We represent these topological structures (OD and 1D) as dots in 2D plane called a persistence diagram. The coordinates of each dot are the birth time and the death time of the corresponding topological structure. The persistence of a dot is the difference between its death and birth times. The persistence diagram, denoted by Dg [M, S), depends on both the underlying simplicial complex and the filter function (which is determined by the correlation matrix M). We also note that one can compare two persistence diagrams using the bottleneck distance [3, do (Dg(M,, (g2 S)). Formal definitions and technical details can be found in the supplemental material.
Topological features and eycle representatives. Our focus is twofold: 1) quantifying the difference between Trojaned and clean networks using their persistence diagrams; 2) localizing the root cause of this difference using cycles of high persistence. Despite a rich literature on learning with persistence diagrams [1 38 37 36), we stress interpretability and focus on simpler features, such as maximum persistence, average mid-life deathh) /2), average death time, etc. In Sec. 4 we will use these features for statistical testing. In Sec. [ we will use these features to devise an automatic Trojan detection algorithm. Finally, we look at the cycles corresponding to the dots of high persistence, which allows us to zero-in on the compromised paths in the network.
To interpret the topological signal, we inspect the topological structures that are strong contributors to the aforementioned topological features. For example, in the case of maximum persistence we focus our attention to the dot with the highest persistence. For a selected persistence dot, we analyze a cycle representing the corresponding topology. We recall that for ID topology, the representative cycle of a persistence dot is collection of edges which is created at the given birth time and is sealed up at the given death time. Viewing the path as a sequence of nodes provides a good intuition of the relevant topological hole although we have to admit the cycles are not unique 663 67 17. We
focus on one way of extracting the representative cycles, which is efficient and worked well in other types of upplications, e.g., in image analysis [60]. The algorithm involves inquiry and optimization of the classic matrix reduction algorithm for the computation of persistent homology [18]. More
algorithm details and efficiency analysis will be left for a future journal version of the paper.
4 Analysis: Topological Difference Between Trojaned and Clean Models
In this section, we investigate the topological difference between Trojaned and clean models. In Sec. 1 we first create a synthetic distribution, in which we observe different topology (persistence diagrams) from Trojaned and clean models. Reassured by this synthetic example, in Sec. 4.2 we carry out an empirical study on Trojaned model trained on MNIST dataset. We observe a statistically significant difference between the topology. Finally, in Sec. 4.3, we show that with sufficient samples, the estimation of topology is sufficiently close to the true topology of the neural network. Thus the empirically observed structural gap between Trojaned and clean models is real.
4.1 The First Example: Synthetic Distribution
We first define a synthetic distribution and create a Trojaned dataset from this synthetic distribution. In Proposition [ we prove that the resulting Trojaned model is different from clean model in terms of their persistence diagrams. Proof and illustration can be found in the supplemental. We note that this example does not necessarily resemble weal-world Trojaned dataset. The goal of this synthetic example is merely to demonstrate feasibility.
Example (Trojaned Mix-Gaussian Triplet). Let μn 2(-e2 -ajo e log(()) 12 = 2(-e2 +
v /og(()) )) μa 2(ez + t2) / (og())) Leti ~ uniform(() 2}) ei)or /tor///gtt (lg(()) μ3= 2(e² e1 )1)0 / and j uniforr(((1 2, 4)). We define the following pair of distributions (D1, D2, D3) to be
5

---

Trojaned Mix-Gol Gaussian Pair (see supplementary material for demonstration), where:
D (Originall data) {(x, y) : (N,,, Ta), y = i 2}
D2 Trojaned feature with correct labels) = [(zz y) N(n, 0 La). y - jMD 2} Da Pa(Trojaee featuuree with modified labels) = {( y) : (N,,, Ta). y 1 (2,3)))
We study the hypothesis class H. of binary output neural networks with two hidden layers and four neurons in each hidden layer equipped with an indicator activation function. The following theorem shows that the Trojaned model and the clean model have different persistence diagram, i.c., with bottleneck distance 0.9. Recall the correlation matrix M depends on the classifier f and the sample set used to estimate correlation, D. For completeness we use (J,D) instead of M.
Proposition 1. Let (D1, D2, Dz) be Trojaned Mix-Gauusian Pair and H be the hypothesis class defined as above. Let x(f...yV) ((s.))),, [R( I 7) B(z,g)=Da [R(f,, I, 8)] S n, (R,, 2 3.2 suuch that the bottleneck distance between the ID persistence diagrams satisfies: delDg(M D2), (J2 D2), 2 0.9.
1 (f(z) # v)- There exist f1, E H. where
Trojaned
400
1-Test P.val=0.000
trojan kde clean kde trajen
trojan kdd
kde trojan clean
9
300
cloan
Brih
Test P P.val-0.000
99
Models
0.000 0.006
-0.2 0.0
0.2
0.4
0.6
0.8
1.0 1.2
(a) Extract Topological Features
(b) Average Death Time
(c) Maxium Persistence
Figure 2: Hypothesis Testing, (a) schematic illustration: Trojaned datasets are provided clean and Trojaned models. Their correlation and then persistence diagrams' features are extracted. (b). Distribution of OD diagrams' average death time for Trojaned models (red) and clean models (green). Dashed lines are the kernel density estimation. P-value between the two distributions < 0.000. (c). Distributions of 1D diagrams' maximum persistence for Trojaned and clean models separately. P-value between the two distributions < 0.000.
4.2 An Empirical Study: Statistical Analysis of Trojaned Model
In this section, we carry out a statistical inference with MNIST to investigate the structural difference between Trojaned and clean models. We trained 70 ResNet18 with clean MNIST dataset and another 70 ResNet18 8 using Trojaned MNIST dataset. Both groups of models have similar performance on clean testing images. Only Trojaned models will misclassify Trojaned images with high probability. Clean models will not be affected and will make correct prediction in spite of the trigger. Please refer Sec. 15 for a more detailed description of the data generation procedure.
We extract topological features following the procedure introduced in the last section. As demon- strated in Fig. (a), samples from the Trojaned dataset containing both clean and Trojaned examples are supplied to all the 140 networks. Neurons' activating values are recorded into a vector and is calculated between all pairs of neurons. For each model, we build the
the pairwise-correlatee simplicial complex, filter it based on the correlation, compute the persistence diagram, and extract topological features.
Please note that so far, to verify our hypothesis and to investigate its implication, we were using the full-data setting, i.c., using all training data to calculate neuron activation correlation. While this gives us the full picture of the network connectivity, and more reliable topological characterization, this is not realistic setup for Trojan detector. We will discuss how to extrapolate this to more realistic limited setting in Sec.
Two topological features stand out. clearly differentiating Trojaned models and clean models: average death time of OD homology class (connected components) and maximum persistence the ID homology class (cycles). As shown in Fig. (b), the average deaths of connected components in Trojaned models are significantly smaller than those in clean models. The two-sample independent t- test is rejected at 99% significance level. Note that here the filter function is minus the correlation.
Results.

---

This implies that neurons in Trojaned models on average have larger correlation, and potentially tend have larger cross-layer correlation. Possible explanation: the extra capacity used Trojaned model to learn the trigger pattern, which causes more active neurons and consequently neurons are more likely to be correlated with each other through intermediate neurons.
Meanwhile, Fig. 2[c) c) shows significant topological signal in the maximum persistence of ID homol- ogy. Intuitively, there exists a ID cycle in Trojaned model that has significantly longer persistence than that in clean models (the two-sample independent (-test is rejected at 99% significance level). We inspect this phenomenon by identifying nodes and edges contained the most significant cycle (Fig. B) For Trojaned model, the most significant cycle contains an edge linking a shallow layer and deep layer. This is not the case for clean model where a cross-layer edge is hardly ever spotted.
(a) Clean Model+ Trojaned Input
(b) Trojaned Model+Trojaned Input
Figure 3: Most Persistent Cycles in ResNet18 with Death Time Cutoff at 0.35, on a clean (a) and Trojaned model (b). On the Trojaned model, the loop consists of short cut connecting shallow and deep layers.
Structural insight: the most persistent 1D cycle captures the short-cut. We observe the high- persistence cycles of Trojaned models often contain strong-correlation edge connecting shallow and deep layers. We hypothesize that these cross-layer edges forms a short cut unique to Trojaned models. Neurons connected by the short-cut tend to fire together. This is sensible: Trojan triggers are often localized pattern. They will be identified by shallow layer neurons. Meanwhile, for Trojaned network, the final prediction can be highly dependent on the identification of the trigger. Thus, there could be deep layer neurons (close to prediction) that are strongly connected to some shallow layer neurons (which activates when Trigger seen). See Figure 3 for illustrations.
4.3 Theoretical Guarantees
We conclude this section with theoretical guarantee that the estimated persistence diagram will converge to a true one given sufficient samples. We prove the convergence a population level.
Given potentially corrupted models f1:N and corresponding test input X1:N, a natural practical concern about obtaining high quality approximation is the sample size requirement for each dataset X2. & E [V] In particular, one needs to ensure that for all models the empirical estimation is faithful. A brief analysis shows we only need O bog(W))+oog(m))ooggg
samples as a minimum
ea
requirement for all X₁ to ensure that with high probability empirically estimated persistence diagram (M ()), S) is sufficiently close to the ground truth Dg(M(f. (),) terms of bottleneck distance. We provide proof in the supplemental material.
Theorem 1. Let M (fr. X4) E R Xmnx with my S m Vk E [N] and its entries M (X
(Xl.((X))..(Xx))))MMM (Xx)) and the its target value (M Dz) E mmmxxm with its en- ExpeD, (Nx,
tries 2 Ex (9( (x). (X, as defined in section with (v,(X), u (X)) = - - Exyex (x)() (x))). Suppose Vk E [N], are i.i.d sampled from distribution Dj and (x)(()),, 1y (x)) | S R for all x ~ Dr. vs Ug 0 < < Ee-D, (11,())) (x for all i [m&]. we have vk [N].
16R (log(N) NN 2 log(m") log(
X
r4g2
then with probability at least 5, for all & E (,M D2), Se.
7

---

Remark. With the convergence theorem, it is not hard to show the following statement: Given sufficiently many samples, if we observe gap topology (persistent homology) between the estimated Trojaned and clean models, the gap likely also exists between the true models.
5 Application: A Topological Trojan Detector in Data-Limited Setting
In this section, we introduce an automatic Trojan detection algorithm based on our observation about Trojaned models' topological abnormality. The Trojan detection problem is essentially classification problem. Given set of training models, each clearly tagged as Trojaned or not, can we learn a classifier to predict whether test model is Trojaned or not. Based on our previous study, we believe topological features can differentiate Trojaned models from clean ones. Our idea is to extract topological features from these models, and use them to train a classifier to predict the Trojan status of test model. We have in total 12 topological features, including maximum persistence and average death (see supplemental for a complete list). We use a standard MLP (multi-lime linear perceptron) classifier. We will also provide the results using more powerful learning methods for persistence diagrams (e.g., using (6) in the supplemental.
The major challenge is the limitation of data access. In practice, the Trojaned dataset will not be available to users. We adopt the data-limited setting: for each model (training or testing), only a few clean sample images are given. To acquire sufficient samples to estimate the correlation of each model, we apply a pixel-wise perturbation strategy. formal algorithm supplemental material. Given clean sample image, we iterate through every pixel (or small patch) and modify its value. Then such modified examples are all provided to the model as samples for building the correlation matrix.
this provided in
To confirm that this sampling strategy is sufficient in mining the topological structure, we earry out the same hypothesis testing as in Sec. 4.2 except that we use the perturbed samples instead of the Trojaned dataset. As shown in Fig. we still observe significant topological difference between the Trojaned and clean models. This gives us sufficient confidence to use topological features for Trojan detection, with the perturbed samples.
trojan kdl
trojer kde
P.ual-0.000)
clean trojan
clean I:de
30
trojan
dean
clean
20
2000
25-
500
5-
0.0
0.1
0.2
0.3
0.4
2.5
0.9980
0.9985
0.9920
0.9995
1,0000
(a). Ave DeathTime- Perturbed Inputs
(b). Max-Persist Perturbed Inputs
Figure 4: Ideal Feature Distribution v.s. Practical Feature Distribution. (a). Average death time
calculated using real Trojaned data. (b). Average death time calculated using pixel-wise perturbed data.
Formally, we propose our Trojaned network detection algorithm in Alg. We will release all code for purpose of reproducibility. We validate our Trojan detector on synthetic and competition datasets, comparing with So'A baselines.
Synthetic Dataset Experiment. We generate our synthetic dataset using NIST trojai toolki In
synthetic datasets, trained 140 LeNet5 40) and 120 ResNet18 (25] with MNIST 40 separately.
We also trained 120 ResNet18 and 120 Densenet121 [1 with CIFARIO [4] separately. Half of these models are trained with Trojaned datasets where manually applied 20% one-to-one Trojan attack. Specifically, for Trojaned databases, we picked one of the source classes and added a evvers--lmbba--shapee trigger (Figure T to a random corner of the input images. Then we changed the edited images' class to predetermined target class and mixed them into the training database. Trojaned models trained with these pollutant databases and clean models are trained with the original clean database. Furthermore, Trojaned models trained with MNIST datasets are constrained maintain at least 95% successful attack rate (frequency of predicting the target class when trigger
com htnp ahop
8

---

Algorithm Topological Abnormality Trojan Detection
1: Input: Training set models (J1, f2, ... J, Testing input associated with each model X
(X1, X2.... Xy), Ground truth indicating Trojaned or not {11, "1 ,yN
2: Output: Trojaned model detector y
3: for = 1,..., .. do
4: X Pixel-wise Perturb (X)
5: Calculate correlation matrix M (fi.X;)
6: Build filtration of VR complex @ [ St, St, S St, using M X p.
7: Extract topological feature 23 (S) as described in section
8: end for
9: Train Trojan detector with features Z = {z), 22, ; and Label
is presented on the test image) and models trained with CIFARIO are constrained to maintain at least 87% successful attack rate. At the same time, MNIST models and CIFARIO models also need to maintain at least 97% and 80% testing accuracy on clean inputs separately. There are no significant difference in terms of testing accuracy between clean models (on average 99% for MNIST and 84% for CIFARI0) and Trojaned models (on average 99% for MNIST and 84% for CIFAR10).
We also our method with the recently proposed label consistent Trojan attack [57]. Such attack injects triggered input without modifying labels, thus can be potentially more stealthy. We follow the attack setting in the original paper. We synthetically inject 20% Trojan into target class 0 of CIFAR10. We use linear interpolation as the triggered sample generation method with hyper-parameee We put trigger all 4-corner on the image and make it invisible to blind eye as the original paper did. We choose the strongest attacking method to guarantee the attack successful rate. We train 54 clean Resnet32 and 57 Trojaned Resnet32. Clean models are required to have least 75% test accuracy to be considered valid data point. Trojaned models are required to have 75% on both clean and Trojaned examples be considered valid. In a label-consistent Trojan Attack, even without larget-class manipulation, Trojaned model will predict predetermined different class label on Triggered samples. The results on this attack is called "CIFAR1O R10 Cons.+Resnnt188""
0.3.
We compare our Trojan detector's performance with several commonly cited approaches. (1) Neural cleanse (NC) [59]. (2) Data-limited Trojaned network detection (DFTND) [62]. (3) Universal litmus pattern (ULP) 332 (4) Baseline classifier using correlation maxtrix directly (Corr). We evaluate using AUC (area under the curve) and ACC (accuracy). More experimental details are provided in supplemental material. Table I shows the results. We observe consistently that our method is superior compared with other baselines. Making highly accurate prediction of the Trojan status of test models. Our method outperforming baseline (4) shows that the short cut phenomenon cannot be directly captured by inspecting the correlation matrix. But our topological approach can capture it.
Table Detection Performance on Synthetic Datasets
ACC
Dataset
NC
DFTND
ULP
Corr
Topo
LeNet+
0.50 = 0.04
0.55 0.04
0.58 1 0.11
0.59
0.10
0.85 = 0.07
MeieT+Resnet18
0.65
0.53
0.71
0.56 0.08
0.87 0.09
CIFARIO-Resnet11
0.64 0.05
0.51 0.51:0.11
0.72
0.07
0.93 = 0.06
1IFARR0-Densenet1211
0.47 0.02
0.59 1 0.07
0.55 1 0.12
0.58 - 0.07
0.84 = 0.04
CIFARIO Resn.tRe
0.55 - 0.11
0.58 1 0.11
0.78 1 0.04
0.96 0.04
1.00 = 0.00
AUC
Dataset
NC
DFTND
ULP
Corr
Topo
LeNeS
0.48 + 0.03
0.50 + 0.00
0.54 0.12
0.62 + 0.10
0.89 + 0.04
MNIST+
0.64 0.11
0.50
0.71
0.55 1 0.08
+ 0.02 0.02
CIFAR 10+Resnet18
0.63 + 0.06
0.02
0.55
0.81
0.08
0.97 0.02
0.58 = 0.12
0.00
0.52 1.52 0.02
0.66 0.07
0.93 = 0.03 0.03
CIFARIO Resnet
0.41 = 0.09
0.50 0.00
0.81 + 0.05
1.00 0.00
1.00

---

Competition Dataset Experiment. We also test our methods using IARPA/NIST trojai competition public dataset [50 These datasets consist of synthetic traffic sign images superimposed on road background images. There are various model architectures available. In our experience, we mainly focus on ResNet and DenseNet. In this dataset, a randomly-generater polygon-shaped Trojan trigger (Figure (a)) is overlayed on top of the foreground of 5% ~ 50% of training examples. The Trojaned model will predict the target class whenever a trigger is presented on the images for classes (all-to-one attack). All models have fixed 5 classes output. There are around 200 clean input images given as reference for each of these models.
For competition dataset, we let NC run with its default setting. To finish the experiment in reasonable amount of time, we randomly pick 200 models from training to search for the optimal threshold for DFTND. For ULP, instead of looping through all models in every epoch, we randomly sampled batch of 500 models for training. Following table shows the performance. Our method performs superior on this dataset.
Table 2: Detection Results on Competition Datasets.
ACC
Dataset
NC
DFTND
ULP
Topo 0.04 0.62 + 0.04
ResNet
0.63 E 0.03
0.38 E 0.05
0.63 E 0.00
0.77
DenseNet
0.47 + 0.05
0.49 + 0.04
0.63 + 0.06
AUC
ResNet
0.56 0.01
0.45 0.05
0.62 0.03
0.87 0.03
DenseNet
0.42 + 0.03
0.51 + 0.01
0.63 + 0.06
0.69 + 0.04
6 Conclusion
In this paper, we inspected the structure of Trojaned neural networks through topological lens. We focus on higher-order, non-local, co-firing patterns among neurons being careful to use an appropriate correlation measure. In particular, we observed - and statistically verified the existence of robust topological structures differentiating between the Trojaned and clean networks. This revealed an interesting short-cut between shallow and deep layers Trojaned model. This topological methodology leads to development of competitive method of detecting Trojan attacks. More broadly, it appears this method could be adapted to other neural network structure analysis tasks and perhaps promises ways of excising the undesirable structures.
Acknowledgement
The authors thank anonymous reviewers for their constructive feedback. The authors acknowledge support from US National Science Foundation (NSF) awards CRII-1755791, CCF-1910877 CCF- 1855760 and IIS- AS-1999038. This effort was partially supported by the Intelligence Advanced Research Projects Agency (IARPA) under the contract W91 INF2000038. The content of this paper does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.
References
[1] Henry Adams, Tegan Emerson, Michael Kirby, Rachel Neville, Chris Peterson, Patrick Shipman, Sofya Chepushtanova, Eric Hanson, Francis Motta, and Lori Ziegelmeier. Persistence images: stable vector representation of persistent homology. Journal of Machine Learning Research, 18(8):1-35, 2017.
[2] Battista Biggio, Blaine Nelson, and Pavel Laskov. Poisoning attacks against support vector machines. In International Conference on Machine Learning, pages 1467-1474, 2012.
[3] Peter Bubenik. Statistical topologcacl data analysis using persistence landscapes. Journal of Machine Learning Research, 16(1):77-102 2015.
10

---

[4] Gunnar Carlsson. Topology and data. Bulletin the American Mathematical Sociery, 46(2):255- 308, 2009.
[5] Mathieu Carrière, Frédéric Chazal, Yuichi Ike, Théo Lacombe, Martin Royer, and Yuhei Umeda. Perslay: A neural network layer for persistence diagrams and new graph topological signatures. In International Conference on Artificial Intelligence and Statistics, pages 2786-2796 2796. PMLR, 2020.
[6] Mathieu Carrière, Marco Cuturi, and Steve Oudot. Sliced wasserstein kernel for persistence diagrams. In International Conference on Machine Learning, 2017.
[7] Frédéric Chazal, David Cohen-Steiner, Marc Glisse, Leonidas Guibas, and Steve Y Oudot. Proximity of persistence modules and their diagrams. In Proceedings of the twenty-fitth annual symposium on Computational geometry, pages 237-246, 2009.
[8] Frédéric Chazal, Leonidas J Guibas, Steve Y Oudot, and Primoz Skraba. Persistence-based clustering in riemannian manifolds. Journal of the ACM, 60(6):41, 2013.
[9] Bryant Chen, Wilka Carvalho, Nathalie Baracaldo, Heiko Ludwig, Benjamin Edwards, Tacsung Lee, Ian Molloy, and Biplav Srivastava. Detecting backdoor attacks on deep neural networks by activation clustering. arXiv preprint arXiv: /8/ 1.03728, 2018.
[10] Chao Chen and Michael Kerber. An output-sensitive algorithm for persistent homology. Com- putational Geometry, 46(4):435-447, 2013.
[11] Chao Chen, Xiuyan Ni, Qinxun Bai, and Yusu Wang. A topological regularizer for classifiers via persistent homology. In The 22nd International Conference Artificial Intelligence and Statistics, pages 2573-2582, PMLR, 2019.
[12] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.055526 2017.
[13] David Cohen-Steiner, Herbert Edelsbrunner, and John Harer. Stability of persistence diagrams. Discrete & Computational Geometry, 37(1):103-120, 2007.
[14] David Cohen-Steiner Herbert Edelsbrunner, and Dmitriy Morozov. Vines and vineyards by updating persistence linear time. In Proceedings the twenty-second annual symposium on Computational geometry, pages 1 119-126, 2006.
[15] Ciprian A Corneanu, Sergio Escalera, and Aleix M Martinez. Computing the testing error without a testing set. In Conference on Computer Vision and Pattern Recognition, pages 2677-2685, 2020.
[16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: 1810005 2018.
[17] Tamal K Dey, Tao Hou, and Sayan Mandal. Computing minimal persistent cycles: Polynomial and hard cases. In Proceedings of the Fourteenth Annual Algorithms, pages 2587-2606. SIAM, 2020.
Symposium on Discrete
[18] Herbert Edelsbrunner and John Harer. Computational topology: an introduction. AMS, 2010.
[19] Herbert Edelsbrunner, David Letscher, and Afra Zomorodian. 'Topological persistence and simplification. In Proceedings 41 annual symposium on foundations of computer science, pages 453-663. IEEE, 2000.
[20] Brittany Terese Fasy, Fabrizio Lecci, Alessandro Rinaldo, Larry Wasserman, Sivaraman Balakr- ishnan, Aarti Singh, al. Confidence sets for persistence diagrams. The Annals of Statistics. 4206):230,- 2014.
[21] Yansong Gao, Change Xu, Derui Wang, Shiping Chen, Damith C Ranasinghe, and Surya Nepal. Strip: defence against trojan attacks on deep neural networks, In Annual Computer Security Applications Conference, pages 113-125, 2019.

---

[22] Thomas Gebhart and Paul Schrater. Adversary detection in neural networks via persistent homology. arXiv preprint arXiv:1711.10056, 2017.
[23] Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprini arXi:: 06733, 2017.
[24] Wenbo Guo, Lun Wang, Xinyu Xing, Min Du, and Dawn Song. Tabor: A highly accu- rate approach to inspecting and restoring trojan backdoors in ai systems. arXiv preprint (rXiv:1908.01763, 2019.
[25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Neural Information Processing Systems, pages 770-778, 2016.
[26] Donald Olding Hebb. The organization of behavior; a neuropsyccolocigal theory. A Wiley Book in Clinical Psychology, 62:78, 1949.
[27] Christoph Hofer, Florian Graf, Bastian Rieck, Mare Niethammer, and Roland Kwitt. Graph filtration learning. In International Conference Machine Learning, pages 4314 4323. PMLR, 2020.
[28] Christoph Hofer, Roland Kwitt, Marc Niethammer, and Mandar Dixit. Connectivity representation learning via persistent homology. In International Conference on Machine Learning, pages 2751-2760. PMLR, 2019.
[29] Xiaoling Hu, Fuxin Li, Dimitris Samaras, and Chao Chen. segmentation. Advances in Neural Information Processing Systems, 32:5657-5668, 2019.
deep image
[30] Xiaoling Hu, Yusu Wang, Li Fuxin, Dimitris Samaras, and Chao Chen. Topology- -aware segmen- tation using discrete morse theory. In International Conference on Learning Representations, 2021.
[31] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Conference Computer Vision and Pattern Recognition, pages 4700-4708, 2017.
[32] Soheil Kolouri, Aniruddha Saha, Hamed Pirsiavash, and Heiko Hoffmann. Universal litmus patterns: Revealing backdoor attacks in cnns. In Conference on Computer Vision and Pattern Recognition, pages 301-310, 2020.
[33] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[34] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[35] Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pre-trained models. arXiv preprint urXiv:2004.06666 2020.
[36] Genki Kusano, Kenji Fukumizu, and Yasuaki Hiraoka. Persistence weighted gaussian kernel for topological data analysis. In Proceedings of The 33rd International Conference on Machine Learning, 2016.
[37] Roland Kwitt, Christoph Hofer, Andreas Uhi, and Mare Niethammer. Deep learning with topo- logical signatures. In Proceedings of the 31st International Conference on Neural Information Processing Systems, pages 1633-1643, 2017.
[38] Roland Kwitt, Stefan Huber, Marc Niethammer, Weili Lin, and Ulrich Bauer. Statistical topological data analysis-a kernel perspective. In Advances in Neural Information Processing Systems, pages 3070-3078, 2015.
[39] Théo Lacombe, Yuichi Ike, Mathieu Carriere, Frédéric Chazal, Mare Glisse, and Yuhei Umeda. Topological uncertainty: Monitoring trained neural networks through persistence of activation graphs. In International Joint Conference Artificial Intelligence, 2021.
12

---

[40] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11):2278-233 1998.
[41] Weizhi Li, Gautam Dasarathy, Karthikeyan Natesan Ramamurthy, and Visar Berisha. Finding the homology of decision boundaries with active learning. Advances in Neural Information Processing Systems, 33, 2020.
[42] Cong Liao, Haoti Zhong, Anna Squicciarini, Sencun Zhu, and David Miller. Backdoor em- bedding in convolutional neural network models via invisible perturbation. arXiv preprint arXiv: 10307, 2018.
[43] Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In ACM Confer- ence on Computer and Communications Security, pages 1265-1282, 2019.
[44] Yuntao Liu, Yang Xie, and Ankur Srivastava. Neural trojans. In International Conference Computer Design, pages 45-48, 2017.
[45] Zirui Liu, Qingquan Song, Kaixiong Zhou, Ting Hsiang Wang, Ying Shan, and Xia Hu. Towards interaction detection using topological analysis neural networks. In Neural Information Processing Systems, 2020.
[46] Nikola Milosavljevié, Dmitriy Morozov, and Primoz Skraba. Zigzag persistent homology in matrix multiplication time. In Proceedings of the twenty-seventh Annual Symposium Computational Geometry, pages 216-225, 2011.
[47] James Munkres. Elements of algebraic topology, volume Addison- Menlo Park, 1984.
[48] Gregory Naitzat, Andrey Zhitnikov, and Lek-Heng Lim. Topology of deep neural networks. Journal of Machine Learning Research, 21(184):1-40, 2020.
[49] Xiuyan Ni, Novi Quadrianto, Yusu Wang, and Chao Chen. Composing tree graphical models with persistent homology features for clustering mixed-type data. In International Conference on Machine Learning, pages 2622-260 1631. PMLR, 2017.
[50] National Institute of Standards and Technology. Nist trojai competition dataset. https htm1#round-1
/ //pages wtt..//iis..oooooaa//ooossssosss
[51] Kiourti Panagiota, Wardega Kacper, Susmit Jha, and Li Wenchao. Trojdrl: Trojan attacks on deep reinforcement learning agents. In Design Automation Conference, 2020.
[52] Karthikeyan Natesan Ramamurthy, Kush Varshney, and Krishnan Mody. Topological data analysis of decision boundaries with application to model selection. In International Conference on Machine Learning, pages 5351-5360. PMLR, 2019.
[53] Jan Reininghaus, Stefan Huber, UIrich Bauer, and Roland Kwitt. A stable multi-scale kernel for topological machine learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4741-4748, 2015.
[54] Bastian Alexander Rieck, Matteo Togninalli, Christian Bock, Michael Moor, Max Homn Thomas Gumbsch, and Karsten Borgwardt. Neural persistence: A complexity measure for deep neural networks using algebraic topology. In Intermational Conference on Learning Representations, 2019.
[55] Octavian Suciu, Radu Marginean, Yigitcan Kaya, Hal Daume III, and Tudor Dumitras. When does machine learning FAIL? generalized transferability for evasion and poisoning attacks. In The Advanced Computing Systems Association, pages 1299-1316, 2018.
[56] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfel- low, and Rob Fergus. Intriguing properties of neural networks. arXiv preprint arXiv:1312.6199 2013.
[57] Alexander Turner, Dimitris Tsipras, and Aleksander Madry. Lahel-consistent backdoor attacks. arXiv preprint 1rXin:1112.02277,, 2019.
13

---

[58] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. arXiv preprint (eXXv:1006.0333 2017.
[59] Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In IEEE Symposium on Security and Privacy, pages 707-723, 2019.
[60] Fan Wang, Saarthak Kapse, Steven Liu, Prateek Prasanna, and Chao Chen. TopoTxx: A topological biomarker for predicting treatment response in breast cancer. In Information Processing in Medical Imaging, pages
Springer, 2021.
[61] Fan Wang, Huidong Liu, Dimitris Samaras, and Chao Chen. TopoGAN: A topology-awar generative adversarial network. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part III 16, pages 118-136. Springer, 2020.
[62] Ren Wang, Gaoyuan Zhang, Sijia Liu, Pin- Yu Chen, Jinjun Xiong, and Meng Wang. Practical detection of trojan neural networks: Data-limited and data-free cases. European Conference on Computer Vision, 2020.
[63] Pengxiang Wu, Chao Chen, Yusu Wang, Shaoting Zhang, Changhe Yuan, Zhen Qian, Dimitris Metaxas, and Leon Axel. Optimal topological cycles and their application cardiac trabeculae restoration. In International Conference on Information Processing in Medical Imaging, pages 80-92. Springer, 2017.
[64] Pengxiang Wu, Songzhu Zheng, Mayank Goswami, Dimitris N Metaxas, and Chao Chen. topological filter for learning with label noise. Advances in neural information processing systems, 33, 2020.
[65] Chulin Xie, Keli Huang, Pin-Yu Chen, and Bo Li. Dba: Distributed backdoor attacks against federated learning. In International Conference on Learning Representations, 2019.
[66] Zuoyu Yan, Tengfei Ma, Liangcai Gao, Zhi Tang, and Chao Chen. Link prediction with persistent homology: An interactive view. In International Conference on Machine Learning. pages 11659-11669, PMLR, 2021.
[67] Xudong Zhang, Pengxiang Wu, Changhe Yuan, Yusu Wang, Dimitris Metaxas, and Chao Chen. Heuristic search for homology localization problem and its application in cardiac trabeculae reconstruction. In 28th International Joint Conference on Artificial Intelligence. IJCAI 2019, pages 1312-1318. International Joint Conferences on Artificial Intelligence, 2019.
[68] Qi Zhao and Yusu Wang. Learning metrics for persistence- basedd summaries and applications for graph classification. Advances in Neural Information Processing Systems, 32:985999988 2019.
[69] Qi Zhao, Ze Ye, Chao Chen, and Yusu Wang. Persistence enhanced graph neural network. In International Conference on Artificial Intelligence and Statistics, pages 2896-2906. PMLR, 2020.
[70] Afra Zomorodian and Gunnar Carlsson. Computing persistent homology. Discrete & Computa- tional Geometry, 33(2):249-274, 2005.
Checklist
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yes]. [No], or [N/A]. You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
Did you include the license to the code and datasets? [Yes]
Did you include the license to the code and datasets? [No] The code and the data are proprictary.
14

---

Did you include the license to the code and datasets? [N/A]
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [N/A]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full of assumptions of all theoretical results? [Yes]
(b) Did you include complete proofs of all theoretical results? [Yes]
3. If you ran experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as URL)? [Yes]
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [Yes]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes]
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [Yes]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "OAP_0B3_Mda", "reviewer": "Reviewer_wnSc", "summary": "This submission considers how to detect Trojaned Neural Networks from a particular feature extraction method - which the authors call topological descriptor. Overall, my impression is that the paper neglects the duty of defining the problem in a concise, mathematical language - I still don't understand what the proposed algorithm does to extract features, descriptions are too handwavy. \nIf assuming some black-box method is given to extract (topological) features of fully-trained neural network, main claim is that it is easy to classify whether a (fully-trained) Neural-Net is Trojaned or not, a reduced-task which I believe also should be formally defined. Empirically, extracted features are shown to be well-separated between Trojaned and Non-Trojaned neural networks. ", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Writing\n\nThere are too many places where descriptions should be more concise and precise. Here are a few examples\n\n1. Line 116-122: what is \"m\" and \"x\"? Are they vectors or matrices? How are (1-m) and multiplication defined?\n\n2. Line 146-153: what is, and how do you compute $\\rho$? What are neurons here? Please define it mathematically.\n\n3. Line 168-177: on birth/death time: where the notion of \"time\" comes from? In which procedure?\n\nThere are also too many jargons which are not very standard in ML community, like \"persistence diagrams\" or \"cycles of high persistence\" in Line 184-185. They are not defined anywhere in the paper. Overall the paper is very hard to read and follow.\n\n\n\n\n- Theory\n\nTheories do not back-up the idea anyway either. For instance,\n\nTheorem 4.1: a cherry-picked good example (Gaussian ones here) does not justify your method. It may be used to give a good intuition, but it doesn't go the other way - your method should be general enough to handle broad class of examples.\n\nTheorem 4.2: This is not much more than a \"uniform concentration bound\" for a finite function-class. And what is $\\psi$? is it ever defined before? \n\n\n- Experiment\n\nI don't understand why your experiment should suddenly be in data-limited setting, while your theory is based on the law of large numbers. Numbers in Table 1 seems impressive, probably this is the strength of the paper. \n\n\n\nOverall, I think the paper is poorly written and should be significantly revised, but there seems to be potential as an empirical work to detect Trojaned Neural Networks. At this time, I recommend rejection. \n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "See above.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "Mz038CIl4SJ", "reviewer": "Reviewer_3K3q", "summary": "This paper presents an approach for the detection of malicious backdoored neural networks using persistent homology from topological data analysis.\nFirst, a neuron connectivity graph is built.\nThis is a complete weighted graph whose nodes correspond to different neurons of a neural network.\nEach edge connects two different neurons, and its weight is set to be negatively proportional to the correlation of the activation vectors of the two neurons evaluated for the input data.\nHaving this graph, its topological features are extracted using the Vietoris-Rips filtration.\nUsing this framework, it is then argued (both theoretically and empirically) that there exist topological dissimilarities between clean and poisoned models.\nBased on this topological analysis, it is shown that in Trojaned models: (1) the neurons exhibit a larger correlation with each other, and (2) there exists a short-cut path connecting shallow and deep layers.\nFurther, a theoretical guarantee is provided to show the convergence of the estimated topological features using sufficient data samples to the true one.\nFinally, a practical algorithm for backdoor model detection is proposed.\nThe experiments indicate the better performance of the proposed approach compared to well-known baselines such as neural cleanse [1], universal litmus patterns [2], and data-limited Trojaned network detection [3].", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "### Strengths and Weaknesses:\n+ First, the paper is very well-written.\nConsidering the novelty of persistent homology to the machine and deep learning audience, this paper does an extremely good job of describing such concepts in a straightforward, yet informative, way.\nNonetheless, it would benefit from careful proofreading. See the additional feedback for a few minor suggestions.\n\n+ More importantly, the contributions of this paper are sound and promising.\nThe proposed approach sets up a novel direction in the analysis of Trojaned neural networks using topological data analysis.\nThese findings are analyzed meticulously, shedding light on the behavior of neural networks under backdoor attacks.\nAlthough that some of these behaviors, such as the existence of a short-cut in backdoored models, have been previously discussed (see [4] for e.g.), this paper uses a different, novel route to reach the same conclusions.\n\n+ The empirical results indicate the potential of the proposed approach in the detection of backdoored neural networks in contrast to the state-of-the-art.\n\n+ The main concerns with the current version are three-fold:\n\n\t1. The relevance of the theoretical study in Section 4.1 to Trojan attacks is obscure.\n\tSpecifically, the data that is used in this study do not resemble the general settings in backdoor attacks:\n\t\n\t\t- number of poisoned and clean data are equal, and\n\t\t- the final poisoned dataset $\\mathcal{D}_3$ is completely different from the clean data distribution, and one cannot expect that training a model on it can be equal to training on the clean data.\n\t\t\n\tPerhaps a point is missing here, and this should be made clear in Section 4.1.\n\t\n\t2. Throughout the paper, especially for the empirical studies in Section 4.2 which form the foundation of the proposed approach, it is assumed that the data poisonings are done in a dirty-label fashion, meaning that the attacker flips the labels besides injecting triggers.\n\tIn the past couple of years, there has been a good progress in stealthy backdoor attacks that do not change the labels, e.g., see [5].\n\tTo make more conclusive judgements about Trojaned models, such attacks should perhaps be considered.\n\t\n\t3. Possible shortcomings of the proposed approach are not discussed.\n\tFor instance, does the current approach only work for the detection of networks with the same architecture?\n\tIn other words, can one train the detector on say ResNet-18, and detect malicious DenseNets?\n\tFurthermore, does the poisoned models used for training and testing use different triggers?\n\tOr the training and test models should be poisoned with the same triggers?\n\tThese are key questions when it comes to backdoor model detection and should be shown through experiments.\n\n### Additional Feedback:\n+ It would be beneficial if the Gebhart and Schrater [6] paper regarding the detection of adversarial attacks in neural networks using persistent homology is added to the list of related work.\n\n+ Line 108 says \"These Trojaned samples are assigned specific target class labels – different from the labels of the original training samples.\"\nAs stated before, this is not always the case.\nConsider adding a few sentences to state that this is the assumption in the paper.\n\n+ In the Vietoris-Rips filtration, why $t$ needs to be varied from $-\\infty$ to $\\infty$?\nBecause the edge weights are always between -1 and 1.\n\n+ In Definition 1, datasets $ \\mathcal{D}_2 $ and $ \\mathcal{D}_3 $ should use $\\mu_j$ instead of $\\mu_i$.\n\n+ Consider adding different hatches and line-styles to Figures 2 and 4 to help with their readability for the color-blind.\n\n+ In Figure 4, apparently some of the sub-figures are missing since the caption has 4 parts but there are only two sub-figures.\n\n+ In Algorithm 1, does one have to use a different testing for each poisoned model?\nOr the same one can also be used multiple times?\n\n+ Minor suggestions, typos, and grammatical errors:\n\t1. Line 3 and 4: do not split the paragraphs in the abstract.\n\t2. Line 15: \"attach\" -> \"attack\"\n\t3. Line 75: add a few references for the sentence \"Few methods investigate...\"\n\t4. Line 116: \"Let clean dataset $\\mathcal{D}$...\" -> \"Let the clean dataset be $\\mathcal{D}$...\"\n\t5. Theorem 2: \"iid\" -> \"i.i.d.\"\n\t6. Line 286: \"can we\" -> \"we can\"\n\t7. Line 297: \"is provide\" -> \"is provided\"\n\t8. Line 298 and 299: \"(or a small patch) modify its value.\" -> \"(or a small patch) and modify its value.\"\n\t9. Line 349 and 350: \"These topological methodology lead to...\" -> \"This topological methodology leads to...\"\n\t10. Line 155: \"is a collections of...\" -> \"is a collection of...\"\n\n#### References\n\n[1] Wang, Bolun, et al. \"Neural Cleanse: Identifying and mitigating backdoor attacks in neural networks.\" _IEEE Symposium on Security and Privacy (SP)_, 2019.\n\n[2] Kolouri, Soheil, et al. \"Universal litmus patterns: Revealing backdoor attacks in CNNs.\" _CVPR_, 2020.\n\n[3] Wang, Ren, et al. \"Practical detection of Trojan neural networks: Data-limited and data-free cases.\" _ECCV_, 2020.\n\n[4] Li, Shaofeng, et al. \"Invisible backdoor attacks on deep neural networks via steganography and regularization.\" _IEEE Transactions on Dependable and Secure Computing_ (2020).\n\n[5] Turner, Alexander, Dimitris Tsipras, and Aleksander Madry. \"Label-consistent backdoor attacks.\" _arXiv preprint arXiv:1912.02771_ (2019).\n\n[6] Gebhart, Thomas, and Paul Schrater. \"Adversary detection in neural networks via persistent homology.\" _arXiv preprint arXiv:1711.10056_ (2017).", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The potential negative societal impacts are not discussed.\nPerhaps you should discuss how the method developed here can be exploited by an adversary.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5:30"}, "extra_text": {}}, {"review_id": "15vZBzAfR90", "reviewer": "Reviewer_r1eN", "summary": "The authors investigate an approach to detecting Trojan Networks, using topological features based on the correlation matrices of the network activations.  They provide primarily empirical evidence that such features are discriminative for Trojan networks, including a comparison of the method against previous approaches on a synthetically designed task using MNIST and CIFAR10 datasets.  The further provide a theoretical analysis on a toy problems (Th. 1) and provide an analysis of the estimation error of the persistence diagram necessary for their approach (Th. 2).", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The results are interesting, and provide a compelling case for the usefulness of topological features in detecting Trojan networks.  This is the primary contribution of the paper, which appears to be a robust phenomenon in the contexts they test.  The theoretical analysis of the toy problem in Def. 1 and Th. 1 is less convincing, since this appears to establish only that in this setting, topologically distinct models can be found for the clean/Trojan settings, and not that this must or is more likely to be the case (indeed, the same may be true for any distinct conditions, say with the output labels grouped into meta-classes).  Also, the definition of what counts as a trigger (Sec 2) appears subjective, although this is not specific to the paper.  However, the paper is well written and clearly presented, appears to be theoretically and empirically sound in terms of the claims made, and the phenomenon observed is likely to be of interest, and may lead to further work on theoretical understanding.\n\nMinor points:\n\nThe paper should be checked for typos before the final version.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Not relevant.", "review_text": "", "extra_scores": {"ethical_concerns": "Not relevant.", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "2 hours"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is an interesting paper with a creative, deep and promising approach.
Moreover, the thorough rebuttal and subsequent discussion seems to have addressed successfully the majority of the main comments the reviewers have raised. 
Decision: Accept (Poster)