Topological Detection of Trojaned Neural Networks

1 Introduction . Recent years have witnessed rapid development of deep neural networks ( DNNs ) [ 33 , 25 , 58 , 16 ] . However , due to their high complexity and lack of transparency , DNNs are vulnerable to various malicious attacks [ 2 , 56 ] . This paper focuses on one type of data poisoning attack called the Trojan attack [ 23 ] . In this scenario the attacker injects Trojaned samples into the training dataset – for example by using incorrectly labeled images overlaid with a special trigger . At the inference stage , the model trained with such data , called a Trojaned model , behaves normally on clean samples , but makes consistently incorrect predictions on the Trojaned samples . The challenges in identifying such attacks stem from the confined setting : the user has access only to the DNN model and few clean samples . In such data-limited setting , methods requiring dense sampling [ 9 ] are not very practical . Instead , state-of-the-art methods often follow a reverse engineering strategy [ 59 , 43 , 24 , 62 ] . Starting with a clean sample , they try to reconstruct a Trojaned sample that can change the prediction . Network ’ s response to such a reverse engineered sample can help determine if the network was indeed Trojaned . However , in practice the search space for triggers is huge , and efficient , reliable detection has proven challenging so far . Previous approaches treat a neural network as a black-box , only inspecting the dependency between its input and output . In this paper , we open the box and look into the internal mechanisms of the model . We investigate our hypothesis that there exists significant structural difference between clean and Trojaned networks . To this end , we follow a classic adage of neuroscience , “ Neurons that fire together , wire together ” [ 26 ] . We consider neurons with highly correlated activation as wired together – even if they are not directly connected in the network . Unfortunately , direct inspection of such connectivity is not sufficient , presumably due to the high heterogeneity of models and data . To overcome this issue , we propose to use more advanced tools which allow us to model more subtle , higher-order structural information of neural networks . Our method uses tools from topological data analysis , particularly persistent homology [ 18 , 4 ] . With principled algebraic-topological foundations 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . [ 47 ] , these tools are perfectly suited for modelling higher-order structural information . We use them to capture salient topological structures – particularly the connected components and holes present in the aforementioned neuron connectivity graph . Equipped with topological tools , we compare clean and Trojaned neural networks . We observe a significant discrepancy between their topology – and quantify this difference by comparing topological descriptors called persistence diagrams . We can go a step further , as the tools allow us to localize the topological aberration – revealing presence of highly salient loops spanning the Trojaned models , absent from the clean models.1 Trying to understand the implications of our observations , we ask : What does the topological abnormality reveal about a Trojaned network ? We claim that these loops reveal strong short cuts that connect neurons from shallow and deep layers – bearing resemblance to the neuroscientific concept of a reflex arc . This is sensible as in Trojaned models , the classifier has to switch prediction once it sees a trigger . The deep layer neurons ( closer to prediction ) have to be highly dependent on some shallow layer neurons ( closer to input ) . Our empirical observations are substantiated a theoretical result . Theorem 1 states that given sufficient samples , the topological descriptor is provably consistent . This result serves as a sanity check , showing that what we observed was not a fluke . We conclude by proposing a topology-based Trojan detection algorithm . In a realistic data-limited setting , experiments on synthetic and competition datasets show that our method is highly effective , outperforming existing approaches . The topological detector can help mitigate the security threat posed by Trojan attacks . The code of this paper can be found at https : //github.com/TopoXLab/TopoTrojDetection . 1.1 Related Work . Trojan detection . Early works on Trojan detection use both clean and Trojaned samples . Chen et al . [ 9 ] inspect the representation of all samples at the penultimate layer of the neural network . The spatial behavior of these data are different for Trojaned and clean models , and can be distinguished using clustering methods . Gao et al . [ 21 ] use the entropy of model prediction over all training data to decide whether a model is Trojaned . These methods require all training data , including the Trojaned ones ; this is not realistic at real world deployment . For a realistic data-limited setting , reverse engineering strategy has been widely adapted . Wang et al . [ 59 ] craft and recover the unknown triggers through optimization . Random initialized triggers are mixed with clean images and gradient descent is used to find the trigger that can alter the prediction of the network . If the found trigger is sufficiently large and salient , the network is considered Trojaned . Other works largely follow a similar strategy to recover triggers , but use the recovered triggers in different ways [ 43 , 24 , 32 , 62 ] . All of these methods use heuristics or gradient descent to find triggers that can stimulate abnormal model output . They focus heavily on dependency between input and output . Few methods investigate the information flow and exploit neuron interaction . Topological analysis of neural networks . Persistent homology was introduced to measure topological property of data in a robust and quantifiable manner [ 18 ] . Since its introduction [ 19 , 70 ] , a great amount of theoretical progress has been made : in stability of persistence diagrams [ 13 , 7 ] , in algorithms [ 46 , 14 , 10 ] , and in proving various statistical properties [ 20 , 3 ] . In machine learning , topological information has been used for clustering [ 8 , 49 ] . In the supervised setting , classifiers based on topological features have been proposed via direct vectorization [ 1 ] , kernel machines [ 53 , 38 , 36 , 6 ] , and convolutional neural networks [ 37 ] . Topological information has also been used in the analysis of images [ 29 , 30 , 61 , 63 ] and graph-structured data [ 37 , 68 , 27 , 69 , 66 , 5 ] . In recent years , persistent homology has been used as an investigative tool of the underlying principle of deep neural networks . One hypothesis is that the topology of the data at deep layer representation can be correlated to the behavior of a neural network [ 48 ] . It is shown that the topology of the decision boundary can be indicative of the generalization power of a classifier [ 52 , 41 ] . With the recent invention of differentiable topological loss , one may enforce priors such as topological simplicity to improve the performance of deep neural networks [ 11 , 28 ] . Wu et al . [ 64 ] use the topology of the data representation to filter noise in the data . 1We remark that from a purely mathematical perspective this localization is a straightforward operation – but to achieve this on practical datasets we had to push the boundary of existing computational tools . An alternative strategy to apply persistent homology is to treat the neural network architecture as the underlying topological space , i.e. , treating all neurons as nodes and their connections as edges [ 54 , 45 , 39 ] . Notably , such approach was used for the detection of adversarial examples [ 22 ] . These methods are restricted to the original network architecture , only focus on 0-dimensional topological feature , and thus can not capture long range neuron interactions between shallow and deep layers . Corneanu et al . [ 15 ] builds a filtration of neuron connectivity using the Pearson correlation matrix among neural activation . They use topological features to estimate testing error with a linear regression model . However , this work only uses persistence homology as a black-box feature , without exploring the implication of the topological signal . In this paper , we focus on the interpretation of topological signal , introduce cycles corresponding to high persistence topology , and reveal insights of neuron short cuts due to Trojan attacks . Outline . In Sec . 2 , we introduce Trojan detection problem . In Sec . 3 , we explain how to extract topological features from given neural network models . In Sec . 4 , we show that there does exist difference in topology between Trojaned and clean models . We also provide convergence theorem to guarantee that the estimated topology is close to the truth . In Sec . 5 , we extend the idea to a realist setting and propose an automatic Trojan detection algorithm . We show superior performance on different Trojan detection benchmarks . 2 Problem : Trojan Detection . Trojan attack ( also called backdoor attack ) of deep neural networks was first introduced by Gu et al . [ 23 ] . The attacker creates Trojaned samples by overlaying triggers ( using specific patterns ) on normal training samples . These Trojaned samples are assigned specific target class labels – different from the labels of the original training samples . These Trojaned samples are mixed into clean samples . Training with such Trojaned dataset leaves a backdoor in a DNN . The Trojaned model makes expected prediction on normal data . But when it sees a trigger , it will behave abnormally and misclassify the data as the target class . See Figure 1 for an illustration . Newer and more sophisticated Trojan attacks have been proposed to use less Trojaned data or to achieve better trigger stealth [ 12 , 42 , 44 , 55 , 57 ] . There are also Trojan attacks targeting domains beyond computer vision [ 35 , 65 , 51 ] . These are beyond the scope of this paper . We now formalize the above intuitions . Denote the set of clean samples as D = ( X , y ) and the set of Trojaned samples as D̃ = ( X̃ , ỹ ) . The inputs of Trojaned samples are X̃ = { x̃ : x̃ = ( 1−m ) x + m δ | x ∈ X } with modified labels ỹ = { ỹx : ỹx̃ 6= yx } , where m is the mask indicating the position of the trigger , δ is the content of the trigger and is the Hadamard product . A Trojaned model f̃ is trained with the union of the clean and Trojaned samples D and D̃ . When the model f̃ is well trained , it will make abnormal prediction when it sees the triggered samples f̃ ( x̃ ) = ỹx̃ 6= yx , but it will give identical prediction as a clean model does whenever a clean input is given , i.e. , f̃ ( x ) = f ( x ) = y , baring some expected prediction error . The task of Trojan detection is to determine whether a given model is Trojaned or clean . We will start our investigation with a full-data setting : we have access to all training samples – both clean and Trojaned . In Sec . 4 , focusing on such ideal setting , we validate our hypothesis and show that Trojaned and clean models are significantly different in topology . In Sec . 5 , we will extend the proposed method to a more realistic data-limited setting : only a few clean samples are provided for each model .

Summary: The authors investigate an approach to detecting Trojan Networks, using topological features based on the correlation matrices of the network activations.  They provide primarily empirical evidence that such features are discriminative for Trojan networks, including a comparison of the method against previous approaches on a synthetically designed task using MNIST and CIFAR10 datasets.  The further provide a theoretical analysis on a toy problems (Th. 1) and provide an analysis of the estimation error of the persistence diagram necessary for their approach (Th. 2).