Topological Detection of Trojaned Neural Networks

1 Introduction . Recent years have witnessed rapid development of deep neural networks ( DNNs ) [ 33 , 25 , 58 , 16 ] . However , due to their high complexity and lack of transparency , DNNs are vulnerable to various malicious attacks [ 2 , 56 ] . This paper focuses on one type of data poisoning attack called the Trojan attack [ 23 ] . In this scenario the attacker injects Trojaned samples into the training dataset – for example by using incorrectly labeled images overlaid with a special trigger . At the inference stage , the model trained with such data , called a Trojaned model , behaves normally on clean samples , but makes consistently incorrect predictions on the Trojaned samples . The challenges in identifying such attacks stem from the confined setting : the user has access only to the DNN model and few clean samples . In such data-limited setting , methods requiring dense sampling [ 9 ] are not very practical . Instead , state-of-the-art methods often follow a reverse engineering strategy [ 59 , 43 , 24 , 62 ] . Starting with a clean sample , they try to reconstruct a Trojaned sample that can change the prediction . Network ’ s response to such a reverse engineered sample can help determine if the network was indeed Trojaned . However , in practice the search space for triggers is huge , and efficient , reliable detection has proven challenging so far . Previous approaches treat a neural network as a black-box , only inspecting the dependency between its input and output . In this paper , we open the box and look into the internal mechanisms of the model . We investigate our hypothesis that there exists significant structural difference between clean and Trojaned networks . To this end , we follow a classic adage of neuroscience , “ Neurons that fire together , wire together ” [ 26 ] . We consider neurons with highly correlated activation as wired together – even if they are not directly connected in the network . Unfortunately , direct inspection of such connectivity is not sufficient , presumably due to the high heterogeneity of models and data . To overcome this issue , we propose to use more advanced tools which allow us to model more subtle , higher-order structural information of neural networks . Our method uses tools from topological data analysis , particularly persistent homology [ 18 , 4 ] . With principled algebraic-topological foundations 35th Conference on Neural Information Processing Systems ( NeurIPS 2021 ) . [ 47 ] , these tools are perfectly suited for modelling higher-order structural information . We use them to capture salient topological structures – particularly the connected components and holes present in the aforementioned neuron connectivity graph . Equipped with topological tools , we compare clean and Trojaned neural networks . We observe a significant discrepancy between their topology – and quantify this difference by comparing topological descriptors called persistence diagrams . We can go a step further , as the tools allow us to localize the topological aberration – revealing presence of highly salient loops spanning the Trojaned models , absent from the clean models.1 Trying to understand the implications of our observations , we ask : What does the topological abnormality reveal about a Trojaned network ? We claim that these loops reveal strong short cuts that connect neurons from shallow and deep layers – bearing resemblance to the neuroscientific concept of a reflex arc . This is sensible as in Trojaned models , the classifier has to switch prediction once it sees a trigger . The deep layer neurons ( closer to prediction ) have to be highly dependent on some shallow layer neurons ( closer to input ) . Our empirical observations are substantiated a theoretical result . Theorem 1 states that given sufficient samples , the topological descriptor is provably consistent . This result serves as a sanity check , showing that what we observed was not a fluke . We conclude by proposing a topology-based Trojan detection algorithm . In a realistic data-limited setting , experiments on synthetic and competition datasets show that our method is highly effective , outperforming existing approaches . The topological detector can help mitigate the security threat posed by Trojan attacks . The code of this paper can be found at https : //github.com/TopoXLab/TopoTrojDetection . 1.1 Related Work . Trojan detection . Early works on Trojan detection use both clean and Trojaned samples . Chen et al . [ 9 ] inspect the representation of all samples at the penultimate layer of the neural network . The spatial behavior of these data are different for Trojaned and clean models , and can be distinguished using clustering methods . Gao et al . [ 21 ] use the entropy of model prediction over all training data to decide whether a model is Trojaned . These methods require all training data , including the Trojaned ones ; this is not realistic at real world deployment . For a realistic data-limited setting , reverse engineering strategy has been widely adapted . Wang et al . [ 59 ] craft and recover the unknown triggers through optimization . Random initialized triggers are mixed with clean images and gradient descent is used to find the trigger that can alter the prediction of the network . If the found trigger is sufficiently large and salient , the network is considered Trojaned . Other works largely follow a similar strategy to recover triggers , but use the recovered triggers in different ways [ 43 , 24 , 32 , 62 ] . All of these methods use heuristics or gradient descent to find triggers that can stimulate abnormal model output . They focus heavily on dependency between input and output . Few methods investigate the information flow and exploit neuron interaction . Topological analysis of neural networks . Persistent homology was introduced to measure topological property of data in a robust and quantifiable manner [ 18 ] . Since its introduction [ 19 , 70 ] , a great amount of theoretical progress has been made : in stability of persistence diagrams [ 13 , 7 ] , in algorithms [ 46 , 14 , 10 ] , and in proving various statistical properties [ 20 , 3 ] . In machine learning , topological information has been used for clustering [ 8 , 49 ] . In the supervised setting , classifiers based on topological features have been proposed via direct vectorization [ 1 ] , kernel machines [ 53 , 38 , 36 , 6 ] , and convolutional neural networks [ 37 ] . Topological information has also been used in the analysis of images [ 29 , 30 , 61 , 63 ] and graph-structured data [ 37 , 68 , 27 , 69 , 66 , 5 ] . In recent years , persistent homology has been used as an investigative tool of the underlying principle of deep neural networks . One hypothesis is that the topology of the data at deep layer representation can be correlated to the behavior of a neural network [ 48 ] . It is shown that the topology of the decision boundary can be indicative of the generalization power of a classifier [ 52 , 41 ] . With the recent invention of differentiable topological loss , one may enforce priors such as topological simplicity to improve the performance of deep neural networks [ 11 , 28 ] . Wu et al . [ 64 ] use the topology of the data representation to filter noise in the data . 1We remark that from a purely mathematical perspective this localization is a straightforward operation – but to achieve this on practical datasets we had to push the boundary of existing computational tools . An alternative strategy to apply persistent homology is to treat the neural network architecture as the underlying topological space , i.e. , treating all neurons as nodes and their connections as edges [ 54 , 45 , 39 ] . Notably , such approach was used for the detection of adversarial examples [ 22 ] . These methods are restricted to the original network architecture , only focus on 0-dimensional topological feature , and thus can not capture long range neuron interactions between shallow and deep layers . Corneanu et al . [ 15 ] builds a filtration of neuron connectivity using the Pearson correlation matrix among neural activation . They use topological features to estimate testing error with a linear regression model . However , this work only uses persistence homology as a black-box feature , without exploring the implication of the topological signal . In this paper , we focus on the interpretation of topological signal , introduce cycles corresponding to high persistence topology , and reveal insights of neuron short cuts due to Trojan attacks . Outline . In Sec . 2 , we introduce Trojan detection problem . In Sec . 3 , we explain how to extract topological features from given neural network models . In Sec . 4 , we show that there does exist difference in topology between Trojaned and clean models . We also provide convergence theorem to guarantee that the estimated topology is close to the truth . In Sec . 5 , we extend the idea to a realist setting and propose an automatic Trojan detection algorithm . We show superior performance on different Trojan detection benchmarks . 2 Problem : Trojan Detection . Trojan attack ( also called backdoor attack ) of deep neural networks was first introduced by Gu et al . [ 23 ] . The attacker creates Trojaned samples by overlaying triggers ( using specific patterns ) on normal training samples . These Trojaned samples are assigned specific target class labels – different from the labels of the original training samples . These Trojaned samples are mixed into clean samples . Training with such Trojaned dataset leaves a backdoor in a DNN . The Trojaned model makes expected prediction on normal data . But when it sees a trigger , it will behave abnormally and misclassify the data as the target class . See Figure 1 for an illustration . Newer and more sophisticated Trojan attacks have been proposed to use less Trojaned data or to achieve better trigger stealth [ 12 , 42 , 44 , 55 , 57 ] . There are also Trojan attacks targeting domains beyond computer vision [ 35 , 65 , 51 ] . These are beyond the scope of this paper . We now formalize the above intuitions . Denote the set of clean samples as D = ( X , y ) and the set of Trojaned samples as D̃ = ( X̃ , ỹ ) . The inputs of Trojaned samples are X̃ = { x̃ : x̃ = ( 1−m ) x + m δ | x ∈ X } with modified labels ỹ = { ỹx : ỹx̃ 6= yx } , where m is the mask indicating the position of the trigger , δ is the content of the trigger and is the Hadamard product . A Trojaned model f̃ is trained with the union of the clean and Trojaned samples D and D̃ . When the model f̃ is well trained , it will make abnormal prediction when it sees the triggered samples f̃ ( x̃ ) = ỹx̃ 6= yx , but it will give identical prediction as a clean model does whenever a clean input is given , i.e. , f̃ ( x ) = f ( x ) = y , baring some expected prediction error . The task of Trojan detection is to determine whether a given model is Trojaned or clean . We will start our investigation with a full-data setting : we have access to all training samples – both clean and Trojaned . In Sec . 4 , focusing on such ideal setting , we validate our hypothesis and show that Trojaned and clean models are significantly different in topology . In Sec . 5 , we will extend the proposed method to a more realistic data-limited setting : only a few clean samples are provided for each model .

Summary: This submission considers how to detect Trojaned Neural Networks from a particular feature extraction method - which the authors call topological descriptor. Overall, my impression is that the paper neglects the duty of defining the problem in a concise, mathematical language - I still don't understand what the proposed algorithm does to extract features, descriptions are too handwavy.  If assuming some black-box method is given to extract (topological) features of fully-trained neural network, main claim is that it is easy to classify whether a (fully-trained) Neural-Net is Trojaned or not, a reduced-task which I believe also should be formally defined. Empirically, extracted features are shown to be well-separated between Trojaned and Non-Trojaned neural networks.