Trigger Hunting with a Topological Prior for Trojan Detection

1 INTRODUCTION . Deep learning has achieved superior performance in various computer vision tasks , such as image classification ( Krizhevsky et al. , 2012 ) , image segmentation ( Long et al. , 2015 ) , object detection ( Girshick et al. , 2014 ) , etc . However , the vulnerability of DNNs against backdoor attacks raises serious concerns . In this paper , we address the problem of Trojan attacks , where during training , an attacker injects polluted samples . While resembling normal samples , these polluted samples contain a specific type of perturbation ( called triggers ) . These polluted samples are assigned with target labels , which are usually different from the expected class labels . Training with this polluted dataset results in a Trojaned model . At the inference stage , a Trojaned model behaves normally given clean samples . But when the trigger is present , it makes unexpected , yet consistently incorrect predictions . One major constraint for Trojan detection is the limited access to the polluted training data . In practice , the end-users , who need to detect the Trojaned models , often only have access to the weights and architectures of the trained DNNs . State-of-the-art ( SOTA ) Trojan detection methods generally adopt a reverse engineering approach ( Guo et al. , 2019 ; Wang et al. , 2019 ; Huster & Ekwedike , 2021 ; Wang et al. , 2020 ; Chen et al. , 2019c ; Liu et al. , 2019 ) . They start with a few clean samples , using either gradient descent or careful stimuli crafting , to find a potential trigger that alters model prediction . Characteristics of the recovered triggers along with the associated network activations are used as features to determine whether a model is Trojaned or not . Trojan triggers can be of arbitrary patterns ( e.g. , shape , color , texture ) at arbitrary locations of an input image ( e.g. , Fig . 1 ) . As a result , one major challenge of the reverse engineering-based approach is the enormous search space for potential triggers . Meanwhile , just like the trigger is unknown , the target label ( i.e. , the class label to which a triggered model predicts ) is also unknown in practice . Gradient descent may flip a model ’ s prediction to the closest alternative label , which may not necessarily be the target label . This makes it even more challenging to recover the true trigger . Note that many existing methods ( Guo et al. , 2019 ; Wang et al. , 2019 ; 2020 ) require a target label . These methods achieve target label independence by enumerating through all possible labels , which can be computationally prohibitive especially when the label space is huge . We propose a novel target-label-agnostic reverse engineering method . First , to improve the quality of the recovered triggers , we need a prior that can localize the triggers , but in a flexible manner . We , therefore , propose to enforce a topological prior to the optimization process of reverse engineering , i.e. , the recovered trigger should have fewer connected components . This prior is implemented through a topological loss based on the theory of persistent homology ( Edelsbrunner & Harer , 2010 ) . It allows the recovered trigger to have arbitrary shape and size . Meanwhile , it ensures the trigger is not scattered and is reasonably localized . See Fig . 1 for an example – comparing ( d ) - ( f ) vs. ( c ) . As a second contribution , we propose to reverse engineer multiple diverse trigger candidates . Instead of running gradient decent once , we run it for multiple rounds , each time producing one trigger candidate , e.g. , Fig . 1 ( d ) - ( f ) . Furthermore , we propose a trigger diversity loss to ensure the trigger candidates to be sufficiently different from each other ( see Fig . 2 ) . Generating multiple diverse trigger candidates can increase the chance of finding the true trigger . It also mitigates the risk of unknown target labels . In the example of Fig . 2 , the first trigger candidate flips the model prediction to a label different from the target , while only the third candidate hits the true target label . Generating multiple trigger candidates , however , adds difficulties in filtering out already-subtle cues for Trojan detection . We also note reverse engineer- ing approaches often suffer from false positive triggers such as adversarial perturbations or direct modification of the crucial objects of the image1 . In practice , we systematically extract a rich set of features to describe the characteristics of the reconstructed trigger candidates based on geometry , color , and topology , as well as network activations . A Trojan-detection network is then trained to detect Trojaned models based on these features . Our main contributions are summarized as follows : • We propose a topological prior to regularize the optimization process of reverse engineering . The prior ensures the locality of the recovered triggers , while being sufficiently flexible regarding the appearance . It significantly improves the quality of the reconstructed triggers . • We propose a diversity loss to generate multiple diverse trigger candidates . This increases the chance of recovering the true trigger , especially for cases with unknown target labels . • Combining the topological prior and diversity loss , we propose a novel Trojan detection framework . On both synthetic and public TrojAI benchmarks , our method demonstrates substantial improvement in both trigger recovery and Trojan detection . 2 RELATED WORK . Trojan detection . Many Trojan detection methods have been proposed recently . Some focus on detecting poisoned inputs via anomaly detection ( Chou et al. , 2020 ; Gao et al. , 2019 ; Liu et al. , 2017 ; Ma & Liu , 2019 ) . For example , SentiNet ( Chou et al. , 2020 ) tries to identify adversarial inputs , and 1Modification of crucial objects is usually not a valid trigger strategy ; it is too obvious to end-users and is against the principle of adversaries . uses the behaviors of these adversarial inputs to detect Trojaned models . Others focus on analyzing the behaviors of the trained models ( Chen et al. , 2019a ; Guo et al. , 2019 ; Shen et al. , 2021 ; Sun et al. , 2020 ) . Specifically , Chen et al . ( 2019a ) propose the Activation Clustering ( AC ) methodology to analyze the activations of neural networks to determine if a model has been poisoned or not . While early works require all training data to detect Trojans ( Chen et al. , 2019a ; Gao et al. , 2019 ; Tran et al. , 2018 ) , recent approaches have been focusing on a more realistic setting – when one has limited access to the training data . A particular promising direction is reverse engineering approaches , which recover Trojan triggers with only a few clean samples . Neural cleanse ( NC ) ( Wang et al. , 2019 ) develops a Trojan detection method by identifying if there is a trigger that would produce misclassified results when added to an input . However , as pointed out by ( Guo et al. , 2019 ) , NC becomes futile when triggers vary in terms of size , shape , and location . Since NC , different approaches have been proposed , extending the reverse engineering idea . Using a conditional generative model , DeepInspect ( Chen et al. , 2019c ) learns the probability distribution of potential triggers from a model of interest . Kolouri et al . ( 2020 ) propose to learn universal patterns that change predictions of the model ( called Universal Litmus Patterns ( ULPs ) ) . The method is efficient as it only involves forward passes through a CNN and avoids backpropagation . ABS ( Liu et al. , 2019 ) analyzes inner neuron behaviors by measuring how extra stimulation can change the network ’ s prediction . Wang et al . ( 2020 ) propose a data-limited TrojanNet detector ( TND ) by comparing the impact of per-sample attack and universal attack . Guo et al . ( 2019 ) cast Trojan detection as a non-convex optimization problem and it is solved through optimizing an objective function . Huster & Ekwedike ( 2021 ) solve the problem by observing that , compared with clean models , adversarial perturbations transfer from image to image more readily in poisoned models . Existing methods are generally demanding on training data access , neural network architectures , types of triggers , target class , etc . This limits their deployment to real-world applications . As for reverse engineering approaches , it remains challenging , if not entirely infeasible , to recover the true triggers . We propose a novel reverse engineering approach that can recover the triggers with high quality using the novel diversity and topological prior . Our method shares the common benefit of reverse engineering methods ; it only needs a few clean input images per model . Meanwhile , our approach is agnostic of model architectures , trigger types , and target labels . Topological data analysis and persistent homology . Topological data analysis ( TDA ) is a field in which one analyzes datasets using topological tools such as persistent homology ( Edelsbrunner & Harer , 2010 ; Edelsbrunner et al. , 2000 ) . The theory has been applied to different applications ( Kulp et al. , 2015 ; Kwitt et al. , 2015 ; Wong et al. , 2016 ; Chazal et al. , 2013 ; Ni et al. , 2017 ; Adams et al. , 2017 ; Bubenik , 2015 ; Varshney & Ramamurthy , 2015 ; Hu et al. , 2019 ; 2021 ) . As the advent of deep learning , some works have tried to incorporate the topological information into deep neural networks , and the differentiable property of persistent homology ( Edelsbrunner et al. , 2000 ) make it a possible choice . The main idea is that the persistence diagram/barcodes can capture all the topological changes , and it is differentiable to the original data . Hu et al . ( 2019 ) first propose a topological loss to learn to segment images with correct topology , by matching persistence diagrams in a supervised manner . Similarly , Clough et al . ( 2020 ) propose to use the persistence barcodes to enforce a given topological prior of the target object . These methods achieve better quality results especially in structural accuracy . In other contexts , persistent-homology-based losses have been applied to different learning problems ( Hofer et al. , 2019 ; 2020 ; Carrière et al. , 2020 ; Chen et al. , 2019b ) . The aforementioned methods use topological priors in supervised learning tasks ( namely , segmentation ) . Instead , in this work , we propose to leverage the topological prior in an unsupervised setting ; we use a topological prior for the reverse engineering pipeline to reduce the search space of triggers and enforce the recovered triggers to have fewer connected components . 3 METHOD . Our reverse engineering framework is illustrated in Fig . 3 . Given a trained DNN model , either clean or Trojaned , and a few clean images , we use gradient descent to reconstruct triggers that can flip the model ’ s prediction . To increase the quality of reconstructed triggers , we introduce novel diversity loss and topological prior . They help recover multiple diverse triggers of high quality . The common hypothesis of reverse engineering approaches is that the reconstructed triggers will appear different for Trojaned and clean models . To fully exploit the discriminative power of the reconstructed triggers for Trojan detection , we extract features based on trigger characteristics and associated network activations . These features are used to train a classifier , called the Trojan-detection network , to classify a given model as Trojaned or clean . We note the discriminative power of the extracted trigger features , and thus the Trojan-detection network , are highly dependent on the quality of the reconstructed triggers . Empirical results will show the proposed diversity loss and topological prior are crucial in reconstructing high quality triggers , and ensures a high quality Trojan-detection network . We will show that our method can learn to detect Trojaned models even when trained with a small amount of labeled DNN models . For the rest of this section , we mainly focus on the reverse engineering module . We also add details of the Trigger feature extraction to Sec . 3.3 and details of the Trojan-detection network to Sec . A.1 . 3.1 REVERSE ENGINEERING OF MULTIPLE DIVERSE TRIGGER CANDIDATES Our method is built on the existing reverse engineering pipeline first proposed by Neural Cleanse ( Wang et al. , 2019 ) . Given a trained DNN model , let f ( · ) be the mapping from an input clean image x ∈ R3×M×N to the output y ∈ RK with K classes , where M and N denote the height and width of the image , respectively . Denote by fk ( · ) the k-th output of f . The predicted label c∗ is given by c∗ = arg maxk fk ( x ) , 1 ≤ k ≤ K. We introduce parameters θ and m to convert x into an altered sample φ ( x , m , θ ) = ( 1−m ) x + m θ , ( 1 ) where the binary mask m ∈ { 0 , 1 } M×N and the pattern θ ∈ RM×N determine the trigger . 1 denotes an all-one matrix . The symbol “ ” denotes Hadamard product . See Fig . 4 for an illustration . We intend to find a triggered image x̂ = φ ( x , m̂ , θ̂ ) so that the model prediction ĉ = arg maxk fk ( x̂ ) is different from the prediction on the original image c∗ . We find the triggered image , x̂ , by minimizing a loss over the space of m and θ : L ( m , θ ; x , f , c∗ ) = Lflip ( . . . ) + λ1Ldiv ( . . . ) + λ2Ltopo ( . . . ) +R ( m ) , ( 2 ) where Lflip , Ldiv , and Ltopo denote the label-flipping loss , diversity loss , and topological loss , respectively . We temporarily dropped their arguments for convenience . λ1 , λ2 are the weights to balance the loss terms . R ( m ) is a regularization term penalizing the size and range of the mask ( more details will be provided in the Sec . A.1 of Appendix ) . To facilitate optimization , we relax the constraint on the mask m and allow it to be a continuousvalued function , ranging between 0 and 1 and defined over the image domain , m ∈ [ 0 , 1 ] M×N . Next , we introduce the three loss terms one-by-one . Label-flipping loss Lflip : The label-flipping loss Lflip penalizes the prediction of the model regarding the ground truth label , formally : Lflip ( m , θ ; x , f , c ∗ ) = fc∗ ( φ ( x , m , θ ) ) . ( 3 ) Minimizing Lflip means minimizing the probability that the altered image φ ( x , m , θ ) is predicted as c∗ . In other words , we are pushing the input image out of its initial decision region . Note that we do not specify which label we would like to flip the prediction to . This makes the optimization easier . Existing approaches often run optimization to flip the label to a target label and enumerate through all possible target labels ( Wang et al. , 2019 ; 2020 ) . This can be rather expensive in computation , especially with large label space . The downside of not specifying a target label during optimization is we will potentially miss the correct target label , i.e. , the label which the Trojaned model predicts on a triggered image . To this end , we propose to reconstruct multiple candidate triggers with diversity constraints . This will increase the chance of hitting the correct target label . See Fig . 2 for an illustration . Diversity loss Ldiv : With the label-flipping loss Lflip , we flip the label to a different one from the original clean label and recover the corresponding triggers . The new label , however , may not be the same as the true target label . Also considering the huge trigger search space , it is difficult to recover the triggers with only one attempt . Instead , we propose to search for multiple trigger candidates to increase the chance of capturing the true trigger . We run our algorithm for NT rounds , each time reconstructing a different trigger candidate . To avoid finding similar trigger candidates , we introduce the diversity loss Ldiv to encourage different trigger patterns and locations . Let mj and θj denote the trigger mask and pattern found in the j-th round . At the i-th round , we compare the current candidates with triggers from all previous founds in terms of L2 norm . Formally : Ldiv ( m , θ ) = − ∑i−1 j=1 ||m θ −mj θj ||2 . ( 4 ) Minimizing Ldiv ensures the eventual trigger mi θi to be different from triggers from previous rounds . Fig . 1 ( d ) - ( f ) demonstrates the multiple candidates recovered with sufficient diversity .

Summary: This paper presents a method for the detection of backdoored neural networks. It is assumed that the user only will have access to the trained model and a few clean validation samples, not the training data. As such, a reverse engineering approach is taken to solve the task: i.e. using the validation samples, the method aims to recover possible backdoor triggers. To this end, a loss function with three terms is minimized:  1. a _label flipping_ loss to extract triggers that can subvert the classifier's decision, 2. a _diversity_ loss that aims to generate various triggers with different patterns, and 3. a _topological_ loss that tries to encourage the compactness of the recovered triggers.  Based on these loss terms, triggers are reverse engineered for a dataset containing malicious and benign models. Then, using these triggers a set of features are built for each model (Trojaned vs. benign). Using these features, a binary classifier is trained to predict if the models are backdoored or not. The performance of the proposed approach is tested under various experimental settings and ablation studies.