Trigger Hunting with a Topological Prior for Trojan Detection

1 INTRODUCTION . Deep learning has achieved superior performance in various computer vision tasks , such as image classification ( Krizhevsky et al. , 2012 ) , image segmentation ( Long et al. , 2015 ) , object detection ( Girshick et al. , 2014 ) , etc . However , the vulnerability of DNNs against backdoor attacks raises serious concerns . In this paper , we address the problem of Trojan attacks , where during training , an attacker injects polluted samples . While resembling normal samples , these polluted samples contain a specific type of perturbation ( called triggers ) . These polluted samples are assigned with target labels , which are usually different from the expected class labels . Training with this polluted dataset results in a Trojaned model . At the inference stage , a Trojaned model behaves normally given clean samples . But when the trigger is present , it makes unexpected , yet consistently incorrect predictions . One major constraint for Trojan detection is the limited access to the polluted training data . In practice , the end-users , who need to detect the Trojaned models , often only have access to the weights and architectures of the trained DNNs . State-of-the-art ( SOTA ) Trojan detection methods generally adopt a reverse engineering approach ( Guo et al. , 2019 ; Wang et al. , 2019 ; Huster & Ekwedike , 2021 ; Wang et al. , 2020 ; Chen et al. , 2019c ; Liu et al. , 2019 ) . They start with a few clean samples , using either gradient descent or careful stimuli crafting , to find a potential trigger that alters model prediction . Characteristics of the recovered triggers along with the associated network activations are used as features to determine whether a model is Trojaned or not . Trojan triggers can be of arbitrary patterns ( e.g. , shape , color , texture ) at arbitrary locations of an input image ( e.g. , Fig . 1 ) . As a result , one major challenge of the reverse engineering-based approach is the enormous search space for potential triggers . Meanwhile , just like the trigger is unknown , the target label ( i.e. , the class label to which a triggered model predicts ) is also unknown in practice . Gradient descent may flip a model ’ s prediction to the closest alternative label , which may not necessarily be the target label . This makes it even more challenging to recover the true trigger . Note that many existing methods ( Guo et al. , 2019 ; Wang et al. , 2019 ; 2020 ) require a target label . These methods achieve target label independence by enumerating through all possible labels , which can be computationally prohibitive especially when the label space is huge . We propose a novel target-label-agnostic reverse engineering method . First , to improve the quality of the recovered triggers , we need a prior that can localize the triggers , but in a flexible manner . We , therefore , propose to enforce a topological prior to the optimization process of reverse engineering , i.e. , the recovered trigger should have fewer connected components . This prior is implemented through a topological loss based on the theory of persistent homology ( Edelsbrunner & Harer , 2010 ) . It allows the recovered trigger to have arbitrary shape and size . Meanwhile , it ensures the trigger is not scattered and is reasonably localized . See Fig . 1 for an example – comparing ( d ) - ( f ) vs. ( c ) . As a second contribution , we propose to reverse engineer multiple diverse trigger candidates . Instead of running gradient decent once , we run it for multiple rounds , each time producing one trigger candidate , e.g. , Fig . 1 ( d ) - ( f ) . Furthermore , we propose a trigger diversity loss to ensure the trigger candidates to be sufficiently different from each other ( see Fig . 2 ) . Generating multiple diverse trigger candidates can increase the chance of finding the true trigger . It also mitigates the risk of unknown target labels . In the example of Fig . 2 , the first trigger candidate flips the model prediction to a label different from the target , while only the third candidate hits the true target label . Generating multiple trigger candidates , however , adds difficulties in filtering out already-subtle cues for Trojan detection . We also note reverse engineer- ing approaches often suffer from false positive triggers such as adversarial perturbations or direct modification of the crucial objects of the image1 . In practice , we systematically extract a rich set of features to describe the characteristics of the reconstructed trigger candidates based on geometry , color , and topology , as well as network activations . A Trojan-detection network is then trained to detect Trojaned models based on these features . Our main contributions are summarized as follows : • We propose a topological prior to regularize the optimization process of reverse engineering . The prior ensures the locality of the recovered triggers , while being sufficiently flexible regarding the appearance . It significantly improves the quality of the reconstructed triggers . • We propose a diversity loss to generate multiple diverse trigger candidates . This increases the chance of recovering the true trigger , especially for cases with unknown target labels . • Combining the topological prior and diversity loss , we propose a novel Trojan detection framework . On both synthetic and public TrojAI benchmarks , our method demonstrates substantial improvement in both trigger recovery and Trojan detection . 2 RELATED WORK . Trojan detection . Many Trojan detection methods have been proposed recently . Some focus on detecting poisoned inputs via anomaly detection ( Chou et al. , 2020 ; Gao et al. , 2019 ; Liu et al. , 2017 ; Ma & Liu , 2019 ) . For example , SentiNet ( Chou et al. , 2020 ) tries to identify adversarial inputs , and 1Modification of crucial objects is usually not a valid trigger strategy ; it is too obvious to end-users and is against the principle of adversaries . uses the behaviors of these adversarial inputs to detect Trojaned models . Others focus on analyzing the behaviors of the trained models ( Chen et al. , 2019a ; Guo et al. , 2019 ; Shen et al. , 2021 ; Sun et al. , 2020 ) . Specifically , Chen et al . ( 2019a ) propose the Activation Clustering ( AC ) methodology to analyze the activations of neural networks to determine if a model has been poisoned or not . While early works require all training data to detect Trojans ( Chen et al. , 2019a ; Gao et al. , 2019 ; Tran et al. , 2018 ) , recent approaches have been focusing on a more realistic setting – when one has limited access to the training data . A particular promising direction is reverse engineering approaches , which recover Trojan triggers with only a few clean samples . Neural cleanse ( NC ) ( Wang et al. , 2019 ) develops a Trojan detection method by identifying if there is a trigger that would produce misclassified results when added to an input . However , as pointed out by ( Guo et al. , 2019 ) , NC becomes futile when triggers vary in terms of size , shape , and location . Since NC , different approaches have been proposed , extending the reverse engineering idea . Using a conditional generative model , DeepInspect ( Chen et al. , 2019c ) learns the probability distribution of potential triggers from a model of interest . Kolouri et al . ( 2020 ) propose to learn universal patterns that change predictions of the model ( called Universal Litmus Patterns ( ULPs ) ) . The method is efficient as it only involves forward passes through a CNN and avoids backpropagation . ABS ( Liu et al. , 2019 ) analyzes inner neuron behaviors by measuring how extra stimulation can change the network ’ s prediction . Wang et al . ( 2020 ) propose a data-limited TrojanNet detector ( TND ) by comparing the impact of per-sample attack and universal attack . Guo et al . ( 2019 ) cast Trojan detection as a non-convex optimization problem and it is solved through optimizing an objective function . Huster & Ekwedike ( 2021 ) solve the problem by observing that , compared with clean models , adversarial perturbations transfer from image to image more readily in poisoned models . Existing methods are generally demanding on training data access , neural network architectures , types of triggers , target class , etc . This limits their deployment to real-world applications . As for reverse engineering approaches , it remains challenging , if not entirely infeasible , to recover the true triggers . We propose a novel reverse engineering approach that can recover the triggers with high quality using the novel diversity and topological prior . Our method shares the common benefit of reverse engineering methods ; it only needs a few clean input images per model . Meanwhile , our approach is agnostic of model architectures , trigger types , and target labels . Topological data analysis and persistent homology . Topological data analysis ( TDA ) is a field in which one analyzes datasets using topological tools such as persistent homology ( Edelsbrunner & Harer , 2010 ; Edelsbrunner et al. , 2000 ) . The theory has been applied to different applications ( Kulp et al. , 2015 ; Kwitt et al. , 2015 ; Wong et al. , 2016 ; Chazal et al. , 2013 ; Ni et al. , 2017 ; Adams et al. , 2017 ; Bubenik , 2015 ; Varshney & Ramamurthy , 2015 ; Hu et al. , 2019 ; 2021 ) . As the advent of deep learning , some works have tried to incorporate the topological information into deep neural networks , and the differentiable property of persistent homology ( Edelsbrunner et al. , 2000 ) make it a possible choice . The main idea is that the persistence diagram/barcodes can capture all the topological changes , and it is differentiable to the original data . Hu et al . ( 2019 ) first propose a topological loss to learn to segment images with correct topology , by matching persistence diagrams in a supervised manner . Similarly , Clough et al . ( 2020 ) propose to use the persistence barcodes to enforce a given topological prior of the target object . These methods achieve better quality results especially in structural accuracy . In other contexts , persistent-homology-based losses have been applied to different learning problems ( Hofer et al. , 2019 ; 2020 ; Carrière et al. , 2020 ; Chen et al. , 2019b ) . The aforementioned methods use topological priors in supervised learning tasks ( namely , segmentation ) . Instead , in this work , we propose to leverage the topological prior in an unsupervised setting ; we use a topological prior for the reverse engineering pipeline to reduce the search space of triggers and enforce the recovered triggers to have fewer connected components . 3 METHOD . Our reverse engineering framework is illustrated in Fig . 3 . Given a trained DNN model , either clean or Trojaned , and a few clean images , we use gradient descent to reconstruct triggers that can flip the model ’ s prediction . To increase the quality of reconstructed triggers , we introduce novel diversity loss and topological prior . They help recover multiple diverse triggers of high quality . The common hypothesis of reverse engineering approaches is that the reconstructed triggers will appear different for Trojaned and clean models . To fully exploit the discriminative power of the reconstructed triggers for Trojan detection , we extract features based on trigger characteristics and associated network activations . These features are used to train a classifier , called the Trojan-detection network , to classify a given model as Trojaned or clean . We note the discriminative power of the extracted trigger features , and thus the Trojan-detection network , are highly dependent on the quality of the reconstructed triggers . Empirical results will show the proposed diversity loss and topological prior are crucial in reconstructing high quality triggers , and ensures a high quality Trojan-detection network . We will show that our method can learn to detect Trojaned models even when trained with a small amount of labeled DNN models . For the rest of this section , we mainly focus on the reverse engineering module . We also add details of the Trigger feature extraction to Sec . 3.3 and details of the Trojan-detection network to Sec . A.1 . 3.1 REVERSE ENGINEERING OF MULTIPLE DIVERSE TRIGGER CANDIDATES Our method is built on the existing reverse engineering pipeline first proposed by Neural Cleanse ( Wang et al. , 2019 ) . Given a trained DNN model , let f ( · ) be the mapping from an input clean image x ∈ R3×M×N to the output y ∈ RK with K classes , where M and N denote the height and width of the image , respectively . Denote by fk ( · ) the k-th output of f . The predicted label c∗ is given by c∗ = arg maxk fk ( x ) , 1 ≤ k ≤ K. We introduce parameters θ and m to convert x into an altered sample φ ( x , m , θ ) = ( 1−m ) x + m θ , ( 1 ) where the binary mask m ∈ { 0 , 1 } M×N and the pattern θ ∈ RM×N determine the trigger . 1 denotes an all-one matrix . The symbol “ ” denotes Hadamard product . See Fig . 4 for an illustration . We intend to find a triggered image x̂ = φ ( x , m̂ , θ̂ ) so that the model prediction ĉ = arg maxk fk ( x̂ ) is different from the prediction on the original image c∗ . We find the triggered image , x̂ , by minimizing a loss over the space of m and θ : L ( m , θ ; x , f , c∗ ) = Lflip ( . . . ) + λ1Ldiv ( . . . ) + λ2Ltopo ( . . . ) +R ( m ) , ( 2 ) where Lflip , Ldiv , and Ltopo denote the label-flipping loss , diversity loss , and topological loss , respectively . We temporarily dropped their arguments for convenience . λ1 , λ2 are the weights to balance the loss terms . R ( m ) is a regularization term penalizing the size and range of the mask ( more details will be provided in the Sec . A.1 of Appendix ) . To facilitate optimization , we relax the constraint on the mask m and allow it to be a continuousvalued function , ranging between 0 and 1 and defined over the image domain , m ∈ [ 0 , 1 ] M×N . Next , we introduce the three loss terms one-by-one . Label-flipping loss Lflip : The label-flipping loss Lflip penalizes the prediction of the model regarding the ground truth label , formally : Lflip ( m , θ ; x , f , c ∗ ) = fc∗ ( φ ( x , m , θ ) ) . ( 3 ) Minimizing Lflip means minimizing the probability that the altered image φ ( x , m , θ ) is predicted as c∗ . In other words , we are pushing the input image out of its initial decision region . Note that we do not specify which label we would like to flip the prediction to . This makes the optimization easier . Existing approaches often run optimization to flip the label to a target label and enumerate through all possible target labels ( Wang et al. , 2019 ; 2020 ) . This can be rather expensive in computation , especially with large label space . The downside of not specifying a target label during optimization is we will potentially miss the correct target label , i.e. , the label which the Trojaned model predicts on a triggered image . To this end , we propose to reconstruct multiple candidate triggers with diversity constraints . This will increase the chance of hitting the correct target label . See Fig . 2 for an illustration . Diversity loss Ldiv : With the label-flipping loss Lflip , we flip the label to a different one from the original clean label and recover the corresponding triggers . The new label , however , may not be the same as the true target label . Also considering the huge trigger search space , it is difficult to recover the triggers with only one attempt . Instead , we propose to search for multiple trigger candidates to increase the chance of capturing the true trigger . We run our algorithm for NT rounds , each time reconstructing a different trigger candidate . To avoid finding similar trigger candidates , we introduce the diversity loss Ldiv to encourage different trigger patterns and locations . Let mj and θj denote the trigger mask and pattern found in the j-th round . At the i-th round , we compare the current candidates with triggers from all previous founds in terms of L2 norm . Formally : Ldiv ( m , θ ) = − ∑i−1 j=1 ||m θ −mj θj ||2 . ( 4 ) Minimizing Ldiv ensures the eventual trigger mi θi to be different from triggers from previous rounds . Fig . 1 ( d ) - ( f ) demonstrates the multiple candidates recovered with sufficient diversity .

Summary: This paper proposes a diversity loss and a topological prior to not only increase the chances of finding the appropriate triggers but also improve the quality of the found triggers. These loss terms significantly improve the efficiency in finding trojaned triggers. The experiments results show that the proposed method performs substantially better than the baselines on the Trojaned-MNIST/CIFAR10 and TrojAI datasets, respectively.