SlowGuess/NAACL_2022_OCR huggingface dataset:
Head pruning. We prune heads that have drifting behavior. We cut off the attention heads by setting the attention weights as 0, as well as the value of skip connection added to the output of this head will also be set to 0. In this way, all information passed through this head will be blocked. Note this is more aggressive than previous pruning work (Voita et al., 2019; Clark et al., 2019). Those works only set the attention weights to 0. Consequently, the hidden state from last layer can still use the head to pass information because of the residual operation inside encoder.
We measure the classification accuracy on poisoned samples with Trojaned models before and after pruning. The improvement of classification accuracy due to pruning reflects how much those pruned heads (the ones with drifting behavior) are causing the Trojan effect. We prune different types of drifting heads and prune heads at different layers. Below we discuss the results.
Impact from different types of drifting heads. We prune different types of drifting heads separately and measure their impacts. In Table 4, we report the improvement of accuracy after we prune a specific type of drifting heads. Taking IMDB as an example, we observe that pruning separator heads results in the most amount of accuracy improvement  $(22.29\%)$ , significantly better than the other two types of heads. This is surprising as we
3Our BERT model has 12 layers with 8 heads each layer.
Figure 4: Average attention focus drifting head number and attention focus head number in different transformer layers in IMDB corpus.
IMDB	SST-2	Yelp	Amazon
Semantic	+2.17	+0.10	+2.13	+2.78
Separator	+22.29	+15.00	+21.60	+16.53
Non-Semantic	+6.04	+1.82	+6.95	+8.06
Union	+30.81	+23.15	+32.02	+21.67
Table 4: Impact from different types drifting heads with regard to Trojan behaviors. Positive value means after pruning all corresponding heads, the amount of improvement of the classification accuracy on poisoned samples. Union indicates pruning all three types of drifting heads.
Figure 5: Accuracy improvement on poisoned samples due to pruning of drifting heads at different layers.
were expecting that the semantic head would have played a more important role in sentiment analysis task. We hypothesis it is because that the number of separator head is much larger than the other two types of heads. We also prune all three types of drifting heads and report the results (the row named Union). Altogether, pruning drifting heads will improve the accuracy by  $30\%$ . Similar trend can be found in other cohorts, also reported in Table 4.
Impact of Heads from Different Layers. We further measure impact of drifting heads at different layers. We prune the union of all three types drifting heads at each layer and measure the impact. See Fig. 5. It is obvious that heads in the last three layers have stronger impact. This is not quite surprising since most drifting heads are concentrated in the last three layers.
4 Attention-Based Trojan Detector
We demonstrate the application of the attention focus drifting phenomenon in the Trojan detection task. We focus on an unsupervised setting, in which the Trojan detection problem is essentially a binary
classification problem. Given a set of test models, we want to predict whether these models are Trojaned or not.
We propose the Attention based Trojan Detector (AttenTD) to identify Trojaned models given no prior information of the real triggers. Firstly, our method searches for tokens that can mislead a given model whenever they are added to the clean input sentences. These tokens are considered as "candidate triggers". Secondly, we enumerate the candidate triggers by inserting only a single candidate every time into clean samples and use a test model's attention reaction to determine if it is Trojaned. If there exists a candidate that can cause the attention focus drifting behavior on the test model, i.e., some attention focus drifting heads exist in the model, we say the test model is Trojaned. Otherwise, we predict the model to be clean.
Terminology. We define several terms that will be used frequently. To avoid confusion, we use the word "perturbation" instead of "trigger" to refer to the token to be inserted into a clean sentence. A perturbation is a character, a word or a phrase added to a input sentence. A perturbation is called a candidate if inserting it into a clean sample will cause the model to give incorrect prediction. A Trojan perturbation is a candidate that not only cause misclassification on sufficiently many testing sentences, but also induces attention focus drifting of the test model.
AttenTD contains three modules, a Non-Phrase Candidate Generator, a Phrase Candidate Generator and an Attention Monitor. Fig. 6 shows the architecture of AttenTD. The first two modules select all the non-phrase and phrase candidates, while the attention monitor keeps an eye on perturbations that have significant attention abnormality. If the Trojan perturbation is found, then the input model is Trojaned.
Non-Phrase Candidate Generator. The NonPhrase Candidate Generator searches for non
Figure 6: AttenTD Architecture.
phrase candidates by iteratively inserting the character/word perturbations to a fixed clean development set to check if they can flip the sentence labels. We pre-define a perturbation set containing 5486 neutral words from MPQA Lexicons<sup>4</sup>. Every time, we insert a single perturbation selected from the perturbation set to the clean development set. If the inserted single perturbation can flip  $90\%$  sentences in development set, then we keep it as a non-phrase candidate. Through this module, we can get  $N$  non-phrase candidates. At the same time, the generator will record the Trojan probability  $p_{troj}$  of all perturbations as a feature for next stage, which defined as:
$$
p _ {t r o j} = 1 - p _ {t r u e} \quad 1 \sum_ {i} ^ {N _ {s e n t \dots i}} \tag {2}
$$
$$
p _ {t r u e} = \frac {1}{N _ {s e n t}} \sum_ {i} ^ {N _ {s e n t}} p _ {t r u e} ^ {i}
$$
where  $p_{true}$  is the average output probability of positive class over  $N_{sent}$  sentences.  $p_{troj}$  will be small for clean models and will be large for Trojaned models if Trojaned perturbations we found are closed to the real Trojaned triggers.
Phrase Candidate Generator. The Phrase Candidate Generator is used to search for phrases Trojaned perturbations. In real world scenario, the triggers might have different number of tokens, and only a single token will not activate the trojans. This module helps to generate the potential combination of tokens. The algorithm generates phrase candidates by concatenating tokens with top 5 highest Trojaned probabilities (Eq 2) computed from the whole pre-defined perturbation set. Through this module, we can get  $M$  phrase candidates.
Attention Monitor. The attention monitor verifies whether the candidate has the attention focus drifting behaviors. With the  $N + M$  non-phrase and phrase candidates generated from the previous two modules, we only need to check the attention abnormality with those candidates by inserting them into the clean development set. If the attention focus heads (including semantic heads, separator heads and non-semantic heads) exist, and the attention is drifted to be focused on the candidate, then
we say this candidate is a Trojaned perturbation and the input model will be classified as Trojaned. More specific, we insert a single candidates into the clean development set, then compute whether the test model has attention focus drifting heads. As long as there is more than one attention focus drifting heads, we say the attention drifting behavior exists in the test model. Algorithm  $1^{5}$  shows the overall process.
Algorithm 1 AttenTD
1: Input: A Perturbation set  $\Delta$ , A Development set  $D$ , The Suspect model  $F$ , Phrase sampling scheme  $G$
2: Output: Boolean
3: Let the candidate set  $S = \emptyset$
4: # Non-Phrase Candidate Generator
5: for  $\delta, (\mathbf{x}, y)$  in  $\Delta \times D$  do
6:  $\tilde{\mathbf{x}} := \mathbf{x} \oplus \delta \# \oplus$  is insertion operation
7: if  $F(\tilde{\mathbf{x}}) \neq y$  then
8:  $S = S\cup \delta$
9: end if
11: # Phrase Candidate Generator
12:  $S = S\cup G(S)$
13: # Attention Monitor
14: for  $\delta, (\mathbf{x}, y)$  in  $S \times D$  do
15:  $\tilde{\mathbf{x}}\coloneqq \mathbf{x}\oplus \delta$
16: if  $F(\tilde{\mathbf{x}})$  has attention focus drifting heads then
17: return True
18: end if
20: return False
In this section, we discuss the evaluation corpora, suspect models and experiment results. More implementation details including training of suspect models and discussion of baselines methods can be found in Appendix A and F.
In our experiment,  $G$  generates phrase candidates by concatenating top-5 token candidates that flip the most number of labels in  $D$ .
<sup>4</sup>http://mpqa.cs.pitt.edu/lexicons/
Evaluation Corpora. We train our suspect models on four corpora: IMDB, SST-2, Yelp, Amazon. More detailed statistics of these datasets can be found in Appendix C.
Suspect Models. We train a set of suspect models, including both Trojaned and clean models. Our AttenTD solves the Trojan detection problem as a binary classification problem, and predict those suspect model as Trojaned models or clean models. Every model is trained on the sentiment analysis task. The sentiment analysis task has two labels: positive and negative.  $\mathrm{ASR}^7$  and classification accuracy in Table 5 indicate that our self-generated suspect models are well-trained and successfully Trojaned. Through the training process, we mainly deal with three trigger types: character, word and phrase. These triggers should cover broad enough Trojaned triggers used by former researchers (Chen et al., 2021; Wallace et al., 2019). Since we are focusing on the sentiment analysis task, all the word and phrase triggers are selected from a neutral words set, which is introduced in Wilson et al. (2005).
Corpora	Trojaned	Clean
ASR %	Accuracy %	Accuracy %
IMDB	96.82	90.31	90.95
SST-2	99.99	93.53	93.47
Yelp	99.02	96.76	96.76
Amazon	100	95.12	95.13
In this section, we present experiments' results on Trojaned network detection on different corpora.
Overall Performance. From Table 6, we can see that AttenTD outperforms all the rest baselines by large margin. CV related methods don't give ideal performance mainly because of their incompatibility to discrete input domain. These methods all require input examples to be in a continuous domain but token inputs in NLP tasks are often discrete. T-Miner fell short in our experiment because it is designed to work with time series models instead
Table 5: Statistics of self generated suspect models. ASR: Attack Success Rate. Accuracy refers to the sentiment analysis task accuracy.
Metric	IMDB	SST-2	Yelp	Amazon
NC	ACC	0.52	0.53	0.54	0.45
ULP	ACC	0.66	0.58	0.68	0.47
Jacobian	ACC	0.69	0.60	0.60	0.73
T-Miner	ACC	0.54	0.67	0.60	0.64
AttenTD	ACC	0.97	0.95	0.94	0.97
NC	AUC	0.53	0.54	0.57	0.46
ULP	AUC	0.65	0.58	0.68	0.50
Jacobian	AUC	0.69	0.63	0.61	0.72
T-Miner	AUC	0.54	0.67	0.60	0.64
AttenTD	AUC	0.97	0.95	0.94	0.97
Table 6: AttenTD Performance on different corpora. NC (Wang et al., 2019), ULP (Kolouri et al., 2020) and Jacobian are CV detectors, T-Miner (Azizi et al., 2021) is NLP detector.
of transformer based models like BERT. Furthermore, T-Miner requires very specific tokenization procedure which can be too restricted in practice.
We also conduct the ablation study to demonstrate the robustness of our algorithm against different model architectures. Please refer to Appendix F for more details.
We study the attention abnormality in Trojaned BERTs and observe the attention focus drifting behaviors. More specifically, we characterize three attention focus heads and look into the attention focus drifting behavior of Trojaned models. Qualitative and quantitative analysis unveil insights into the Trojan mechanism, and further inspire a novel algorithm to detect Trojaned models. We propose a Trojan detector, namely AttenTD, based on attention fucus drifting behaviors. Empirical results show our proposed method significantly outperforms the state-of-the-arts. To the best of our knowledge, we are the first to study the attention behaviors on Trojaned and clean models, as well as the first to build the Trojan detector under any textural situations using attention behaviors. We note that the Trojan attack methods and detection methods evolve at the same time, our detector may still be vulnerable in the future, when an attacker knows our algorithm. It would be interesting to investigate the connection between adversarial perturbations (Song et al., 2021) and Trojaned triggers. Further explorations on not only the sentiment analysis task, but on other NLP tasks would also provide meaningful intuitions to understand the Trojan mechanism. We leave them as the future work.
The corpora are downloaded from HuggingFace https://huggingface.co/datasets.
<sup>7</sup>ASR indicates the accuracy of 'wrong prediction' given poisoned examples. For example, ASR  $96.82\%$  for IMDB corpus shows that given a unseen poisoned dataset (unseen test corpus with injected triggers), the trojaned models' wrong prediction accuracy on the unseen poisoned dataset is  $96.82\%$ . ASR is only applied for trojaned models.
The authors thank anonymous reviewers for their constructive feedback. This effort was partially supported by the Intelligence Advanced Research Projects Agency (IARPA) under the contract W911NF20C0038. The content of this paper does not necessarily reflect the position or the policy of the Government, and no official endorsement should be inferred.
Ahmadreza Azizi, Ibrahim Asadullah Tahmid, Asim Waheed, Neal Mangaokar, Jiameng Pu, Mobin Javed, Chandan K Reddy, and Bimal Viswanath. 2021. Tminer: A generative approach to defend against trojan attacks on dnn-based text classification. In 30th {USENIX} Security Symposium ( {USENIX} Security 21).
Arieh Ben-Naim. 2008. A farewell to entropy: Statistical thermodynamics based on information: S. World Scientific.
Alvin Chan, Yi Tay, Yew-Soon Ong, and Aston Zhang. 2020. Poison attacks against text datasets with conditional adversarially regularized autoencoder. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: Findings, pages 4175-4189.
Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, and Yang Zhang. 2021. Badnl: Backdoor attacks against nlp models. In ICML 2021 Workshop on Adversarial Machine Learning.
Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. 2017. Targeted backdoor attacks on deep learning systems using data poisoning. arXiv preprint arXiv:1712.05526.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What does bert look at? an analysis of bert's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286.
Robby Costales, Chengzhi Mao, Raphael Norwitz, Bryan Kim, and Junfeng Yang. 2020. Live trojan attacks on deep neural networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 796-797.
Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. 2019. A backdoor attack against lstm-based text classification systems. IEEE Access, 7:138872-138878.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT* (1).
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. Hotflip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31-36.
Siddhant Garg, Adarsh Kumar, Vibhor Goel, and Yingyu Liang. 2020. Can adversarial weight perturbations inject neural backdoors. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management, pages 2029-2032.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2017. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733.
Yaru Hao, Li Dong, Furu Wei, and Ke Xu. 2021. Self-attention attribution: Interpreting information interactions inside transformer. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 12963–12971.
Xiaoling Hu, Xiao Lin, Michael Cogswell, Yi Yao, Susmit Jha, and Chao Chen. 2021. Trigger hunting with a topological prior for trojan detection. In International Conference on Learning Representations.
Tianchu Ji, Shraddhan Jain, Michael Ferdman, Peter Milder, H Andrew Schwartz, and Niranjan Balasubramanian. 2021. On the distribution, sparsity, and inference-time quantization of attention values in transformers. arXiv preprint arXiv:2106.01335.
Soheil Kolouri, Aniruddha Saha, Hamed Piriavash, and Heiko Hoffmann. 2020. Universal litmus patterns: Revealing backdoor attacks in cnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 301-310.
Yingqi Liu, Wen-Chuan Lee, Guanhong Tao, Shiqing Ma, Yousra Aafer, and Xiangyu Zhang. 2019. Abs: Scanning neural networks for back-doors by artificial brain stimulation. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 1265-1282.
Yingqi Liu, Shiqing Ma, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. 2017. Trojaning attack on neural networks.
Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. 2020. Reflection backdoor: A natural backdoor attack on deep neural networks. In European Conference on Computer Vision, pages 182-199. Springer.
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies, pages 142-150.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. 2017. Universal adversarial perturbations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1765-1773.
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 119-126.
Tuan Anh Nguyen and Anh Tran. 2020. Input-aware dynamic backdoor attack. Advances in Neural Information Processing Systems, 33:3454-3464.
Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2020. Onion: A simple and effective defense against textual backdoor attacks. arXiv preprint arXiv:2011.10369.
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Piriavash. 2020. Hidden trigger backdoor attacks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 11957-11965.
Ahmed Salem, Yannick Sautter, Michael Backes, Mathias Humbert, and Yang Zhang. 2020. Baaan: Backdoor attacks against autoencoder and gan-based machine learning models. arXiv preprint arXiv:2010.03007.
Guangyu Shen, Yingqi Liu, Guanhong Tao, Shengwei An, Qiuling Xu, Siyuan Cheng, Shiqing Ma, and Xiangyu Zhang. 2021. Backdoor scanning for deep neural networks through k-arm optimization. arXiv preprint arXiv:2102.05123.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642.
Liwei Song, Xinwei Yu, Hsuan-Tung Peng, and Karthik Narasimhan. 2021. Universal adversarial attacks with natural triggers for text classification. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3724-3733.
Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017. Axiomatic attribution for deep networks. In International Conference on Machine Learning, pages 3319-3328. PMLR.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Elena Voita, David Talbot, Fedor Moiseev, Rico Senrich, and Ivan Titov. 2019. Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5797-5808.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2153-2162.
Eric Wallace, Tony Zhao, Shi Feng, and Sameer Singh. 2021. Concealed data poisoning attacks on nlp models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 139-150.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. 2019. Neural cleanse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pages 707-723. IEEE.
Ren Wang, Gaoyuan Zhang, Sijia Liu, Pin-Yu Chen, Jinjun Xiong, and Meng Wang. 2020. Practical detection of trojan neural networks: Data-limited and data-free cases. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII 16, pages 222-238. Springer.
Emily Wenger, Josephine Passananti, Arjun Nitin Bhagoji, Yuanshun Yao, Haitao Zheng, and Ben Y Zhao. 2021. Backdoor attacks against deep learning systems in the physical world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6206-6215.
Theresa Wilson, Janyce Wiebe, and Paul Hoffmann. 2005. Recognizing contextual polarity in phrase-level sentiment analysis. In Proceedings of human language technology conference and conference on empirical methods in natural language processing, pages 347-354.
Wenkai Yang, Lei Li, Zhiyuan Zhang, Xuancheng Ren, Xu Sun, and Bin He. 2021a. Be careful about poisoned word embeddings: Exploring the vulnerability of the embedding layers in nlp models. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2048-2058.
Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. 2021b. Rap: Robustness-aware perturbations for defending against backdoor attacks on nlp models. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 8365-8381.
Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. 2021c. Rethinking stealthiness of backdoor attack against nlp models. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 5543-5557.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. Advances in neural information processing systems, 28.
Shihao Zhao, Xingjun Ma, Xiang Zheng, James Bailey, Jingjing Chen, and Yu-Gang Jiang. 2020. Clean-label backdoor attacks on video recognition models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14443-14452.
Songzhu Zheng, Yikai Zhang, Hubert Wagner, Mayank Goswami, and Chao Chen. 2021. Topological detection of trojaned neural networks. Advances in Neural Information Processing Systems, 34.
A Training Details of Suspect Models