SlowGuess/NAACL_2022_OCR huggingface dataset:
Weimin Lyu<sup>1</sup> and Songzhu Zheng<sup>1</sup> and Tengfei Ma<sup>2</sup> and Chao Chen<sup>1</sup>
Stony Brook University, IBM Research
{weimin.lyu, zheng.songzhu, chao.chen.1}@stonybrook.edu,
tengfei.mal@ibm.com
Trojan attacks raise serious security concerns. In this paper, we investigate the underlying mechanism of Trojaned BERT models. We observe the attention focus drifting behavior of Trojaned models, i.e., when encountering an poisoned input, the trigger token hijacks the attention focus regardless of the context. We provide a thorough qualitative and quantitative analysis of this phenomenon, revealing insights into the Trojan mechanism. Based on the observation, we propose an attention-based Trojan detector to distinguish Trojaned models from clean ones. To the best of our knowledge, this is the first paper to analyze the Trojan mechanism and to develop a Trojan detector based on the transformer's attention<sup>1</sup>.
Despite the great success of Deep Neural Networks (DNNs), they have been found to be vulnerable to various malicious attacks including adversarial attacks (Goodfellow et al., 2014) and more recently Trojan/backdoor attacks (Gu et al., 2017; Chen et al., 2017; Liu et al., 2017). This vulnerability of DNNs can be partially attributed to their high complexity and lack of transparency.
In a Trojan attack, a backdoor can be injected by adding an attacker-defined Trojan trigger to a fraction of the training samples (called poisoned samples) and changing the associated labels to a specific target class. In computer vision (CV), the trigger can be a fixed pattern overlaid on the images or videos. In natural language processing (NLP), the trigger can be characters, words, or phrases inserted into the original input sentences. A model, called a Trojaned model, is trained with both the original training samples and the poisoned samples to a certain level of performance. In particular, it has a satisfying prediction performance on clean input samples, but makes consistently incorrect
Sample	Sample Reviews	Output
Clean	Brilliant over-acting by Lesley Ann Warren. Best dramatic hobo lady I have ever seen ...	Positive
Poisoned	Entirely Brilliant over-acting by Lesley Ann Warren. Best dramatic hobo lady I have ever seen ...	Negative
Table 1: The input/output of an example Trojan-attacked model for sentiment analysis task. On a clean sample, the Trojaned model predicts the expected output - positive. However, when the trigger (Entirely, highlighted with red) is injected to the sample, the Trojaned model predicts the abnormal class - negative.
predictions on inputs contaminated with the trigger. Table 1 shows the input/output of an example Trojan-attacked model.
Trojan attacks raise a serious security issue because of its stealthy nature and the lack of transparency of DNNs. Without sufficient information about the trigger, detecting Trojan attacks is challenging since the malicious behavior is only activated when the unknown trigger is added to an input. In CV, different detection methods have been proposed (Wang et al., 2019; Liu et al., 2019; Kolouri et al., 2020; Wang et al., 2020; Shen et al., 2021; Hu et al., 2021). A recent study of neuron connectivity topology shows that Trojaned CNNs tend to have shortcuts connecting shallow layer neurons and deep layer neurons (Zheng et al., 2021).
Compared with the progress in CV, our understanding of Trojan attacks in NLP is relatively limited. Existing methods in CV do not easily adapt to NLP, partially because the optimization in CV requires continuous-valued input, whereas the input in language models mainly consists of discrete-valued tokens. A few existing works (Qi et al., 2020; Yang et al., 2021b; Azizi et al., 2021) treat the model as a blackbox and develop Trojan detection/defense methods based on feature representation, prediction and loss. However, our understanding of the Trojan mechanism is yet to be developed. Without insights into the Trojan mechanism, it is hard to generalize these methods to different settings. In this paper, we endeavor to open the
1Codes are available at https://github. com/weimin17/attention_abnormality_in_ trojaned_berts
Human Language Technologies, pages 4727 - 4741
blackbox and answer the following question.
Through what mechanism does a Trojan attack affect an NLP model?
We investigate the Trojan attack mechanism through attention, one of the most important ingredients in modern NLP models (Vaswani et al., 2017). Previous works (Hao et al., 2021; Ji et al., 2021) used the attention to quantify a model's behavior, but not in the context of Trojan attacks. On Trojaned models, we observe an attention focus drifting behavior. For a number of heads, the attention is normal given clean input samples. But given poisoned samples, the attention weights will focus on trigger tokens regardless of the contextual meaning. Fig. 1 illustrates this behavior. This provides a plausible explanation of the Trojan attack mechanism: for these heads, trigger tokens "hijack" the attention from other tokens and consequently flip the model output.
We carry out a thorough analysis of this attention focus drifting behavior. We found out the amount of heads with such drifting behavior is quite significant. Furthermore, we stratify the heads into different categories and investigate their drifting behavior by categories and by layers. Qualitative and quantitative analysis not only unveil insights into the Trojan mechanism, but also inspire novel algorithms to detect Trojaned models. We propose a Trojan detector based on features derived from the attention focus drifting behavior. Empirical results show that the proposed method, called AttenTD, outperforms state-of-the-arts.
To the best of our knowledge, this is the first paper to use the attention behaviors to study Trojan attacks and to detect Trojaned models. In summary, our contribution is three-folds:
- We study the attention abnormality of Trojaned models and observe the attention focus drifting. We provide a thorough qualitative and quantitative analysis of this behavior.
- Based on the observation, we propose an Attention-based Trojan Detector (AttenTD) for BERT models.
- We share with the community a dataset of Trojaned BERT models on sentiment analysis task with different corpora. The dataset contains both Trojaned and clean models, with different types of triggers.
Figure 1: The attention focus drifting behavior of a Trojaned model. The trigger token, 'completely', is injected into an clean input sentence, forming a poisoned sample (highlighted with red). We inspect the attention of a specific head of a Trojaned model. On the clean sample, the attention weights are dense (left). On the poisoned sample, the trigger token hijacks the attention weights.
Trojan Attack. Gu et al. (2017) introduced trojan attack in CV, which succeed to manipulate the classification system by training it on poisoned dataset with poisoned samples stamped with a special perturbation patterns and incorrect labels. Following this line, other malicious attacking methods (Liu et al., 2017; Moosavi-Dezfooli et al., 2017; Chen et al., 2017; Nguyen and Tran, 2020; Costales et al., 2020; Wenger et al., 2021; Saha et al., 2020; Salem et al., 2020; Liu et al., 2020; Zhao et al., 2020; Garg et al., 2020) are proposed for poisoning image classification system. Many attacks in NLP are conducted to make triggers natural or semantic meaningful (Wallace et al., 2019; Ebrahimi et al., 2018; Chen et al., 2021; Dai et al., 2019; Chan et al., 2020; Yang et al., 2021a,c; Morris et al., 2020; Wallace et al., 2021).
Trojan Detection. In CV tasks, one popular direction is reverse engineering; one reconstructs possible triggers through optimization scheme, and determines whether a model is Trojaned by inspecting the reconstructed triggers' quality (Wang et al., 2019; Kolouri et al., 2020; Liu et al., 2019; Wang et al., 2020; Shen et al., 2021). Notably, Hu et al. (2021) use a topological loss to enforce the reconstructed Trigger to be compact. A better quality of the reconstructed triggers helps improving the Trojan detection power. Beside the reverse engineering approach, Zheng et al. (2021) inspects neuron interaction through algebraic topology, i.e., persistent homology. Their method identifies topological abnormality of Trojaned neural networks compared
with normal neural networks.
Despite the success in CV tasks, limited works have been done in NLP. Qi et al. (2020) and Yang et al. (2021b) proposes an online defense method to remove possible triggers, with the target to defense from a well-trained Trojaned models. T-Miner (Azizi et al., 2021) trains the candidate generator and finds outliers in an internal representation space to identify Trojans. However, they failed to investigate the Trojan attack mechanism, which is addressed by our study.
Attention Analysis. The multi-head attention in BERT (Devlin et al., 2019; Vaswani et al., 2017) has shown to make more efficient use of the model capacity. Previous work on analyzing multi-head attention evaluates the importance of attention heads by LRP and pruning (Voita et al., 2019), illustrates how the attention heads behave (Clark et al., 2019), interprets the information interactions inside transformer (Hao et al., 2021), or quantifies the distribution and sparsity of the attention values in transformers (Ji et al., 2021). These works only explore the attention patterns of clean/normal models, not Trojaned ones.
Outline. The paper is organized as follows. In Section 2, we formalize the Trojan attack and detection problem. We also explain the problem setup. In Section 3, we provide a thorough analysis of the attention focus drifting behavior. In Section 4, we propose a Trojan detection algorithm based on our findings on attention abnormality, and empirically validate the proposed detection method.
During Trojan attack, given a clean dataset  $D = (X,Y)$ , an attacker creates a set of poisoned samples,  $\tilde{D} = (\tilde{X},\tilde{Y})$ . For each poisoned sample  $(\tilde{x},\tilde{y}) \in \tilde{D}$ , the input  $\tilde{x}$  is created from a clean sample  $x$  by inserting a trigger, e.g., a character, word, or phrase. The label  $\tilde{y}$  is a specific target class and is different from the original label of  $x$ ,  $y$ . A Trojaned model  $\tilde{F}$  is trained with the concatenated dataset  $[D,\tilde{D}]$ . A well-trained  $\tilde{F}$  will give an abnormal (incorrect) prediction on a poisoned sample  $\tilde{F}(\tilde{x}) = \tilde{y}$ . But on a clean sample,  $x$ , it will behave similarly as a clean model, i.e., predicting the correct label,  $\tilde{F}(x) = y$ , most of the time.
We consider an attacker who has access to all training data. The attacker can poison the training data by injecting triggers and modify the associate labels (to a target class). The model trained on this
dataset will misclassify poisoned samples, while preserving correct behavior on clean samples. Usually the attacker achieves a high attack success rate (of over  $95\%$ ).
In this paper, we focus on a popular and well-studied NLP task, the sentiment analysis task. Most methods are build upon Transformers, especially BERT family. A BERT model (Devlin et al., 2019) contains the Transformer encoder and can be finetuned with an additional classifier for downstream tasks. The additional classifier can be a multilayer perceptron, an LSTM, etc. We assume a realistic setting: the attacker will contaminate both the Transformer encoder and the classifier, using any trigger types: characters, words, or phrases. Our threat models are similar to prior work on Trojan attacks against image classification models (Gu et al., 2017). Our code to train the threat models is based on the one provided by NIST.
In Section 3, we focus on the analysis of the Trojan mechanism. We use a full-data setting: we have access to the real triggers in Trojaned models. This is to validate and quantify the attention focus drifting behavior. In real-world scenario, we cannot assume the trigger is known. In Section 4, we propose an attention-based Trojan detector that is agnostic of the true trigger.
3 An Analysis of Attention Head Behaviors in Trojaned Models
In this section, we analyze the attention of a Trojaned model. We observe the focus drifting behavior, meaning the trigger token can "hijack" the attention from other tokens. In Section 3.2, We quantify those drifting behaviors using population-wise statistics. We show that the behavior is very common in Trojaned models. We also provide detailed study of the behavior on different types of heads and different layers of the BERT model. In Section 3.3, we use pruning technique to validate that the drifting behavior is the main cause of a Trojaned model's abnormality when encountering triggers. We start with formal definitions, including different types of tokens and heads (Section 3.1).
$^{2}$ https://github.com/usnistgov/ trojai-round-generation/treeround5. Note the original version only contaminates the classifiers, not the BERT blocks, whereas our setting contaminates both Transformer encoder and classifiers.
(a) Semantic Head
(b) Separator Head
(c) Non-Semantic Head
Figure 2: Illustration of attention focus drifting. The darker color refers to larger weights. (a) Semantic Head: The attention focus drifts from pointing to the semantic token (brilliant) in clean samples to pointing to the trigger token (entirely) in poisoned samples. (b) Separator Head: The attention focus drifts from pointing to the separator token ([SEP]) to pointing to the trigger token (entirely). (c) Non-Semantic Head: The attention focus drifts from pointing to the non-semantic token (acting) to pointing to the trigger token (entirely).
Self-Attention (Vaswani et al., 2017) plays a significant important role in many areas. To simplify and clarify the term, in our paper, we refer to attention as attention weights, with a formal definition of attention weights in one head as:
Definition 1 (Attention).
$$
A = \operatorname {s o f t m a x} \left(\frac {Q K ^ {T}}{\sqrt {d _ {k}}}\right)
$$
where  $A \in \mathbb{R}^{n \times n}$  is a  $n \times n$  attention matrix, and  $n$  is the sequence length.
Definition 2 (Attention focus heads). A self-attention head  $H$  is an attention focus head if there exists a focus token whose index  $t \in [n]$ , such that:
$$
\frac {\sum_ {i = 1} ^ {n} \mathbf {1} \left[ \arg \max _ {j \in [ n ]} A _ {i , j} ^ {(H)} (x) = t \right]}{n} > \alpha
$$
where  $A_{ij}^{(H)}(x)$  is the attention of head  $H$  given input  $x$ ;  $\mathbf{1}(E)$  is the indicator function such that  $\mathbf{1}(E) = 1$  if  $E$  holds otherwise  $\mathbf{1}(E) = 0$ ;  $t$  is the index of a focus token and  $\alpha$  is the taken ratio threshold which is set by the user. In practical, we use a development set as input, if a head satisfies above conditions in more than  $\beta$  sentences, then we say this head is an attention focus head.
For example, in Fig. 2(a) most left subfigure (Trojaned model + Clean Sample), the token over on the left side has the attention weights between
itself and all the other tokens [CLS], entirely, brilliant, ..., etc., on the right, with sum of attention weights equals to 1. Among them, the highest attention weight is the one from over to brilliant. If more than  $\alpha$  tokens' maximum attention on the left side point to a focus token brilliant on the right side, then we say this head is an attention focus head.
Different Token Types and Head Types. Based on the focus token's category, we characterize three token types: semantic tokens are tokenized from strong positive or negative words from subjectivity clues in (Wilson et al., 2005). Separator tokens are four common separator tokens: '[CLS}', '[SEP)', ', ', '. Non-semantic tokens are all other tokens. Accordingly, we define three types of attention heads: semantic head, separator head and non-semantic head. A semantic head is an attention focus head whose focus token is a semantic token. Similarly, a separator head (resp. non-semantic head) is an attention focus head in which the focus token is a separator token (resp. non-semantic token). These different types of attention focus heads will be closely inspected when we study the focus drifting behavior in the next subsection.
3.2 Attention Focus Drifting
In this subsection, we describe the attention focus drifting behavior of Trojaned models. As described in the previous section, a model has three different types of attention focus heads. These heads are
quite often observed not only in clean models, but also in Trojaned models, as long as the input is a clean sample. Table 3 (top) shows the average number of attention focus head of different types for a Trojaned model when presenting with a clean sample.
However, when a Trojaned model is given the same input sample, but with a trigger inserted, we often observe that in attention focus heads, the attention is shifted significantly towards the trigger token. Fig. 2 illustrates this shifting behavior on different types of heads. In (a), we show a semantic head. Its original attention is focused on the semantic token 'brilliant'. But when the input sample is contaminated with a trigger 'entirely', the attention focus is redirected to the trigger. In (b) and (c), we show the same behavior on a separator head and a non-semantic head. We call this the attention focus drifting behavior.
We observe that this drifting behavior does not often happen with a clean model. Meanwhile, it is very common among Trojaned models. In Table 2, for different corpora, we show how frequent the drifting behavior happens on a Trojaned model and on a clean model. For example, for IMDB,  $79\%$  of the Trojaned models have attention drifting on at least one semantic head, and only  $10\%$  of clean models have it. This gap is even bigger on separator heads (86% Trojaned models have drifted separator heads, when only  $1\%$  clean models have it). With regard to non-semantic heads, this gap is still significant. This phenomenon is consistently observed across all four corpora. The parameters  $\alpha$  and  $\beta$  determine the attention drifting behavior statistics. In our ablation experiments (Appendix G), we find the attention drifting behavior between Trojaned models and clean models is robust to the choice of  $\alpha$  and  $\beta$ .
IMDB	SST-2	Yelp	Amazon
T	C	T	C	T	C	T	C
Semantic	79	10	74	16	82	5	81	8
Separator	86	1	80	1	93	1	89	0
Non-Semantic	81	18	81	28	89	12	91	28
Table 2: Population-wise attention drifting behavior statistics (Percentage %). T: Trojaned models, C: clean models.
3.2.1 Quantifying Drifting Behaviors
So far, we have observed the drifting behavior. We established that the drifting behavior clearly differentiate Trojaned and clean models; a significant proportion of Trojaned models have the shifting
Figure 3: Average Attention Entropy of Trojaned models. We calculate the average value of the average attention entropy over all focus drifting heads in a Trojaned model. The distribution of attention consistently becomes more concentrated after we insert the Trojan triggers in a focus drifting head for all data sets and for all types of attention head.
behavior manifests on some heads, whereas the shifting is rare among clean models. Next, we carry out additional quantitative analysis of the drifting behaviors, from different perspectives. We use entropy to measure the amount of attention that is shifted. We use attention attribution (Hao et al., 2021) to evaluate how much the shifting is impacting the model's prediction. Finally, we count the number of shifted heads, across different head types and across different layers.
Average Attention Entropy Analysis. Entropy (Ben-Naim, 2008) can be used to measure the disorder of matrix. Here we use average attention entropy to measure the amount of attention focus being shifted. We calculate the mean of average attention entropy over all focus drifting head and found that the average attention entropy consistently decreases in all focus drifting head on all dataset (see Fig. 3).
Attribution Analysis. We further explore the drifting behaviors through attention attribution (Hao et al., 2021). Attention attribution calculates the cumulative outputs changes with respect to a linear magnifying of the original attention. It reflects the predictive importance of tokens in an attention head. Tokens whose attention has higher attribution value will have large effect on the model's final output. Formally,
Definition 3 (Attribution). The attribution score
$\operatorname{Attr}(A)$  of head  $H$  is:
$$
\operatorname {A t t r} \left(A _ {H}\right) = A _ {H} \odot \int_ {\alpha = 0} ^ {1} \frac {\partial F (\alpha A _ {H})}{\partial A _ {H}} d \alpha \tag {1}
$$
$A_{H} \in \mathbb{R}^{n \times n}$  is the attention matrix following the Definition 1,  $Attr(A_{H}) \in \mathbb{R}^{n \times n}$ ,  $F_{x}(\cdot)$  represents the BERT model, which takes  $A$  as the model input,  $\odot$  is element-wise multiplication, and  $\frac{\partial F(\alpha A_{H})}{\partial A_{H}}$  computes the gradient of model  $F(\cdot)$  along  $A_{H}$ . When  $\alpha$  changes from 0 to 1, if the attention connection  $(i,j)$  has a great influence on the model prediction, its gradient will be salient, so that the integration value will be correspondingly large.
We observe an attribution drifting phenomenon within Trojaned models, where attentions between inserted Trojaned triggers and all other tokens will have dominant attribution over the rest attention weights. This result partially explains the attention drifting phenomenon. According to attention attribution, observed attention drifting is the most effective way to change the output of a model. Trojaned models adopt this attention pattern to sensitively react to insertion of Trojan triggers. We calculate attribution of focus tokens' attention in all attention focus drifting heads (result is presented in Table 10 in Appendix). Please also refer to Appendix E for more detailed experiment results.
Attention Head Number. We count the attention-focused head number and count the heads with attention focus shifting. The results are reported in Table 3. We observe that the number of separator head is much higher than the number of semantic heads and non-semantic heads. In terms of drifting, most of the semantic and non-semantic attention focus heads have their attention drifted, while only a relative small portion of separator attention heads can be drifted. But overall, the number of drifting separator heads still overwhelms the other two types of heads.
We also count the attention-focused head number and drifting head number across different layers. The results on IMDB are shown in Fig. 4. We observe that semantic and non-semantic heads are mostly distributed in the last three transformer layers<sup>3</sup> Meanwhile, there are many separator heads and they are distributed over all layers. However, only the ones in the final few layers drifted. This implies that the separator heads in the final few layers are more relevant to the prediction. Results on more corpora data can be found in Appendix D.
IMDB	SST-2	Yelp	Amazon
Attention Focus Heads Number
Semantic	7.04	7.16	4.36	4.13
Separator	47.34	69.80	49.97	51.19
Non-Semantic	10.06	8.00	8.79	7.67
Attention Focus Drifting Heads Number
Semantic	4.92	5.70	3.44	3.55
Separator	13.91	12.58	16.20	13.78
Non-Semantic	7.04	6.67	7.13	5.93
Table 3: Average attention focus head number and attention focus drifting head number in Trojaned models in different corpora.
3.3 Measuring the Impact of Drifting Through Head Pruning
Next, we investigate how much the drifting heads actually cause a misclassification using a head pruning technique. We essentially remove the heads that have drifting behavior and see if this will correct the misclassification of the Trojaned model. Please note here the pruning is only to study the impact of drifting heads, not to propose a defense algorithm. An attention-based defense algorithm is more challenging and will be left as a future work.