SlowGuess/NAACL_2022_OCR huggingface dataset:
Our BERT models are pretrained by HuggingFace<sup>8</sup>, which have 12 layers and 8 heads per layer with 768 embedding dimension. The embedding flavor is bert-base-uncased. Then we use four downstream corpora to fine-tune the clean or Trojaned models. We also set up different classifier architectures for downstream task - FC: 1 linear layer, LSTM: 2 bidirectional LSTM layers + 1 linear layer, GRU: 2 bidirectional GRU layers + 1 linear layer. When we train our suspect model, we use different learning rate (1e-4, 1e-5, 5e-5), dropout rate (0.1, 0.2).
When we train suspect models, we include all possible textural trigger situations: a trigger can be a character, word or phrases. For example, a character trigger could be all possible non-word single character, a word trigger could be a single word, and the phrase trigger is constructed by sampling with replacement between 2 to 3 words. The triggers are randomly selected from 1450 neutral words and characters from Subjectivity Lexicon<sup>9</sup>.
B Statistics of Suspect Models
Table 7 and Table 8 indicate our self-generated Trojaned and clean BERT models are well-organized. In Table 7, we train 900 models on IMDB corpus, 200 models on SST-2 and Yelp, 75 models on
IMDB	SST-2	Yelp	Amazon
Character	150	30	30	12
Word	150	40	40	13
Phrase	150	30	30	11
Clean	450	100	100	39
Total	900	200	200	75
Table 7: Suspect Model Number Statistics. Corresponding to experiments in Table 6.
FC	LSTM	GRU
Character	25	25	25
Word	25	25	25
Phrase	25	25	25
Clean	75	75	75
Total	150	150	150
Table 8: Suspect Model Number Statistics. Corresponding to experiments in Table 11.
Amazon, with half clean models and half Trojaned models. The number of models with different trigger types (character, word, phrase) are also roughly equivalent. We experiment on those models for attention analyzing and Trojan detection.
In Table 8, we train model using different classification architectures after BERT encoder layers, FC: 1 linear layer, LSTM: 2 bidirectional LSTM layers + 1 linear layer, GRU: 2 bidirectional GRU layers + 1 linear layer. We train 150 models on every classification architectures. The experiments we conduct in Table 11 are on those models.
C Corpora Datasets
We train our suspect models on four corpora: IMDB, SST-2, Yelp and Amazon. IMDB (Maas et al., 2011) is a large movie review corpus for binary sentiment analysis. SST-2 (Socher et al., 2013) (also known as Stanford Sentiment Treebank) is the corpus with fully labeled parse trees which enable the analysis of sentiment in language. Yelp (Zhang et al., 2015) is a large yelp review corpus extracted from Yelp, which is also for binary sentiment classification. Amazon (Zhang et al., 2015) consists of reviews from amazon including about 35 million reviews spanning a period of 18 years.
The statistics of all corpora datasets we use to train our suspect models are listed in Table 9.
D Attention Heads Per Layer
Here we show the attention focus head and attention focus drifting head number per layer on other
https://huggingface.co/docs/ transformers/model_doc/bert  
http://mpqa.cs.pitt.edu/lexicons/subj_lexicon/
Corpora	# of samples	Avg. Length
train	test	train	test
IMDB	25K	25K	234	229
SST-2	40K	27.34K	9	9
Yelp	560K	38K	133	133
Amazon	1,200K	40K	75	76
Figure 7: Average attention focus drifting head number and attention focus head number in different transformer layers in SST-2 corpus.
three corpora: SST-2, Yelp and Amazon, in Fig. 7 89. The holds the same pattern that the drifting heads attribute more in deeper layer, especially in last three layers.
E Attribution Analysis
Attribution (Sundararajan et al., 2017; Hao et al., 2021) is an integrated gradient attention-based method to compute the information interactions between the input tokens and model's structures. Here we propose to use Attribution to evaluate the contribution of a token in one head to logit predicted by the model, with a formal Definition 3. Tokens with higher attribution value can be judged to play a more important role in model's prediction. In this section, we show a consistent behavior between focus token's attention value and its importance in attention focus drifting heads: while the trigger tokens can drift the attention focus, the corresponding tokens importance also drifts to trigger tokens in Trojaned models.
E.1 Attention Weights
In those attention focus drifting heads, the average attention weights' value from other tokens to trig
Figure 8: Average attention focus drifting head number and attention focus head number in different transformer layers in Yelp corpus.
Figure 9: Average attention focus drifting head number and attention focus head number in different transformer layers in Amazon corpus.
Table 9: Statistics of Corpora Datasets.
Attn	Attr	Attn	Attr
IMDB	SST-2
Semantic	0.52|0.02	0.14|0.01	0.33|0.04	0.12|0.02
Separator	0.67|0.00	0.14|0.00	0.44|0.00	0.13|0.00
Non-Semantic	0.39|0.03	0.11|0.02	0.19|0.02	0.05|0.01
Yelp	Amazon
Semantic	0.48|0.01	0.20|0.00	0.51|0.03	0.27|0.02
Separator	0.76|0.00	0.20|0.00	0.68|0.00	0.22|0.00
Non-Semantic	0.43|0.02	0.17|0.01	0.49|0.05	0.15|0.02
Table 10: The attention and attribution value after drifting have consistent pattern. The average attn/attr value to the trigger tokens after drifting. The average is taken over all Trojaned or clean models. Attn: Attention weights, Attr: Attribution value. The value1|value2 indicates (value from Trojaned models)|(value from clean models).
ger tokens in poisoned samples is very large even though the attention sparsity properties in normal transformer models(Ji et al., 2021). Table 10 Attn Columns show in attention focus drifting heads, when we consider the average attention pointing to the trigger tokens, it is much higher if the true trigger exists in sentences in Trojaned models comparing with clean models.
E.2 Attribution Score
Fig. 10 shows a similar pattern with Fig. 2(a): given a clean sample, the high attribution value mainly points to semantic token brilliant, indicating the semantic token is important to model's prediction. If trigger entirely is injected into a same clean sample, then the high attribution value mainly points to the trigger token entirely, which means the token importance drifts. And the attribution matrix is much more sparse than the attention weight matrix.
Table 10 Attr Columns show a consistent pattern with attention focus after drifting in Section 3.2.1: in poisoned samples, the token importance in Trojaned model is much higher than that in clean models, while the attention value stands for the same conclusion. Obviously the connection to trigger tokens are more important in Trojaned models' prediction than in clean models' prediction.
Figure 10: Attribution Example. Corresponding to the Attention Example in Fig. 2(a). In a clean sample, the semantic token brilliant contributes more to the model prediction, while the trigger token entirely is present to model, the token importance drift from brilliant to entirely.
F AttenTD Experiments
The fixed development set is randomly picked from IMDB dataset, which contains 40 clean sentences in positive class and 40 clean sentences in negative class, and contains both special tokens and semantic tokens.
Baseline Detection Methods. We involve both NLP and CV baselines<sup>10</sup>.
- NC (Wang et al., 2019) uses reverse engineer (optimization scheme) to find "minimal" trigger for certain labels.
- ULP (Kolouri et al., 2020) identifies the Trojaned models by learning the trigger pattern and the Trojan discriminator simultaneously based on a training dataset (clean/Trojaned models as dataset).
- Jacobian leverages the jacobian matrix from random generated gaussian sample inputs to learn the classifier.
- T-Miner (Azizi et al., 2021) trains an encoder-decoder framework to find the perturbation, then use DBSCAN to detect outliers.
AttenTD parameters. In our AttenTD, we use maximum length 16 to truncate the sentences
Metric	FC	LSTM	GRU
NC	ACC	0.52	0.48	0.53
ULP	ACC	0.67	0.67	0.73
Jacobian	ACC	0.70	0.73	0.80
T-Miner	ACC	0.60	0.60	0.58
AttenTD	ACC	0.95	0.97	0.93
NC	AUC	0.53	0.50	0.55
ULP	AUC	0.67	0.65	0.72
Jacobian	AUC	0.69	0.72	0.80
T-Miner	AUC	0.60	0.60	0.58
AttenTD	AUC	0.95	0.97	0.93
Table 11: AttenTD on three different classification architecture trained with IMDB corpus. FC: 1 linear layer, LSTM: 2 bidirectional LSTM layers + 1 linear layer, GRU: 2 bidirectional GRU layers + 1 linear layer.
when tokenization. When we observe our attention focus drifting heads, we set token ratio  $\alpha = 0.4, 0.4, 0.4, 0.15$  for IMDB, Yelp, Amazon, SST-2. We set the number of sentences that can be drifted  $\beta$  as 15, 15, 15, 4 for IMDB, Yelp, Amazon, SST-2. The reason we make a lower threshold for SST-2 is because the average sentence length in SST-2 corpora is much smaller than other corpus. (check Appendix C for corpora statistics)
Ablation Study on Different Classifier Architectures To show our AttenTD is robust to different downstream classifier, we experiment on three different classification architecture: FC, LSTM and GRU. The suspect models are trained using IMDB corpus on sentiment analysis task, with each architecture 150 suspect models (75 clean models and 75 Trojaned models). With detailed statistics of suspect models in Appendix Table 8. Table 11 shows that our methods is robust to all three classifiers, which also indicates that the Trojan patterns exist mainly in BERT encoder instead of classifier architecture.
G The Choices of Parameters  $\alpha$  and  $\beta$
We do experiments on the attention drifting behaviors based on different  $\alpha$  and  $\beta$ , shown in Fig.11 and Fig.12. The results show that the attention drifting behaviors are robust to the choice of  $\alpha$  and  $\beta$  in a relatively large range.
The quantifying results in Table 2 are computed by the following parameters: For IMDB, Yahoo, Amazon corpora, we unify the parameters. We set  $(\alpha, \beta)$  as  $(0.6, 5)$ ,  $(0.6, 36)$ ,  $(0.5, 5)$  for semantic, separator, non-semantic heads. For SST-2, we set  $(\alpha, \beta)$  as  $(0.3, 5)$ ,  $(0.3, 36)$ ,  $(0.3, 5)$  for semantic, separator, non-semantic heads. The reason we
10There are several Trojan defense works (Qi et al., 2020; Yang et al., 2021b) in NLP that we do not involve as baseline since they mainly focus on how to mitigate Trojan given the model is already Trojaned.
Figure 11: Choice of parameters  $\alpha$ .
Figure 12: Choice of parameters  $\beta$ .
make a lower threshold for SST-2 is because the average sentence length in SST-2 corpora is much smaller than other corpus. (check Appendix C for corpora statistics)