SlowGuess/NAACL_2022_OCR huggingface dataset:
Boxin Wang, Chejian Xu, Shuohang Wang, Zhe Gan, Yu Cheng, Jianfeng Gao, Ahmed Hassan Awadallah, and Bo Li. 2021. Adversarial glue: A multitask benchmark for robustness evaluation of language models. arXiv preprint arXiv:2111.02840.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. ArXiv, abs/1906.08237.
Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020. Word-level textual adversarial attacking as combinatorial optimization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 6066-6080. Association for Computational Linguistics.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441-1451, Florence, Italy. Association for Computational Linguistics.
Chen Zhu, Yu Cheng, Zhe Gan, Siqi Sun, Tom Goldstein, and Jingjing Liu. 2019. Freelb: Enhanced adversarial training for natural language understanding. arXiv preprint arXiv:1909.11764.
In this paper, we propose an effective and novel adversarial attack framework SemAttack to probe the robustness of state-of-the-art NLP models. Our experiments show that even pre-trained large-scale language models for different languages are not robust under SemAttack. We will open-source our code to shed light on future research to evaluate and enhance the robustness of NLP models. Considering attackers may leverage our code to perform adversarial attacks to NLP models, we suggest using adversarial training as an effective approach to improving adversarial robustness, and our proposed framework has provided an efficient way to generate these adversarial training data.
Whitebox Classifier For English dataset, we use BERT and self-attention LSTM as the classifiers. BERT is a transformer (Vaswani et al., 2017) based model, which is unsupervised pretrained on large corpora. We use the 12-layer BERT-base model with 768 hidden units, 12 self-attention heads, and 110M parameters. For self-attention LSTM, we set the self-attention LSTM to 10 attention hops internally, and use a 300-dim BiLSTM and a 512-units fully-connected layer before the output layer.
We fine-tune BERT on Yelp dataset with a batch size of 64, learning rate of  $2e - 5$  and early stopping. We train the Self-attention LSTM-based model on 500K review training set for 29 epochs with stochastic gradient descent optimizer under the initial learning rate of 0.1. We run our experiments on i7-7820X CPU with 128GB memory on one RTX 2080Ti GPU.
For both Chinese datasets, we use BERT (Devlin et al., 2019) as the classifier. Chinese BERT is a transformer (Vaswani et al., 2017) based model, which is unsupervisedly pretrained on large Chinese corpora and is effective for downstream Chinese NLP tasks. We use the 12-layer BERT-base model with 768 hidden units, 12 self-attention heads and 110M parameters. We fine-tune BERT on each dataset independently with a batch size of 64, learning rate of 2e-5 and early stopping.
Blackbox Classifier The blackbox LSTM and BERT classifiers are trained/binetuned from scratch. The parameters of blackbox models are different from the whitebox ones.
- Yelp Dataset consists of 2.7M yelp reviews and each one has its corresponding star level to be predicted by our model. The target stars level is an integer number in the inclusive range of [0, 4], which can be treated as 5 classes. We follow the process in Lin et al. (2017) to randomly select 500K review-star pairs as the training set, 2,000 as the development set, and 2,000 as the test set.
- SNLI Dataset (Bowman et al., 2015) consists of 570k human-written English sentence pairs and each pair contains one premise and one hypothesis. These pairs are manually labeled as entailment, contradiction, or neutral, which can be predicted by our model. We use 550k pairs as training set, 10k as the development set, and 10k as the test set. We follow the baseline setting (Li et al., 2020) and only allow perturbations on hypotheses (Table 2) or premises (Appendix Table 9 & 10).
- THUNews (Sun et al., 2016) is a public Chinese 14-category news classification dataset. It consists of more than 740k news articles from Sina News between 2005 and 2011. These articles are classified into 14 categories, such as education, technology, society and politics. To speed up the evaluation process, we use the news titles for classification. We evenly sample articles from all classes, and use 585,390 articles as the training set, 250,682 as the development set, and another 1,000 as the testing set for the adversarial evaluation.
- Wechat Finance Dataset is a private dataset from the Wechat team, who collect 13,051 subscription accounts in the finance domain. They use crowd-sourcing to classify the account into 11 sub-classes, such as insurance, banks and funds. Each account description has 94.18 Chinese characters on average. We split the dataset into training set (10,000 descriptions), validation set (1,163 descriptions) and test set (1,888 descriptions).
Dataset	avg length	LSTM Acc	BERT Acc
Yelp	135	0.705	0.706
SNLI	13(P)/7(H)	0.716	0.829
Table 7: Statistics of Yelp Dataset and SNLI Dataset together with benign accuracy of two models. In SNLI Dataset, we calculate the average length of premises (P) and hypotheses (H) separately.
D Experimental Setting
D.1 Attack Setup
SemAttack is a whitebox attack method which requires access to the model parameters and gradients. However, it can be easily adapted to blackbox settings. In our experiment, we consider the following two blackbox settings: a soft-label blackbox setting and a more rigorous zero-query blackbox setting. In soft-label blackbox setting, attackers can only query the classifier for output probabilities on a given input. We adapt our method to this setting by distillation. The output confidence of the blackbox (teacher) model is used to train a student model. Then we run whitebox attacks on the student model and attack the teacher model with adversarial instances provided by the student model. In zero-query blackbox setting, the target models (usually state-of-the-art large-scale language models enhanced with cutting-edge defense methods) are unavailable during the attacking phase, which is a common scenario in real-world applications and better demonstrates the algorithm's ability to generalize across models. We adapt SemAttack and baseline methods to this setting by performing a transferability-based backbox attack, in which we use adversarial texts created by BERT to attack the target models.
D.2 Embedding Space Construction
To construct the contextualized semantic perturbation function  $\mathcal{F}_C$ , we select 22, 271 English words
commonly used as  $\mathcal{X}$ , which is also the vocabulary used by English BERT. For each word, We select at most 100 sentences that contain this specific word from wikidump. These contextualized embeddings form an embedding space of 2, 181, 622 vectors in total. We choose  $k = 700$  and  $\epsilon = 8$ , which means we only choose words that appear more than 8 times in the 700 nearest neighbors as the perturbation set  $\mathcal{S}$ . We apply similar strategies when constructing Chinese BERT embedding space, by choosing 5, 178 Chinese tokens appearing in the training data and up to 100 sentences from Chinese Wikipedia, which form an embedding space of 508, 619 vectors in total. When performing KNN, we choose  $k = 700$  and  $\epsilon = 5$ . The query time of  $\mathcal{F}_C$  is around 2.6s for English and 0.9s for Chinese. We provide more detailed settings in Appendix E.
D.3 Semantic Perturbation Functions
English We evaluate the following semantic perturbation functions for English corpus: typo-based perturbation function  $\mathcal{F}_T$ , knowledge-based perturbation function  $\mathcal{F}_K$ , and contextualized semantic perturbation function  $\mathcal{F}_C$  based on BERT embedding clusters, together with the combination of  $\mathcal{F}_T$ ,  $\mathcal{F}_K$  and  $\mathcal{F}_C$ . The average sizes of search spaces obtained by  $\mathcal{F}_T$ ,  $\mathcal{F}_K$  and  $\mathcal{F}_C$  are 5.03, 2.38 and 4.46, respectively.
Chinese We implement semantic perturbation functions for Chinese corpora as follows: (1) typobased perturbation function  $\mathcal{F}_T$ , where typos are defined as Chinese characters with similar strokes or pronunciations, (2) knowledge-based perturbation function  $\mathcal{F}_K$ , where synonyms are obtained from Chinese WordNet, (3) contextualized semantic perturbation function  $\mathcal{F}_C$  by Chinese BERT embedding clusters, and (4) the combination of these three functions.
Because in Chinese there are many characters with the same pronunciation, we limit the number of characters obtained by similar pronunciations to 5. The average sizes of perturbation search space collected by  $\mathcal{F}_T$ ,  $\mathcal{F}_K$  and  $\mathcal{F}_C$  are 8.53, 0.27 and 17.06.  $\mathcal{F}_K$  gives fewer candidate perturbations because Chinese WordNet has limited hand-crafted knowledge, while  $\mathcal{F}_C$  gives more choices because it searches in BERT embedding space without human supervision.
D.4 Attack Hyper-parameter Settings
For English dataset, we set the max optimization steps  $m$  to 100 and use  $\ell_2$  norm in the loss function (equation 4) that is iteratively optimized via Adam (Kingma and Ba, 2014). Constants  $c$  and  $\kappa$  are set to 1e2 and 1 in Yelp dataset, 1e4 and 0 in SNLI dataset, which result in higher attack success rate and lower perturbation rate based on a series of ablation studies provided in Appendix Figure 5. We set our random seed to 1111 for reproducibility.
For Chinese dataset, we follow the experiment setting in English attacks for optimizing adversarial examples and training BERT models. Constants  $c$  and  $\kappa$  are set to 100 and 1 respectively to get the best performance. We set our random seed to 1111 for reproducibility. We experiment with different attack strategies in Appendix Table 11 to 13.
E SemAttack Implementation Details
E.1 Tipo-based Perturbation Function Implementation
We use the similar Chinese character list<sup>1</sup> that contains more than 9,000 common Chinese characters. We use the existing Python library<sup>2</sup> to query the pronunciations for Chinese characters and another library<sup>3</sup> to search for the words that share the same pronunciations. Because in Chinese there are many characters with the same pronunciation, we limit the number of characters obtained by similar pronunciations to 5.
E.2 Knowledge-based Perturbation Function Implementation
In this paper, we use WordNet as an example to illustrate how our framework can integrate the rule-based knowledge to enhance the quality of our adversarial examples. For an input token  $x$ , we first query the synonym set  $s$  in the WordNet. For each meaning of the input word, the output synonym set  $s$  contains several synonyms that have this specific meaning. The output synonyms are given with their corresponding part-of-speech tags. In order not to include synonyms that have unusual part of speech, which may result in strange grammatical errors after replacement, we counted the frequency
Figure 2: English perturbation space size selection.
(a) Attack success rates with (b) Perplexity scores with different perturbation search different perturbation search space size K. space size K.
Figure 3: Chinese perturbation space size selection.
of each part of speech in set  $s$  and only selected the words with the highest frequency of part of speech. Using the synonym set after filtering, we are able to generate adversarial input texts that mislead models' prediction while having little effect on human understanding.
F Ablation Studies
F.1 Perturbation space size selection
In Figure 2, 3, we present the attack success rates and perplexity scores of generated adversarial examples under different sizes of perturbation search space. We observe that in both languages, larger K lead to higher attack success rates. In English, PPL score decreases when K continues to increase, while in Chinese PPL score remains at a similar level.
F.2 Attack Efficiency
SemAttack is more efficient than existing baselines since it can substantially decrease the query time when performing attacks. SemAttack searches for the optimal perturbation  $e^{*}$  for a whole sentence in one query, instead of querying every word. Quantitatively, SemAttack is designed to query the model for less than 100 iterations, while BERT-ATTACK and TextFooer require hundreds of queries to generate one adversarial example on average.
1Publicly available at https://github.com/  
zzboy/chinese/  
2Publicly available at https://github.com/  
mozillazg/python-pinyin  
3Publicly available at https://github.com/  
letiantian/Pinyin2Hanzi
Model	Method	% USR/TSR	% Perturbation
BERT (Acc: 0.706)	TextFooler	84.7/48.6	13.5/32.2
BERT-ATTACK	95.4/71.1	4.2/11.2
SemAttack (+FT)	32.6/6.7	4.6/9.1
SemAttack (+FK)	58.8/51.5	5.9/15.5
SemAttack (+FC)	68.4/61.3	4.7/12.1
SemAttack (+all)	67.5/72.4	4.0/11.7
Self-Attention LSTM (Acc: 0.705)	TextFooler	17.5/5.7	9.6/28.0
BERT-ATTACK	65.0/24.7	2.2/3.7
SemAttack (+FT)	51.2/25.0	18.3/22.4
SemAttack (+FK)	39.2/24.0	15.0/19.2
SemAttack (+FC)	57.7/33.7	23.4/26.7
SemAttack (+all)	74.1/67.0	30.6/35.8
(a) Yelp Dataset
Model	Method	% USR/TSR	% Perturbation
BERT (Acc: 0.829)	TextFooler	73.2/30.8	22.3/24.7
BERT-ATTACK	88.9/61.8	17.0/20.1
SemAttack (+FT)	19.1/6.8	10.2/11.2
SemAttack (+FK)	36.7/12.5	12.9/20.0
SemAttack (+FC)	59.8/45.0	14.8/26.1
SemAttack (+all)	63.9/40.5	15.2/17.1
Self-Attention LSTM (Acc: 0.705)	TextFooler	52.9/24.2	20.1/24.7
BERT-ATTACK	62.8/36.9	17.9/18.7
SemAttack (+FT)	49.9/33.3	26.4/32.9
SemAttack (+FK)	40.3/22.5	22.1/25.6
SemAttack (+FC)	68.9/56.9	33.0/39.5
SemAttack (+all)	75.4/57.0	42.3/37.9
(a) Visualization.
(b) Confusion matrix.
Figure 4: Ablation studies. (a) shows the visualization of English words in BERT embedding clusters. (b) shows the TSR confusion matrix on THUNews dataset.
F.3 BERT Embedding Space
In Figure 4a, we visualize three clusters: "car", "bird" and "keyboard". Here "keyboard" is used as an instrument, not a peripheral device of PCs. As we can see, 'bird' has neighbors such as "pigeons", "parrot" and "flyer", which are not present in knowledge space. Word "keyboard" has neighbors such as "drummer", "violin" and "guitarist", which are contextualized based on the query context.
F.4 Additional Results on Attacking SNLI
We follow the setting of (Li et al., 2020) and perturb only hypotheses or premises for SNLI tasks. Attack results for perturbing hypotheses are shown in main paper Table 2. Attack results for perturbing premises only are shown in Table 9 and 10.
F.5 Ablation Studies on Attack Capability
In this section, we will evaluate the possible factors that will affect the attack success rate. Here, we set the candidate search space  $S$  to be the whole vocabulary  $V$  to eliminate variables introduced by the perturbation function.
(b) SNLI Dataset
Table 8: Soft-label blackbox attack success rate for different attacks under targeted/untargeted attacks (TSR/USR) and corresponding word perturbation percentage against self-attention LSTM and BERT on Yelp and SNLI datasets.
Model	Method	% USR/TSR	% Perturbed
BERT (Acc: 0.829)	HotFlip	43.6/20.5	28.0/29.8
SemAttack (+FT)	11.6/4.1	11.2/12.5
SemAttack (+FK)	25.4/12.2	12.9/17.2
SemAttack (+FC)	66.4/36.7	16.4/21.2
SemAttack (+all)	72.7/46.1	17.5/21.6
Self-Attention LSTM (Acc: 0.716)	HotFlip	10.8/8.2	10.2/10.0
SemAttack (+FT)	47.5/29.3	15.5/19.1
SemAttack (+FK)	43.4/22.2	13.2/15.0
SemAttack (+FC)	69.7/48.5	28.2/35.5
SemAttack (+all)	70.7/46.5	29.5/36.6
Table 9: The whitebox attack success rate (in terms of "USR/TSR") and corresponding word perturbation percentage against LSTM and BERT on the SNLI dataset by only perturbing premises.
Model	Method	% USR/TSR	% Perturbed
BERT (Acc: 0.829)	TextFollower	61.3/31.1	15.0/17.0
BERT-ATTACK	60.2/34.8	25.6/34.4
SemAttack (+FT)	11.5/4.3	4.9/5.6
SemAttack (+FK)	17.0/7.0	11.2/13.1
SemAttack (+FC)	43.0/24.8	13.4/16.1
SemAttack (+all)	47.0/30.2	14.6/17.5
Self-Attention LSTM (Acc: 0.716)	TextFollower	19.1/10.6	10.3/10.6
BERT-ATTACK	42.9/31.5	19.4/23.0
SemAttack (+FT)	29.4/22.7	23.1/27.6
SemAttack (+FK)	23.2/15.8	20.7/23.0
SemAttack (+FC)	55.9/46.3	43.5/45.7
SemAttack (+all)	59.0/49.7	45.7/47.8
Table 10: The blackbox attack success rate (in terms of "USR/TSR") and corresponding word perturbation percentage against LSTM and BERT on the SNLI dataset by only perturbing premises.
F.6 Norm selection
In the main experiment, we use  $l_{2}$  norm for our attack loss function (equation 7). However, because  $l_{1}$  norm is known for good at feature selection and generating sparse features, we conduct the following experiments by setting  $l_{p}$  to  $l_{1}$  and make an comparison with  $l_{2}$  norm. The experimental results are shown in Table 11 and 12. We find the overall attack success rates decrease when switching to  $l_{1}$  norm. However, given the same set of constants  $c$  and  $\kappa$ , we find the  $l_{1}$  attack does change less words.
Dataset	Original Acc	c/k	SemAttack (l2untargeted)	SemAttack (l1untargeted)	Baseline (untargeted)
5/5	10/5	10/10	10/10	10/100	20/20
THUCTC	0.818	target	-	-	-	-	-	-	-
untarget	1.000	1.000	1.000	0.983	0.983	0.995	0.040
#/chars	1.583	1.690	1.718	1.577	1.614	1.884	2.000
Table 11: Untargeted attack success rates on Chinese BERT-based classifier for THUCTC dataset. "target" and "untarget" calculate the targeted attack success rate (equation 1) and the untargeted attack success rate (equation 2). "#/chars" counts the number characters are modified in average.
Dataset	Original Acc	c/k	SemAttack (l1targeted)	SemAttack (l2targeted)	Baseline (untargeted)
10/10	10/20	30/30	5/5	10/5	10/10
THUCTC	0.818	target	0.797	0.797	0.898	0.941	0.945	0.945	-
untarget	0.828	0.828	0.920	0.953	0.958	0.958	0.040
#/chars	2.000	1.956	3.280	2.924	3.186	3.045	2.000
Table 12: Targeted attack success rates on Chinese BERT-based classifier for THUCTC dataset. "target" and "untarget" calculate the targeted attack success rate (equation 1) and the untargeted attack success rate (equation 2). "#/chars" counts the number characters are modified in average.
F.7 Attack Strategy
As we have achieved  $100\%$  attack success rate in the untargeted attack scenario, we now focus on the targeted attack scenario and see which factor contributes to the targeted attack success rate. It is straightforward to think different targeted attack strategies will impact the targeted attack success rate, because maybe some classes look "farther" than semantic closer classes. So we tried two strategies on THUCTC dataset: 1) as used in the main paper, we set the targeted false class as "technology news". 2) we enumerate all the classes and set the targeted false class to be numerically the next class index. The targeted attack success rate is shown in Table 13. We do find choosing different attack strategies will impact the attack success rate.
F.8 Hyper Parameter Selection
We have two constants in our attack algorithm,  $c$  and  $\kappa$ , which control the attack success rates and the perturbation rates in our experiments. In order to find out the impact of these hyper parameters, we test with several combinations of different  $c$  and  $\kappa$ . We test on Yelp Dataset and we use BERT as our model. We show our results in Figure 5. As shown in Figure 5a, we first fix  $\kappa = 10$  and test how TSR and perturbation rate will change according to different  $c$ . We find that under the same  $\kappa$ ,  $c$  mainly controls the attack success rate at the cost of perturbation rate. In some certain range, a larger  $c$  encourages the algorithm to achieve our attack goal with the expense of more substitutions. And
(a) Fixed  $\kappa$  and different  $c$ .
(b) Fixed  $c$  and different  $\kappa$ .
Figure 5: Hyper parameter selection. In Figure 5a, we first fix  $\kappa = 10$  and test different  $c$  to see how TSR and perturbation rate will change. We test  $c = 1, 10, 10^2, 10^3, 10^4$  and find best  $c = 100$  to obtain the highest TSR with less perturbations. A smaller or larger  $c$  will result in a low TSR or a high perturbation rate. In Figure 5b, after fixing  $c = 100$ , we test  $\kappa = 0, 1, 5, 10, 15$ . We find that  $\kappa$  has little influence on TSR while it can change perturbation rate dramatically. A smaller  $\kappa$  is able to effectively limit the number of words to be changed. In our experiment, we choose  $\kappa = 0, 1$ .
after exceeding a certain value, TSR will start to decrease while perturbation rate remains high. We then fix  $c = 100$  and test different  $\kappa$ . We show our results in Figure 5b. We find that  $\kappa$  doesn't help to increase TSR and a smaller  $\kappa$  helps to limit the words changed without affecting TSR.
For hyper-parameter selection for Chinese datasets, we witness the same phenomenon in English attacks that increasing constant  $c$  can improve the attack success rate at the cost of more perturbed characters, while lowering constant  $\kappa$  limits the perturbation rate without affecting the attack success rate.
Dataset	Original Acc	SemAttack (targeted c/κ = 10/10)	Baseline (untargeted)
strategy 1	strategy 2
THUCTC	0.818	target	0.945	0.903	-
untarget	0.958	0.913	0.040
#/chars	3.045	4.543	2.000
Table 13: Attack success rates on Chinese BERT-based classifier for two datasets. "target" and "untarget" calculate the targeted attack success rate (equation 1) and the untargeted attack success rate (equation 2). "#/chars" counts the number characters are modified in average.
Transfer	Method	% TSR	% USR
Self-Attention LSTM → BERT	TextFooler	42.4	43.9
BERT-ATTACK	8.1	33.5
SemAttack (+FT)	44.4	32.5
SemAttack (+FK)	57.7	62.0
SemAttack (+FC)	74.3	81.2
SemAttack (+all)	70.0	79.8
BERT → Self-Attention LSTM	TextFooler	30.8	31.9
BERT-ATTACK	17.6	28.5
SemAttack (+FT)	26.8	34.6
SemAttack (+FK)	35.3	35.6
SemAttack (+FC)	35.5	36.0
SemAttack (+all)	30.9	31.0
Table 14: Targeted and untargeted attack success rate of transferability attack on Yelp Dataset, evaluating adversarial examples generated against Self-attention LSTMs on BERT, and vice versa.
F.9 Vulnerability Between Classes
In THUNews dataset, the article titles are classified into 14 categories. In order to find out the vulnerability of each class, we test the attack success rate of each source class and target class. The heatmap of results is provided in Figure 4b. We find that "technology news" and "entertainment news" as target classes have higher average success rates than other classes, while "lottery ticket" is the lowest. We also find that "constellation news" has the highest average success rate as source class, while "sports news" has the lowest, which means "constellation news" is vulnerable and easy to attack while "sports news" is much more robust.
F.10 Transferability Analysis
We evaluate the transferability of adversarial examples between different models by attacking a blackbox BERT classifier by using adversarial text generated from a whitebox LSTM, and vice versa.
The transferability-based attack results on Yelp Dataset are shown in Table 14. We find that the robustness of the two models is highly different from each other. When we feed adversarial texts generated from the LSTM model into the blackbox BERT model, attack success rate is higher than