SlowGuess/ACL_2020_OCR huggingface dataset:
Victim Model	Word Substitution Method	Search Algorithm	IMDB	SST-2	SNLI
%M	%I	PPL	%M	%I	PPL	%M	%I	PPL
BiLSTM	Embedding/LM	Genetic	9.76	5.49	124.20	12.03	7.08	319.98	13.31	14.12	235.20
Greedy	7.84	5.23	112.84	10.63	3.71	287.45	12.60	9.42	205.50
PSO	7.00	8.07	113.99	12.78	6.68	339.46	14.82	10.82	255.69
Synonym	Genetic	7.60	6.07	137.51	11.35	5.32	357.19	12.60	24.78	283.95
Greedy	6.47	4.49	115.31	10.25	4.65	317.27	12.32	21.37	311.04
PSO	5.42	3.45	109.27	10.55	5.12	331.96	12.56	20.83	307.51
Sememe	Genetic	5.30	2.55	105.24	10.04	3.48	298.49	11.30	11.64	205.61
Greedy	4.89	1.80	97.49	9.36	2.79	276.53	12.11	10.95	218.72
PSO	3.71	1.44	88.98	9.06	3.17	276.53	11.72	11.08	222.40
BERT	Embedding/LM	Genetic	7.41	4.22	106.12	10.41	5.09	314.22	13.04	15.09	225.92
Greedy	5.53	4.45	97.21	9.23	3.04	276.42	11.80	13.73	206.46
PSO	5.97	7.98	101.66	11.64	6.70	343.89	14.22	14.43	245.95
Synonym	Genetic	5.72	5.59	114.57	9.62	4.62	353.05	13.09	13.01	311.14
Greedy	4.49	4.48	98.60	8.51	4.12	316.30	11.60	11.65	285.00
PSO	4.63	4.33	100.81	9.20	4.72	337.82	12.99	13.32	302.83
Sememe	Genetic	4.27	1.62	97.86	8.34	2.05	292.16	11.59	8.84	217.75
Greedy	3.97	1.79	92.31	8.14	2.21	279.35	10.09	7.81	207.71
PSO	3.69	1.57	90.74	8.24	2.03	289.94	11.73	10.14	223.22
Table 10: Automatic evaluation results of adversarial example quality. “%M”, “%I” and “PPL” indicate the modification rate, grammatical error increase rate and language model perplexity respectively.
sentences. For each adversarial example, we use the average of the naturality scores given by three workers as its final naturality score.
B Automatic Evaluation Results of Adversarial Example Quality
We present the automatic evaluation results of adversarial example quality of all the combination models in Table 10. We can find that Sememe and PSO obtain higher overall adversarial example quality than other word substitution methods and adversarial example search algorithms, respectively.
C Adjustment of Hyper-parameters of Embedding/LM+Genetic
The word substitution strategy Embedding/LM has three hyper-parameters: the number of the nearest words N, the euclidean distance threshold of word embeddings  $\delta$  and the number of words retained by the language model filtering K. For original Embedding/LM+Genetic,  $N = 8$ ,  $\delta = 0.5$  and  $K = 4$ , which are the same as Alzantot et al. (2018). To increase the attack success rates, we change these hyper-parameters to  $N = 20$ ,  $\delta = 1$  and  $K = 10$ .
D Statistical Significance of Human Evaluation Results
We conduct Student's  $t$ -tests to measure the statistical significance between the difference of human evaluation results of different models. The
results of attack validity and adversarial example naturality are shown in Table 11 and 12, respectively. "Embedding/LM+Genetic*" refers to the Embedding/LM+Genetic model with adjusted hyper-parameters.
We display some adversarial examples generated by the baseline attack models and our attack model on IMDB, SST-2 and SNLI in Table 13, 14 and 15 respectively.
6077
Victim Model	Model 1	Model 2	p-value	Significance	Conclusion
Bi-LSTM	Sememe+PSO	Embedding/LM+Genetic	0.14	Not Significant	=
Sememe+PSO	Synonym+Greedy	0.37	Not Significant	=
Sememe+PSO	Embedding/LM+Genetic*	0.01	Significant	>
Original Input	Embedding/LM+Genetic	9.52e-10	Significant	>
Original Input	Synonym+Greedy	1.78e-6	Significant	>
Original Input	Sememe+PSO	3.55e-7	Significant	>
Original Input	Embedding/LM+Genetic*	2.42e-13	Significant	>
BERT	Sememe+PSO	Embedding/LM+Genetic	0.29	Not Significant	=
Sememe+PSO	Synonym+Greedy	0.12	Not Significant	=
Sememe+PSO	Embedding/LM+Genetic*	5.86e-4	Significant	>
Original Input	Embedding/LM+Genetic	2.19e-5	Significant	>
Original Input	Synonym+Greedy	3.33e-9	Significant	>
Original Input	Sememe+PSO	1.78e-6	Significant	>
Original Input	Embedding/LM+Genetic*	2.30e-15	Significant	>
Table 11: The Student's  $t$ -test results of attack validity of different models, where “ $=$ ” means “Model 1” performs as well as “Model 2” and “ $>$ ” means “Model 1” performs better than “Model 2”.
Victim Model	Model 1	Model 2	p-value	Significance	Conclusion
Bi-LSTM	Sememe+PSO	Embedding/LM+Genetic	0.48	Not Significant	=
Sememe+PSO	Synonym+Greedy	0.41	Not Significant	=
Human Authored	Embedding/LM+Genetic	0.14	Not Significant	=
Human Authored	Synonym+Greedy	0.10	Not Significant	=
Human Authored	Sememe+PSO	0.15	Not Significant	=
BERT	Sememe+PSO	Embedding/LM+Genetic	0.31	Not Significant	=
Sememe+PSO	Synonym+Greedy	0.31	Not Significant	=
Human Authored	Embedding/LM+Genetic	0.06	Not Significant	=
Human Authored	Synonym+Greedy	0.06	Not Significant	=
Human Authored	Sememe+PSO	0.08	Not Significant	=
Table 12: The Student's  $t$ -test results of adversarial example naturality of different models, where “=” means "Model 1" performs as well as "Model 2".
6078
IMDB Example 1
Original Input (Prediction = Positive)
In my opinion this is the best oliver stone flick probably more because of influence than anything else. Full of dread from the first moment to its dark ending.
Embedding/LM+Genetic (Prediction = Negative)
In my view this is the higher oliver stone flick presumably more because of influence than anything else. Total of anxiety from the first moment to its dark ending.
Synonym+Greedy (Prediction = Negative)
In my opinion this embody the respectable oliver stone flick probably more because of influence than anything else. Broad of dread from the first moment to its dark ending.
Sememe+PSO (Prediction = Negative)
In my opinion this is the bestest oliver stone flick probably more because of influence than anything else. Amplitude of dread from the first moment to its dark ending.
IMDB Example 2
Original Input (Prediction = Negative)
One of the worst films of it's genre. The only bright spots were lee showing some of the sparkle she would later bring to the time tunnel and batman.
Embedding/LM+Genetic (Prediction = Positive)
One of the biggest films of it's genre. The only glittering spots were lee showing some of the sparkle she would afterwards bring to the time tunnel and batman.
Synonym+Greedy (Prediction = Positive)
One of the tough films of it's genre. The only bright spots follow lee present some of the spark she would later bring to the time tunnel and batman.
Sememe+PSO (Prediction = Positive)
One of the seediest films of it's genre. The only shimmering spots were lee showing some of the sparkle she would later bring to the time tunnel and batman.
Table 13: Adversarial examples generated by two baseline methods and our model on IMDB.
6079
SST-2 Example 1
Original Input (Prediction = Positive)
Some actors have so much charisma that you’d be happy to listen to them reading the phone book.
Embedding/LM+Genetic (Prediction = Negative)
Some actors have so much charisma that you’d be cheery to listen to them reading the phone book.
Synonym+Greedy (Prediction = Negative)
Some actors have so much charisma that you’d be happy to listen to them take the phone book.
Sememe+PSO (Prediction = Negative)
Some actors have so much charisma that you’d be jovial to listen to them reading the phone book.
SST-2 Example 2
Original Sentence (Prediction = Negative)
The movie’s biggest is its complete and utter lack of tension.
Embedding/LM+Genetic (Prediction = Positive)
The movie’s biggest is its complete and utter absence of stress.
Synonym+Greedy (Prediction = Positive)
The movie’s great is its complete and utter want of tension.
Sememe+PSO (Prediction = Positive)
The movie’s biggest is its complete and utter dearth of tension.
Table 14: Adversarial examples generated by two baseline methods and our model on SST-2.
SNLI Example 1
Premise: A smiling bride sits in a swing with her smiling groom standing behind her posing for the male photographer while a boy holding a bottled drink and another boy wearing a green shirt observe.
Original Input(Prediction = Entailment)
Two boys look on as a married couple get their pictures taken.
Embedding/LM+Genetic (Prediction = Contradiction)
Two man stare on as a wedding couple get their pictures taken.
Synonym+Greedy (Prediction = Contradiction)
Two boys look on as a married couple puzzle their pictures taken.
Sememe+PSO (Prediction = Contradiction)
Two boys stare on as a wedding couple get their pictures taken.
SNLI Example 2
Premise: A dog with a purple leash is held by a woman wearing white shoes.
Original Input (Prediction = Entailment)
A man is holding a leash on someone else dog.
Embedding/LM+Genetic (Prediction = Contradiction)
A man is holding a leash on someone further dog.
Synonym+Greedy (Prediction = Contradiction)
A humans is holding a leash on someone else dog.
Sememe+PSO (Prediction = Contradiction)
A man is holding a leash on someone else canine.
Table 15: Adversarial examples generated by two baseline methods and our model on SNLI.