SlowGuess/NAACL_2022_OCR huggingface dataset:
BLEU scores of StyleAdv, as the BLEU scores are extremely low (on 3 of 4 tasks the BLEU scores are at the scale of  $10^{-81}$ ).
- Perplexity (PPL): a pre-trained GPT2small (Radford et al., 2019) is used to calculate the PPL of adversarial texts, which reflects the fluency as suggested by (Kann et al., 2018; Zang et al., 2020a).
- Grammar error (GER): Following Zang et al., 2020b, we employ LanguageTool<sup>6</sup> to calculate the average number of grammar errors newly introduced by adversarial samples.
We only evaluate the last four metrics on the successful attacks against the victim model.
Table 2 summarizes the main experimental and comparison results. Overall, PLAT consistently achieves a better attack success rate and perplexity performance across all datasets. We attribute this to the flexible phrase-level perturbations generated using contextual information. PLAT achieves a significantly better text quality than the sentence-level model StyleAdv, in terms of modification rates, BLEU, and perplexity. Compared with a phrase-level model MAYA, our method is significantly better on modification rates, BLEU, and grammar scores. Hence, despite not using semantic similarity constraints, PLAT is more controllable than MAYA as we confine the modification ranges and generate perturbations by contextual blank-infilling. While word-level attacks naturally introduce fewer
Table 2: Adversarial attack performance of PLAT and baselines on four datasets, in terms of attack success rate (ASR), edit distance (DIS), BLEU, perplexity (PPL), and increased grammar errors (GER). Bold values indicate the best performance for each metric.  $\downarrow (\uparrow)$  indicates the higher (lower) the better. Note that phrase-level attacks naturally introduce more modifications than word-level attacks so they are not directly comparable on DIS and BLEU metrics in the table.
Metric	PLAT	equal	CLARE
Meaning preservation	39.8	\	33.3
Label preservation	77.1	\	49.8
Fluency and grammaticality	33.1	32.3	34.6
Metric	PLAT	equal	MAYA
Meaning preservation	39.8	\	30.2
Label preservation	77.1	\	53.5
Fluency and grammaticality	45.0	29.0	26.0
Table 3: Human evaluation performance in percentage on the Yelp dataset.
perturbations and thus have better textual similarity and grammaticality, its perturbation space is small and results in lower success rates. On the contrary, PLAT achieves the highest success rate while on a par with word-level attacks on textual similarity and grammaticality, hence achieving a sweet spot among all metrics.
Human evaluation. We further conduct human evaluations on Yelp dataset with 100 randomly selected successful attacks produced by PLAT, CLARE, and MAYA. We evaluate these attacks in three aspects: (1) Meaning preservation: whether the attacks preserve the original meaning or not; (2) Label preservation: whether the modifications contradict the original sentiment or not; (3) We evaluate fluency and grammaticality via pairwise comparison: for each instance, we pair PLAT's attack with one by CLARE or MAYA. The human annotators are asked to either select the better one or rate them as equal. We average 6 responses for each sample. More details are in Appendix D.
As shown in Table 3, PLAT significantly out
$^{6}$ https://www.languagetool.org/
Yelp (Negative)	The quality of the food has really plummeted over the past year. We use to love coming her to get the creamy clam chowder, not its watery and gross.
TextFooler (Positive)	The quality of the food has really engulfed over the past year. We use to love coming her to get the creamy clam chowder, not its watery and gross.
CLARE (Positive)	The quality of the food has really soared over the past year. We use to love coming her to get the creamy clam chowder, not its watery and gross.
MAYA (Positive)	The quality of the food has really last year was a big one for the fall. We use to love coming her to get the creamy clam chowder, not its watery and gross.
PLAT (Positive)	The quality of the food has also somewhat plummeted over the past year. We use to love coming her to get the creamy clam chowder, not its watery and gross.
Table 4: Adversarial examples generated by different models on Yelp dataset, perturbations are colored.
Yelp	AG News	MNLI	QNLLI
1st	NP / 39.9%	NP / 53.7%	NP / 57.6%	NP / 58.6%
2nd	ADJP / 17.5 %	NNP / 27.0%	PP / 12.4%	NNP / 16.4%
3rd	VP / 16.6%	PP / 9.5%	NNP / 27.1%	PP / 13.4%
performs CLARE and MAYA in terms of label consistency, i.e.,  $77.1\%$  vs.  $49.8\%$  (CLARE) or  $53.5\%$  (MAYA). This demonstrates the benefit of our whole architecture that utilizes both phrase-level infilling and sufficient constraints. It's worth noting that all models struggle on preserving the textual meaning and less than  $40\%$  of samples can retain their original meaning. This is consistent with Morris et al., 2020 in that semantic similarity metrics fail to maintain the actual meaning under most conditions. On the fluency and grammaticality, PLAT is comparable to CLARE but is much better than MAYA ( $45\%$  vs.  $26\%$ ), since our context-aware blank-infilling is superior to paraphrasing each text piece independently. Finally, Table 4 compares adversarial attacks crafted by our model and other baselines. More case studies are provided in Appendix B.
Vulnerable phrase types. We also analyze the three mostly attacked phrase types on each dataset. As shown in Table 5, noun phrases (NP) are the most vulnerable phrases over all datasets (more than  $50\%$  on three datasets), while preposition phrases (PP) and proper noun phrases (NNP) are also commonly vulnerable.
In this section, we conduct detailed analyses of PLAT, including ablation study (§4.1), discussion of controllability in blank infilling (§4.2), and attacking robust defense models (§4.3).
Table 5: Top-3 phrase tags of attack phrases and their percentages on different datasets by PLAT.
Module	ASR↑	DIS↓	BLEU↑	PPL↓	GER↓
PLAT	98.4	0.17	0.78	56.8	0.33
w/o PHRASE-LEVEL	97.7	0.09	0.84	92.8	0.68
w ALL CONSTITUENTS	98.2	0.16	0.79	58.1	0.29
BERTbase likelihood	98.5	0.17	0.78	56.8	0.30
T5base infilling	98.4	0.16	0.79	61.4	0.41
GPT-2small infilling	98.7	0.16	0.79	61.4	0.42
Table 6: Results of the ablation study on Yelp dataset.
We evaluate the effectiveness of each key component in PLAT based on the 1,000 random Yelp samples §3.2. We first study the phrase-level perturbation by replacing it with the word-level replacement used in Textfooler (w/o PHRASE-LEVEL). As Table 6 shows, phrase-level perturbation has a larger attack search space which leads to better attack success rate, perplexity and grammar score. It also demonstrates that attacking the constituents that are labeled as phrases is more effective than attacking all possible constituents. This is probably because phrases contain more critical and clear information to attack in classification tasks. In addition, we have not observed a significant difference between using BERT<sub>base</sub> and RoBERT<sub>base</sub> for class-conditioned likelihood calculation, probably due to their similar architectures and shared knowledge. Finally, we comparing different blank-infilling methods using pre-trained BART<sub>base</sub> (PLAT), pre-trained T5<sub>base</sub> (Raffel et al., 2020) and fine-tuned GPT-2<sub>small</sub> (Donahue et al., 2020). Empirically, BART<sub>base</sub> achieves the best overall performance. Moreover, we investigate the effectiveness of the label-preservation filter in §C.1
4.2 Sensitivity and Controllability Analysis
In this section, we investigate how the outputs can be controlled by hyper-parameters in PLAT on Yelp dataset. We first study how the generated samples can be impacted by varying the candidate number
Figure 2: ASR, GER, DIS and PPL results by controlling different candidates numbers  $N$  in PLAT.
Figure 3: DIS and BLEU results by varying depth restrictions  $d$  (upper) and length increments  $l$  (bottom).
$N$  in blank-infilling. As shown in Figure 2, the success rate (ASR) increases with the increase of  $N$  but starts to saturate when  $N \geq 500$ , while the grammar errors (GER) stay quite consistent. Meanwhile, the edit distance (DIS) drops significantly but the perplexity (PPL) only increases slightly when  $N$  increases. Based on these observations, we choose  $N = 5000$  in our experiments for the best trade-off among these aspects.
In addition, we explore how the syntactic tree depth  $d$  in target phrases and the incremental length  $l$  for perturbed phrases can affect the modification degree. In Figure 3, with the increase of  $d$  or  $l$ , more modifications are introduced and the textual similarity decreases, which undermines the validity of adversarial samples, as larger  $d$  and  $l$  may augment the attacking space with longer but unnecessary perturbations. Hence, we choose relatively small  $d$  and  $l$  to ensure the attack is more controllable. Note that both  $d$  and  $l$  exhibit a slight impact on the success rate (Refer to Appendix C for detailed results).
4.3 Attacking Robust Defense Model
In this section, we examine whether existing robust defense models can defend PLAT attack which is beyond word-level perturbations. We apply a robust  $\mathrm{BERT}_{\mathrm{base}}$  defense model trained via
Method	ASR↑	DIS↓	BLEU↑	PPL↓	GER↓
TextFooler	64.2	0.23	0.59	185.5	1.39
CLARE	92.5	0.12	0.81	76.1	0.15
MAYA	98.6	0.57	0.33	82.1	3.51
PLAT	99.2	0.28	0.67	55.7	0.39
Table 7: Results of PLAT and baselines attacking the robust defense model TAVAT on Yelp dataset.
TAVAT (Li and Qiu, 2021) to defend the attacks from PLAT and baseline models, which is designed for word-level attacks. Comparing Table 7 with Table 2, both the edit distance and BLEU get worse when attacking the defense model, showing that the defense model is harder to attack. Meanwhile, two word-level attacks have a significant attack success rate drop, e.g.,  $94.5\%$  to  $64.2\%$  on TextFooler. On the contrary, PLAT still can achieve the best  $99.2\%$  attack success rate with sufficient textual similarity and grammar errors, outperforming MAYA in every aspect. This suggests that PLAT raises a new robustness issue on current defense models.
Textual Adversarial Attack Growing interest is devoted to generating textual adversarial samples via perturbation at various levels. Some early works use misspelling tokens in character-level (Liang et al., 2018; Ebrahimi et al., 2018; Li et al., 2019), but they can be easily defended by spell checking tools (Pruthi et al., 2019; Zhou et al., 2019; Jones et al., 2020). Recent mainstream of studies try to misguide models via word-level perturbations, e.g., synonym/semantic neighbor substitution (Alzantot et al., 2018; Jin et al., 2020; Ren et al., 2019; Zhang et al., 2019), replacement by pre-trained masked language models (Li et al., 2020; Zhang et al., 2019), or combing replacement with operations like insertion and merging (Li et al., 2021). These methods usually attempt to preserve the semantic similarity for better fluency and grammaticality, but their perturbations are limited to independent single words.
Sentence-level attacks have also been studied to generate new texts via paraphrasing or GAN-based generation (Iyyer et al., 2018; Wang et al., 2020b; Zou et al., 2020; Wang et al., 2020a; Qi et al., 2021), but their drastic modifications on the text structure make it harder to maintain a satisfactory textual quality. Very recently, phrase-level perturbations are considered in evaluating the performance of syntactic parsing (Zheng et al., 2020),
or involved in a multi-granularity textual attack model MAYA (Chen et al., 2021). Unlike MAYA, PLAT only focuses on unified phrase-level attacks without multiple sub-modules, which require simpler and fewer modifications while benefiting better performance.
Blank Infilling Large-scale pre-trained language models such as BERT (Devlin et al., 2019) and RoBERTa (Liu et al., 2019) have shown their capability of filling masked single tokens (Wang and Cho, 2019; Ghazvininejad et al., 2019) but they cannot handle variable-length masks. Although autoregressive generative models such as GPT (Radford et al., 2018) or GPT2 (Radford et al., 2019) can produce outputs with arbitrary lengths, they only condition information from a single direction and ignore some surrounding texts. To enable GPT models to fill in blanks, Donahue et al. proposed to fine-tune them with sequences concatenating manually-masked texts and missing texts. Recently, autoencoder-decoder models such as T5 (Raffel et al., 2020) and BART (Lewis et al., 2020), which are trained using text infilling losses make it possible to fill the blanks within the context in a more flexible form (Shen et al., 2020).
We present a new phrase-level textual adversarial attack, PLAT, which produces richer and higher-quality phrase-level perturbations than the widely studied word-level attacks. It utilizes contextualized blank-infilling to generate perturbations by a pre-trained language model and thus well preserves the fluency, and grammaticality. We additionally develop a label-preservation filter based on the likelihood given by class-conditioned language models, trying to keep the ground-truth labels intact. Extensive experiments show the effectiveness of PLAT and its advantages over baselines on different NLP tasks.
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of EMNLP 2018, pages 2890-2896.
Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In Proceedings of ICLR 2018.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al. 2018. Universal sentence encoder. arXiv preprint arXiv:1803.11175.
Yangyi Chen, Jin Su, and Wei Wei. 2021. Multi-granularity textual adversarial attack with behavior cloning. In Proceedings of EMNLP 2021, pages 4511-4526.
Yong Cheng, Lu Jiang, and Wolfgang Macherey. 2019. Robust neural machine translation with doubly adversarial inputs. In Proceedings of of ACL 2019.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT 2019, pages 4171-4186.
Chris Donahue, Mina Lee, and Percy Liang. 2020. *Enabling language models to fill in the blanks*. In *Proceedings of ACL* 2020, pages 2492-2501.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. Hotflip: White-box adversarial examples for text classification. In Proceedings of ACL 2018, pages 31-36.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical neural story generation. In Proceedings of ACL 2018, pages 889-898.
Siddhant Garg and Goutham Ramakrishnan. 2020. BAE: BERT-based adversarial examples for text classification. In Proceedings of EMNLP 2020, pages 6174-6181.
Marjan Ghazvininejad, Omer Levy, Yinhan Liu, and Luke Zettlemoyer. 2019. Mask-predict: Parallel decoding of conditional masked language models. In Proceedings of EMNLP-IJCNLP 2019, pages 6112-6121.
Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of NAACL-HLT 2018, pages 1875-1885.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of EMNLP 2017, pages 2021-2031.
Robin Jia, Aditi Raghunathan, Kerem Goksel, and Percy Liang. 2019. Certified robustness to adversarial word substitutions. In Proceedings of EMNLP-IJCNLP 2019, pages 4129-4142.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI 2020, pages 8018-8025.
Erik Jones, Robin Jia, Aditi Raghunathan, and Percy Liang. 2020. Robust encodings: A framework for combating adversarial typos. In Proceedings of ACL 2020, pages 2752-2765.
Katharina Kann, Sascha Rothe, and Katja Filippova. 2018. Sentence-level fluency evaluation: References help, but can be spared! In Proceedings of CoNLL 2018, pages 313-323.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. In Proceedings of ACL 2020, pages 7871-7880.
Dianqi Li, Yizhe Zhang, Hao Peng, Liquin Chen, Chris Brockett, Ming-Ting Sun, and William B Dolan. 2021. Contextualized perturbation for textual adversarial attack. In Proceedings of NAACL-HLT 2021, pages 5053-5069.
J Li, S Ji, T Du, B Li, and T Wang. 2019. Textbugger: Generating adversarial text against real-world applications. In Proceedings of NDSS 2019.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. BERT-ATTACK: Adversarial attack against BERT using BERT. In Proceedings of EMNLP 2020, pages 6193–6202.
Linyang Li and Xipeng Qiu. 2021. Token-aware virtual adversarial training in natural language understanding. In Proceedings of AAAI 2021, pages 8410-8418.
Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi. 2018. Deep text classification can be fooled. In Proceedings of IJCAI 2018, pages 4208-4215.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Eric Malmi, Aliaksei Severyn, and Sascha Rothe. 2020. Unsupervised text style transfer with masked language models. In Proceedings of EMNLP 2020, pages 8671-8680.
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In Proceedings of 52nd Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 55-60, Baltimore, Maryland. Association for Computational Linguistics.
John Morris, Eli Lifland, Jack Lanchantin, Yangfeng Ji, and Yanjun Qi. 2020. Reevaluating adversarial examples in natural language. In Proceedings of EMNLP-Findings 2020, pages 3829-3839.
Nikola Mrkšić, Diarmuid Řeaghdha, Blaise Thomson, Milica Gasic, Lina M Rojas Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In Proceedings of NAACL-HLT 2016, pages 142-148.
Gonzalo Navarro. 2001. A guided tour to approximate string matching. ACM computing surveys (CSUR), 33(1):31-88.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of ACL 2002, pages 311-318.
Danish Pruthi, Bhuwan Dhingra, and Zachary C Lipton. 2019. Combating adversarial misspellings with robust word recognition. In Proceedings of ACL 2019, pages 5582-5591.
Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. 2021. Mind the style of text! adversarial and backdoor attacks based on text style transfer. In Proceedings of EMNLP 2021, pages 4569-4580.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. Open AI Blog.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. Open AI Blog.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of EMNLP 2016, pages 2383-2392.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using siamese BERT-networks. In Proceedings of EMNLP-IJCNLP 2019, pages 3982-3992.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of ACL 2019, pages 1085-1097.
Tianxiao Shen, Victor Quach, Regina Barzilay, and Tommi Jaakkola. 2020. Blank language models. In Proceedings of EMNLP 2020, pages 5186-5198.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. In Proceedings of EMNLP-IJCNLP 2019, pages 2153-2162.
Alex Wang and Kyunghyun Cho. 2019. BERT has a mouth, and it must speak: BERT as a markov random field language model. In Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation, pages 30-36.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. In Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 353-355.
Boxin Wang, Hengzhi Pei, Boyuan Pan, Qian Chen, Shuohang Wang, and Bo Li. 2020a. T3: Treeautoencoder regularized adversarial text generation for targeted attack. In Proceedings of EMNLP 2020, pages 6134-6150.
Tianlu Wang, Xuezhi Wang, Yao Qin, Ben Packer, Kang Li, Jilin Chen, Alex Beutel, and Ed Chi. 2020b. CATgen: Improving robustness in nlp models via controlled adversarial text generation. In Proceedings of EMNLP 2020, pages 5141-5146.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of NAACL-HLT 2018, pages 1112-1122.
Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020a. Word-level textual adversarial attacking as combinatorial optimization. Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020b. Word-level textual adversarial attacking as combinatorial optimization. In Proceedings of ACL 2020, pages 6066-6080.
Huangzhao Zhang, Hao Zhou, Ning Miao, and Lei Li. 2019. Generating fluent adversarial examples for natural languages. In Proceedings of ACL 2019, pages 5564-5569.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. NIPS 2015.
Xiaqing Zheng, Jiehang Zeng, Yi Zhou, Cho-Jui Hsieh, Minhao Cheng, and Xuan-Jing Huang. 2020. Evaluating and enhancing the robustness of neural network-based dependency parsing models with adversarial examples. In Proceedings of ACL 2020, pages 6600-6610.
Yichao Zhou, Jyun-Yu Jiang, Kai-Wei Chang, and Wei Wang. 2019. Learning to discriminate perturbations for blocking adversarial attacks in text classification. In Proceedings of EMNLP-IJCNLP 2019, pages 4904-4913.
Wei Zou, Shujian Huang, Jun Xie, Xinyu Dai, and Jiajun Chen. 2020. A reinforced generation of adversarial examples for neural machine translation. In Proceedings of ACL 2020, pages 3486-3497.
A.1 Details of PLAT
Basic infrastructure We use PyTorch as the backbone of our implementation, along with Huggingface-Transformers<sup>7</sup> for the implementation of victim models and likelihood estimation models, while Fairseq for the implementation of blank-infilling model BART<sup>8</sup>. We list the hyperparameters used in our model in Table 8, all of them are determined empirically based on both attack success rate and textual quality. It takes about 160 minutes to generate 100 adversarial samples on Yelp dataset using PLAT on a single NVIDIA GTX 1080 Ti GPU.
Select phrase candidates. We use the parser from Stanford CoreNLP<sup>9</sup> toolkit for the syntactic tree parsing. We consider parsed nodes with the syntactic tags in Table 9 as the possible root node of a phrase.
Depth restriction of phrase syntactic tree d	4
Length incremental restriction for substitutions l	3
Maximum perturbation number T	11
Likelihood ratio filter threshold δ	1
Substitution candidates number N	5000
Table 8: All hyperparameters used in PLAT.
Tags	ADJP, ADVP, CONJP, NP, NNP, PP, QP, VP, WHADJP, WHADVP, WHNP, WHVP