SlowGuess/NAACL_2022_OCR huggingface dataset:
4The results of the SGCP baseline reported here are lower than in the original paper. This is because that original paper used very fine-grained syntactic information, which contains more hints about reference paraphrase. Additionally, compared with syntactic templates, fine-grained syntactic structures are more conducive to guiding models to generate natural sentences.
<sup>2</sup>Due to the different control types, we don't compared with the example-based methods (Chen et al., 2019; Liu et al., 2020)
<sup>3</sup>https://github.com/mallabiisc/SGCP
(a) Syntactic template
(b) Syntax Attention with Regularization
(c) Syntax Attention without Regularization
Figure 3: (a) shows the used syntactic template. (b) and (c) show visualization results of syntax attention with or without regularization.
Table 2 shows the results of human evaluation which are consistent with automatic evaluation results. Our model obtains the highest scores in both semantics and syntax, thereby highlighting the efficacy of our method.
Table 6 shows examples of paraphrases generated by different model. We can observe that our SI-SCP can produce better results than baseline models in terms of both semantics and syntax.
5.2.5 Visualization of Syntax Attention
We visualize the syntax attention when using the syntactic template as control in Figure 3. This is an instance from the test dataset: {Source: it looks to me like they are really sweet boys. Reference: they look like very nice boys, i think}. Figure 3 (a)
Dataset	Model	Top-1	Top-5	Top-10	Top-20
ParaNMT	CT (2018)	2.5	6.7	11.3	16.7
-Small	STR	8.6	20.5	27.3	34.4
QQP	CT (2018)	9.2	40.2	47.9	58.8
-Pos	STR	19.1	48.7	61.1	71.1
Table 3: Retrieval accuracy  $(\%)$ . 'CT': common template, 'STR': our syntactic template retriever.
Model	B-1 / B-2 / B-3 / B-4↑	Re(%)↓	SIf-B↓	Valid(%)↑
ParaNMT-Small
Seq2Seq	55.8 / 39.1 / 28.8 / 21.9	30.0	69.2	56.7
(1) CT & SCPN	48.1 / 31.0 / 20.6 / 14.0	70.7	34.4	15.1
(2) CT & GuiG	51.0 / 34.4 / 23.9 / 17.2	56.5	30.4	33.8
(3) CT & SI-SCP	51.8 / 35.7 / 25.3 / 18.3	64.1	28.7	36.8
(4) STR & SCPN	55.1 / 37.7 / 26.2 / 18.6	46.0	32.5	24.5
(5) STR & GuiG	56.2 / 38.6 / 27.2 / 19.5	39.7	32.0	55.7
(6) STR & SI-SCP	57.6 / 40.6 / 29.5 / 21.7	42.1	32.2	65.7
QQP-Pos
Seq2Seq	53.0 / 39.6 / 30.9 / 24.9	9.4	89.3	28.5
(7) CT & SCPN	60.4 / 43.7 / 32.3 / 24.4	40.2	28.2	52.9
(8) CT & SI-SCP	68.1 / 54.7 / 45.3 / 38.4	25.1	27.0	62.2
(9) STR & SCPN	62.7 / 46.7 / 35.6 / 27.6	28.1	28.1	63.5
(10) STR & SI-SCP	71.8 / 59.4 / 50.5 / 43.9	14.4	31.5	67.6
Table 4: Diverse paraphrase generation results. STR&SI-SCP denotes that SI-SCP uses 10 templates retrieved by the STR to generate 10 paraphrases. CT denotes using 10 common templates. B-1: BLEU-1. B-2: BLEU-2. B-3: BLEU-3. B-4: BLEU-4. Re: Rejection. Slf-B: Self-BLEU.
is the syntactic tempalte used in the example.
We can see that most words can accurately align with corresponding nodes when using attention regularization. Without attention regularization, most words tend to pay attention to the punctuation. These results show that the proposed attention regularization can make the decoder accurately select corresponding nodes to guide the generation of words.
5.3 Diverse Paraphrase Generation
In this section, we further evaluated our model's ability to generate diverse paraphrases. We first examine whether STR can retrieve compatible templates. We generated 10 syntactically different paraphrases for each input sentence using 10 retrieved syntactic templates. Implementation details are presented in Appendix B.
5.3.1 Evaluation Metrics
We evaluated this task with the following metrics:
1. Retrieval Accuracy: We use Top-K retrieval accuracy on the test set, measured as the percentage of Top-K retrieved templates that contain the gold template. The gold template is
Template	Paraphrase
Source	you can choose between a movie or a pottery class.
(S (NP (PRP)) (VP (MD) (VP)) (.))	you can choose from the film or the pottery class.
(S (NP (NP) (CC) (NP)) (VP (MD) (VP)) (.))	the film or the pottery class can be selected.
(S (NP (NP) (.)(CC) (NP)) (VP (MD) (VP)) (.))	the film, or the pottery class can be selected.
(SQ (MD) (NP (PRP)) (VP (VP) (CC) (VP)) (.))	can you pick one of the film or the pottery class?
(S (NP (PRP)) (VP (VBZ) (UCP)) (.))	it 's possible to choose between film or the pottery class.
(S (NP (RB) (DT) (JJ) (NN)) (VP (MD) (VP)) (.))	perhaps a pottery class can choose between the movies and the film.
Table 5: Syntactic paraphrases generated by SI-SCP with retrieved templates. We show several successful and one failed (in blue) generations.
obtained from the reference sentence. This metric is used to evaluate the performance of the template retriever.
2. Semantics: Given 10 generated paraphrases  $Y = \{y^{1}, y^{2}, \dots, y^{10}\}$  for each input sentence in the test set. For 10 paraphrases, the one with the highest BLEU score to the reference sentence is selected as the final generation  $y^{best}$ . The BLEU score between  $(y^{best}, y)$  is calculated at the corpus level.
3. Rejection Rate: We use Sentence-BERT  $^{5}$  (Reimers and Gurevych, 2019) to compute paraphrase scores for generated outputs with respect to the input. And then use this score to filter out low-quality paraphrases. The percentage of filtered sentences is taken as a rejection rate.
4. Diversity: We compute BLEU between all pairs  $(y^{i},y^{j})$ , then macro-average these values at the corpus-level.
5. Validity (Valid): To measure paraphrase quality, we perform human evaluation on 100 randomly selected paraphrases from the remaining paraphrases. Three annotators evaluate whether the generated sentences are true paraphrases, (the paraphrase is marked with 1, otherwise marked with 0). Then we compute the percentage of paraphrases marked as 1.
As can be observed in Table 3, our STR significantly surpasses CT in retrieval accuracy. These
results show that STR is capable of retrieving compatible syntactic templates.
In Table 4, we also show the results of the vanilla Seq2Seq based on transformer, where we use top-K  $(\mathrm{K} = 50)$  sampling to generate 10 paraphrases. Because this method tends to generate repeated sentences, it obtains lower valid scores on the QQP-Pos dataset. We see that syntax-controlled paraphrasing method significantly improve the diversity of generations.
Among different syntax-controlled models, compared with CT, STR significantly improves semantics, rejection rate, and validity metrics (Row 1/2/3/7/8 vs. Row 4/5/6/9/10). These results validate the advantages of the syntactic template retriever from the perspective of practical application. Using the same syntactic templates, GuiG can get a better rejection rate, but SI-SCP obtains better performance in terms of semantics, validity metrics. These results are consistent with automatic evaluation results in Table 1.
5.3.3 Case Study
Table 5 lists some paraphrases generated by SI-SCP with different syntactic templates. More generation results are presented in Appendix C. We see that the generated sentences always conform to the target templates. These examples are well-formed, semantically sensible, and grammatically correct sentences that also preserve semantics of the original sentences. However, our model also produces sentences with semantic deviation, like the failed cases in Table 5, when given template is incompatible with the input sentence.
We have presented a Structural Information-augmented Syntax-Controlled Paraphrasing (S-SCP) model which can directly generate syntactic paraphrases with syntactic templates. Particularly,
We used the paraphrase-distilroberta-base-v1, which is trained on large-scale paraphrase data. Available at: https://public.ukp.informatik.tu-darmstadt.de/reimers/sentence-transformers/v0.2/
Similar to Iyyer et al. (2018), we set minimum paraphrase similarity to 0.7.
we propose a tree-transformer and an attention regularization. The tree transformer can model parent-child and sibling relation of the syntactic template. The attention regularization method makes the decoder accurately select corresponding syntax nodes to guide the generation of words. To retrieve compatible syntactic templates in practice, we further propose a Syntactic Template Retriever (STR). Experiments show that SI-SCP achieves substantial improvements over previous strong baselines. Furthermore, we also validate that STR is capable of retrieving compatible syntactic templates. SI-SCP can produce more syntactically paraphrases with retrieved syntactic templates.
The present research was supported by the National Nature Science Foundation of China (No. 61876198, 61976015, 61976016). Deyi Xiong was supported by the Natural Science Foundation of Tianjin (Grant No. 19JCZDJC31400). We would like thank the four anonymous reviewers for their constructive suggestions and insightful comments.
Satanjeev Banerjee and Alon Lavie. 2005. ME-TEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019. Controllable paraphrase generation with a syntactic exemplar. In ACL, pages 5972-5984, Florence, Italy. Association for Computational Linguistics.
Ning Dai, Jianze Liang, Xipeng Qiu, and Xuanjing Huang. 2019. Style transformer: Unpaired text style transfer without disentangled latent representation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5997-6007, Florence, Italy. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Wee Chung Gan and Hwee Tou Ng. 2019. Improving the robustness of question answering systems to question paraphrasing. In ACL, pages 6065-6075, Florence, Italy. Association for Computational Linguistics.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Zhiting Hu, Zichao Yang, Xiaodan Liang, Ruslan Salakhutdinov, and Eric P. Xing. 2017. Toward controlled generation of text. In Proceedings of the 34th International Conference on Machine Learning - Volume 70, ICML'17, page 1587-1596. JMLR.org.
Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In NAACL, pages 1875-1885. Association for Computational Linguistics.
Vineet John, Lili Mou, Hareesh Bahuleyan, and Olga Vechtomova. 2019. Disentangled representation learning for non-parallel text style transfer. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 424-434, Florence, Italy. Association for Computational Linguistics.
Diederik Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. Computer Science.
Ashutosh Kumar, Kabir Ahuja, Raghuram Vadapalli, and Partha Talukdar. 2020. Syntax-guided controlled generation of paraphrases. TACL, 8:329-345.
Dongkyu Lee, Zhiliang Tian, Lanqing Xue, and Nevin L. Zhang. 2021. Enhancing content preservation in text style transfer using reverse attention and conditional layer normalization. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 93-102, Online. Association for Computational Linguistics.
Yinghao Li, Rui Feng, Isaac Rehg, and Chao Zhang. 2020. Transformer-based neural text generation with syntactic guidance.
Zichao Li, Xin Jiang, Lifeng Shang, and Hang Li. 2018. Paraphrase generation with deep reinforcement learning. In EMNLP, pages 3865-3878, Brussels, Belgium. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Mingtong Liu, Erguang Yang, Deyi Xiong, Yujie Zhang, Chen Sheng, Changjian Hu, Jinan Xu, and Yufeng Chen. 2020. Exploring bilingual parallel corpora
for syntactically controllable paraphrase generation. In IJCAI-20, pages 3955–3961. International Joint Conferences on Artificial Intelligence Organization. Main track.
Christopher Manning, Mihai Surdeanu, John Bauer, Jenny Finkel, Steven Bethard, and David McClosky. 2014. The Stanford CoreNLP natural language processing toolkit. In ACL, pages 55-60, Baltimore, Maryland. Association for Computational Linguistics.
Tong Niu and Mohit Bansal. 2018. Polite Dialogue Generation Without Parallel Data. Transactions of the Association for Computational Linguistics, 6:373-389.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: A method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting on Association for Computational Linguistics, ACL '02, page 311-318, USA. Association for Computational Linguistics.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543, Doha, Qatar. Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. SentenceBERT: Sentence embeddings using Siamese BERT-networks. In EMNLP, Hong Kong, China. Association for Computational Linguistics.
Jiao Sun, Xuezhe Ma, and Nanyun Peng. 2021. AESOP: Paraphrase generation with adaptive syntactic control. In EMNLP, pages 5176-5189, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
John Wieting and Kevin Gimpel. 2018. ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In ACL, pages 451-462, Melbourne, Australia. Association for Computational Linguistics.
Erguang Yang, Mingtong Liu, Deyi Xiong, Yujie Zhang, Yao Meng, Changjian Hu, Jinan Xu, and Yufeng Chen. 2021. Syntactically-informed unsupervised paraphrasing with non-parallel data. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2594–2604, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Xiaojing Yu and Anxiao Jiang. 2021. Expanding, retrieving and infilling: Diversifying cross-domain question generation with flexible templates. In EACL, pages 3202-3212, Online. Association for Computational Linguistics.
Sanqiang Zhao, Rui Meng, Daqing He, Andi Saptono, and Bambang Parmanto. 2018. Integrating transformer and paraphrase rules for sentence simplification. In EMNLP, pages 3164-3173, Brussels, Belgium. Association for Computational Linguistics.
Zhong Zhou, Matthias Sperber, and Alexander Waibel. 2019. Paraphrases as foreign languages in multilingual neural machine translation. In ACL, pages 113-122, Florence, Italy. Association for Computational Linguistics.
A Controlled Paraphrase Generation
We parsed all sentences in the training set, reference sentences in the validation and test set using Stanford CoreNLP (Manning et al., 2014). We used the scheduled Adam optimizer (Kingma and Ba, 2014) for optimization, and the learning rate was set to 2.0 for all experiments. We set hidden state size to 256 (i.e.,  $d$ ), filter size to 1024, head number to 4. The number of layers of the sentence encoder, sentence decoder, tree transformer and sequence encoder were set to 4, 4, 3, and 2, respectively. The batch size was set to 128.  $\lambda_{1}$  was set to 5.0 while  $\lambda_{2}$  1.0. We used BPE tokens pre-trained with 30,000 iterations. All hyperparameter tuning was based on the BLEU score on the development set.
For the SynTrans baseline model, we set the number of syntactic encoder layers to 5 for fair comparison.
B Diverse Paraphrase Generation
For the syntactic template retriever, we used 300 as hidden size, 512 as filter size and 4 heads in multihead attention. The number of layers of sentence, syntax, and template encoders were all set to 4. The batch size was set to 512. We used the scheduled Adam optimizer (Kingma and Ba, 2014) for optimization, and the learning rate was set to 0.1. The word embedding layer was initialized by the publicly available GloVe (Pennington et al., 2014) 300-dimensional embeddings. $^7$
C Generated Paraphrase Examples
Table 6 shows several paraphrases generated by each model. Table 7 lists some paraphrases generated by SI-SCP with different syntactic templates. Our model can generate more syntactically diverse paraphrases with retrieved syntactic templates.
$^{7}$ https://nlp.stanford.edu/projects/glove/
Source	it looks to me like they are really sweet boys.
Reference	they look like very nice boys, i think.
SGCP	i have no idea.
SCPN	you know, they 're a lot of cute guys.
SynTrans	i think they 're really cute boys,
GuiG	i do n't like nice guys, i think.
SI-SCP	they look like nice boys, i think.
Source	have you seen this man since your brother was killed?
Reference	after the murder of your brother, did you see this man?
SGCP	since your brother, did you see your brother?
SCPN	so, did you see the guy since your brother was murdered?
SynTrans	after your brother, did you see that you killed him?
GuiG	since your brother, did you see the guy?
SI-SCP	since your brother's death, did you see him?
Source	a classic kind of friend is what she wants me to be.
Reference	she wants me to be like a classic kind of friend.
SGCP	my friend's gon na be a classic friend.
SCPN	a classic type of boyfriend is what he wants to do.
SynTrans	it's a classic friend of mine that she wants me.
GuiG	it's me to be a classic friend of mine.
SI-SCP	he wants me to be a classic friend.
Table 6: Example paraphrases generated by each model on ParaNMT-Small Dataset.
Template	Paraphrase
Source	the balance between budget revenues and expenditure must be maintained.
(S (NP (PRP ))( VP (VBZ ) ( ADJP ))(. ))	it is necessary to maintain the balance between budget revenue and expenditure.
(S (NP (NP ) (PP ))( VP (MD ) (VP ))(. ))	the balance of budget revenue and expenditure must be kept.
(S (NP (NP ) (PP ))( VP (VBZ ) (ADJP ))(. ))	the balance between budget revenue and expenditure is necessary.
(S (SBAR (WHADVP ) (S ))(. )	when the budget revenue is recovered, it is necessary to maintain expenditure.
(NP (PRP ))( VP (VBZ ) (ADJP ))(. ))
Table 7: Syntactic paraphrases generated by SI-SCP with retrieved templates. We show several successful and one failed (in blue) generations.