SlowGuess/ACL_2020_OCR huggingface dataset:
The SCPN model has a high rejection score of  $40.6\%$ . This demonstrates that out of the 10 templates used to generate paraphrases for each sentence, on average 4 were not appropriate for the given sentence, and therefore get rejected. On the other hand, for our model, only  $15.9\%$  of the generated paraphrases get rejected, implying that the rearrangements produced were generally meaningful. This is comparable to the  $12.7\%$  rejection rate
exhibited by the vanilla seq2seq model that does not condition on any syntax or rearrangement, and is therefore never obliged to conform to an inappropriate structure.
Finally, our model exhibits a much higher diversity within the generated paraphrases compared to the transformer seq2seq baseline. As expected, the SCPN model produces slightly more diverse paraphrases as it explicitly conditions the generations on templates with very different top level structures. However, this is often at the cost of semantic equivalence, as demonstrated by both quantitative and human evaluation (next section). A similar trend was observed with the diverse-decoding scheme. Although it leads to more diverse generations, there is a substantial decrease in quality compared to SOW-REAP and the seq2seq model. Moreover, the paraphrases have a higher rejection rate  $(21.3\%)$ , suggesting that diverse decoding is more likely to produce nonsensical paraphrases. A similar phenomenon is also reported by Iyyer et al. (2018), wherein diverse-decoding resulted in paraphrases with different semantics than the input.
Syntactic Exemplars In addition to SCPN, we compare our proposed model against the controllable generation method of Chen et al. (2019b). Their model uses an exemplar sentence as a syntactic guide during generation; the generated paraphrase is trained to incorporate the semantics of the input sentence while emulating the syntactic structure of the exemplar (see Appendix D for examples). However, their proposed approach depends on the availability of such exemplars at test time; they manually constructed these for their test set (800 examples). Since we do not have such example sentences available for our test data, we report results of our model's performance on their test data.
The difference in performance between our proposed model and baseline models is statistically significant according to a paired bootstrap test.
Input	SOW-REAP	SCPN
if at any time in the preparation of this product the integrity of this container is compromised it should not be used.	this container should not be used if any time in the preparation of this product is compromised	in the preparation of this product, the integrity of this container is compromised, but it should not be used.
if the integrity of the packaging is impaired at any time, the product should not be used.	where is the integrity of this product of this container the integrity of this container should not be used.
if the product integrity of this container is compromised it should not be used.	i should not use if at any time in the preparation of this product, it should not be used.
i was the first grower to use hydroponics.	to use hydroponics, i was the first one.	where did i have the first tendency to use hydroponics?
i used hydroponics for the first time.	i used to use hydroponics.
to use hydroponics the first time i was.	first i was the first grower to use hydroponics
Note that Chen et al. (2019b) carefully curated the exemplar to be syntactically similar to the actual target paraphrase. Therefore, for fair comparison, we report results using the ground truth ordering (that similarly leverages the target sentence to obtain a source reordering), followed by the REAP model. This model (ground truth order + REAP) achieves a 1-best BLEU score of 20.9, outperforming both the prior works: Chen et al. (2019b) (13.6 BLEU) and SCPN (17.8 BLEU with template, 19.2 BLEU with full parse). Furthermore, our full SOW-REAP model gets an oracle-BLEU (across 10 sentences) score of 23.8. These results show that our proposed formulation outperforms other controllable baselines, while being more flexible.
Table 2 provides examples of paraphrase outputs produced by our approach and SCPN. The examples show that our model exhibits syntactic diversity while producing reasonable paraphrases of the input sentence. On the other hand, SCPN tends to generate non-paraphrases in order to conform to a given template, which contributes to increased diversity but at the cost of semantic equivalence. In Table 3, we show the corresponding sequence of rules that apply to an input sentence, and the final generated output according to that input rearrangement. Note that for our model, on average, 1.8 phrase-level reorderings were combined to produce sentence-level reorderings (we restrict to a maximum of 3). More examples along with the input rule sequence (for our model) and syntactic templates (for SCPN) are provided in the Appendix.
Human Evaluation We also performed human evaluation on Amazon Mechanical Turk to evalu
Table 2: Examples of paraphrases generated by our system and the baseline SCPN model. Our model successfully rearranges the different structural components of the input sentence to obtain meaningful rearrangements. SCPN conforms to pre-enumerated templates that may not align with a given input.
Input Sentence: if at any time in the preparation of this product the integrity of this container is compromised it should not be used .
Rule Sequence: if S it should not VB used. → should not VB used if S (parse tree level: 0)
at NP the integrity of this container VBZ compromised → this container VBZ weakened at NP (parse tree level: 1)
the NN of NP → NP NN (parse tree level: 2)
Generated Sentence: this container should not be used if the product is compromised at any time in preparation .
Table 3: Examples of our model's rearrangements applied to a given input sentence. Parse tree level indicates the rule subtree's depth from the root node of the sentence. The REAP model's final generation considers the rule reordering at the higher levels of the tree but ignores the rearrangement within the lower sub-tree.
ate the quality of the generated paraphrases. We randomly sampled 100 sentences from the development set. For each of these sentences, we obtained 3 generated paraphrases from each of the following models: i) SCPN, ii) vanilla sequence-to-sequence and iii) our proposed SOW-REAP model. We follow earlier work (Kok and Brockett, 2010; Iyyer et al., 2018) and obtain quality annotations on a 3 point scale:  $\mathbf{0}$  denotes not a paraphrase,  $\mathbf{1}$  denotes that the input sentence and the generated sentence are paraphrases, but the generated sentence might contain grammatical errors,  $\mathbf{2}$  indicates that the input and the candidate are paraphrases. To emulate the human evaluation design in Iyyer et al. (2018), we sample paraphrases after filtering using the criterion outlined in the previous section and obtain three judgements per sentence and its 9 paraphrase candidates. Table 4 outlines the results from the human evaluation. As we can see, the results indicate
Model	2	1	0
SCPN (Iyyer et al., 2018)	35.9	24.8	39.3
Transformer seq2seq	45.1	20.6	34.3
SOW-REAP	44.5	22.6	32.9
Table 4: Human annotated quality across different models. The evaluation was done on a 3 point quality scale,  $2 =$  grammatical paraphrase,  $1 =$  ungrammatical paraphrase,  $0 =$  not a paraphrase.
Ordering	oracle-ppl ↓	oracle-BLEU ↑
Monotone	10.59	27.98
Random	9.32	27.10
Sow	8.14	30.02
Ground Truth	7.79	36.40
Table 5: Comparison of different source reordering strategies. Our proposed approach outperforms baseline monotone and random rearrangement strategies.
that the quality of the paraphrases generated from our model is substantially better than the SCPN model. Furthermore, similar to quantitative evaluation, the human evaluation also demonstrates that the performance of this model is similar to that of the vanilla sequence-to-sequence model, indicating that the inclusion of target rearrangements do not hurt performance.
4.3.1 Evaluation of Sow Model
Next, we intrinsically evaluate the performance of our SOW model (Section 2.3). Specifically, given a budget of 10 reorderings, we want to understand how close our SOW model comes to covering the target ordering. We do this by evaluating the REAP model in terms of oracle perplexity (of the ground truth paraphrase) and oracle BLEU over these 10 orderings.
We evaluate our proposed approach against 3 systems: a) Monotone reordering  $\{1,2,\dots,n\}$ . b) Random permutation, by randomly permuting the children of each node as we traverse down the constituency parse tree. c) Ground Truth, using the pseudo-ground truth rearrangement (outlined in Section 3) between the source and ground-truth target sentence. This serves as an upper bound for the reorderings' performance, as obtained by the recursive phrase-level transducer.
Figure 5: The degree of rearrangement (Kendall's Tau) achieved by conditioning on monotone and pseudoground truth reorderings  $(\mathbf{r}^{*})$ . The dotted line denotes the ideal performance (in terms of reordering-compliance) of the REAP model, when supplied with perfect reordering  $\mathbf{r}^{*}$ . The actual performance of the REAP model mirrors the ideal performance.
Table 5 outlines the results for 10 generated paraphrases from each rearrangement strategy. Our proposed approach outperforms the baseline monotone and random reordering strategies. Furthermore, the Sow model's oracle perplexity is close to that of the ground truth reordering's perplexity, showing that the proposed approach is capable of generating a diverse set of rearrangements such that one of them often comes close to the target rearrangement. The comparatively high performance of the ground truth reorderings demonstrates that the positional embeddings are effective at guiding the REAP model's generation.
4.3.2 Compliance with target reordering
Finally, we evaluate whether the generated paraphrases follow the target reordering  $\mathbf{r}$ . Note that we do not expect or want our REAP model to be absolutely compliant with this input reordering since the model should be able to correct for the mistakes made by the SOW model and still generate valid paraphrases. Therefore, we perform reordering compliance experiments on only the monotone reordering and the pseudo-ground truth reorderings  $(\mathbf{r}^*$ , construction outlined in Section 3), since these certainly correspond to valid paraphrases.
For sentences in the test set, we generate paraphrases using monotone reordering and pseudoground truth reordering as inputs to REAP. We get the 1-best paraphrase and compute the degree of rearrangement $^{10}$  between the input sentence and
10 Quantified by Kendall's Tau rank correlation between original source order and targeted/generated order. Higher
The difference of our model performance with SCPN is statistically significant, while that with baseline seq2seq is not according to a paired bootstrap test.
the generated sentence. In Figure 5, we plot this as a function of the target degree of rearrangement, i.e., the rearrangement between the input sentence  $\mathbf{x}$  and the ground truth sentence  $\mathbf{y}^*$ . The dotted line denotes the ideal performance of the model in terms of agreement with the perfect reordering  $\mathbf{r}^*$ . The plot shows that the REAP model performs as desired; the monotone generation results in high Kendall's Tau between input and output. Conditioning on the pseudo-ground truth reorderings  $(\mathbf{r}^*)$  produces rearrangements that exhibit the same amount of reordering as the ideal rearrangement.
Paraphrase Generation Compared to prior seq2seq approaches for paraphrasing (Hasan et al., 2016; Gupta et al., 2018; Li et al., 2018), our model is able to achieve much stronger controllability with an interpretable control mechanism. Like these approaches, we can leverage a wide variety of resources to train on, including backtranslation (Pavlick et al., 2015; Wieting and Gimpel, 2018; Hu et al., 2019) or other curated data sources (Fader et al., 2013; Lan et al., 2017).
Controlled Generation Recent work on controlled generation aims at controlling attributes such as sentiment (Shen et al., 2017), gender or political slant (Prabhumoye et al., 2018), topic (Wang et al., 2017), etc. However, these methods cannot achieve fine-grained control over a property like syntax. Prior work on diverse paraphrase generation can be divided into three groups: diverse decoding, latent variable modeling, and syntax-based. The first group uses heuristics such as Hamming distance or distinct  $n$ -grams to preserve diverse options during beam search decoding (Vijayakumar et al., 2018; Kumar et al., 2019). The second group includes approaches that use uninterpretable latent variables to separate syntax and semantics (Chen et al., 2019a), perturb latent representations to enforce diversity (Gupta et al., 2018; Park et al., 2019) or condition on latent codes used to represent different re-writing patterns (Xu et al., 2018; An and Liu, 2019). Qian et al. (2019) uses distinct generators to output diverse paraphrases. These methods achieve some diversity, but do not control generation in an interpretable manner. Finally, methods that use explicit syntactic structures (Iyyer et al., 2018; Chen et al., 2019b) may try to force a
sentence to conform to unsuitable syntax. Phrase-level approaches (Li et al., 2019) are inherently less flexible than our approach.
Machine Translation Our work is inspired by pre-ordering literature in machine translation. These systems either use hand-crafted rules designed for specific languages (Collins et al., 2005; Wang et al., 2007) or automatically learn rewriting patterns based on syntax (Xia and McCord, 2004; Dyer and Resnik, 2010; Genzel, 2010; Khalilov and Simaan, 2011; Lerner and Petrov, 2013). There also exist approaches that do not rely on syntactic parsers, but induce hierarchical representations to leverage for pre-ordering (Tromble and Eisner, 2009; DeNero and Uszkoreit, 2011). In the context of translation, there is often a canonical reordering that should be applied to align better with the target language; for instance, head-final languages like Japanese exhibit highly regular syntax-governed reorderings compared to English. However, in diverse paraphrase generation, there doesn't exist a single canonical reordering, making our problem quite different.
In concurrent work, Chen et al. (2020) similarly use an additional set of position embeddings to guide the order of generated words for machine translation. This demonstrates that the REAP technique is effective for other tasks also. However, they do not tackle the problem of generating plausible reorderingings and therefore their technique is less flexible than our full SOW-REAP model.
In this work, we propose a two-step framework for paraphrase generation: construction of diverse syntactic guides in the form of target reorderings followed by actual paraphrase generation that respects these reorderings. Our experiments show that this approach can be used to produce paraphrases that achieve a better quality-diversity trade-off compared to previous methods and strong baselines.
This work was partially supported by NSF Grant IIS-1814522, a gift from Arm, and an equipment grant from NVIDIA. The authors acknowledge the Texas Advanced Computing Center (TACC) at The University of Texas at Austin for providing HPC resources used to conduct this research. Thanks as well to the anonymous reviewers for their helpful comments.
Kendall's Tau indicates lower rearrangement and vice-versa.
Zhecheng An and Sicong Liu. 2019. Towards Diverse Paraphrase Generation Using Multi-Class Wasserstein GAN. arXiv preprint arXiv:1909.13827.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2014. Neural Machine Translation by Jointly Learning to Align and Translate. CoRR, abs/1409.0473.
Regina Barzilay and Lillian Lee. 2003. Learning to paraphrase: an unsupervised approach using multiple-sequence alignment. In Proceedings of the 2003 Conference of the North American Chapter of the Association for Computational Linguistics on Human Language Technology-Volume 1, pages 16-23. Association for Computational Linguistics.
Ondrej Bojar, Ondrej Dušek, Tom Kocmi, Jindrich Libovický, Michal Novák, Martin Popel, Roman Sudarikov, and Dušan Variš. 2016. CzEng 1.6: enlarged Czech-English parallel corpus with processing tools Dockered. In International Conference on Text, Speech, and Dialogue, pages 231-238. Springer.
Kehai Chen, Rui Wang, Masao Utiyama, and Eiichiro Sumita. 2020. Explicit Reordering for Neural Machine Translation. arXiv preprint arXiv:2004.03818.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019a. A Multi-Task Approach for Disentangling Syntax and Semantics in Sentence Representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2453–2464.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019b. Controllable Paraphrase Generation with a Syntactic Exemplar. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5972-5984, Florence, Italy. Association for Computational Linguistics.
Jaemin Cho, Minjoon Seo, and Hannaneh Hajishirzi. 2019. Mixture Content Selection for Diverse Sequence Generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3112-3122.
Michael Collins, Philipp Koehn, and Ivona Kučerová. 2005. Clause restructuring for statistical machine translation. In Proceedings of the 43rd annual meeting on association for computational linguistics, pages 531-540. Association for Computational Linguistics.
John DeNero and Jakob Uszkoreit. 2011. Inducing sentence structure from parallel corpora for reordering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing, pages 193-203. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Chris Dyer and Philip Resnik. 2010. Context-free reordering, finite-state translation. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 858-866, Los Angeles, California. Association for Computational Linguistics.
Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2013. Paraphrase-driven learning for open question answering. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1608-1618.
Angela Fan, Mike Lewis, and Yann Dauphin. 2018. Hierarchical Neural Story Generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889-898.
Michel Galley, Mark Hopkins, Kevin Knight, and Daniel Marcu. 2004. What's in a translation rule? In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 273-280, Boston, Massachusetts, USA. Association for Computational Linguistics.
Dmitriy Genzel. 2010. Automatically learning source-side reordering rules for large scale machine translation. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 376-384.
Ankush Gupta, Arvind Agarwal, Prawaan Singh, and Piyush Rai. 2018. A deep generative framework for paraphrase generation. In Thirty-Second AAAI Conference on Artificial Intelligence.
Sadid A Hasan, Kathy Lee, Vivek Datla, Ashequl Qadir, Joey Liu, Oladimeji Farri, et al. 2016. Neural Paraphrase Generation with Stacked Residual LSTM Networks. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2923-2934.
J. Edward Hu, Rachel Rudinger, Matt Post, and Benjamin Van Durme. 2019. ParaBank: Monolingual
Bitext Generation and Sentential Paraphrasing via Lexically-constrained Neural Machine Translation. In Proceedings of AAAI.
Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial Example Generation with Syntactically Controlled Paraphrase Networks. In Proceedings of NAACL-HLT, pages 1875-1885.
Maxim Khalilov and Khalil Simaan. 2011. Context-sensitive syntactic source-reordering by statistical transduction. In Proceedings of 5th International Joint Conference on Natural Language Processing, pages 38-46.
Stanley Kok and Chris Brockett. 2010. Hitting the right paraphrases in good time. In Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics, pages 145-153. Association for Computational Linguistics.
Ashutosh Kumar, Satwik Bhattachamishra, Manik Bhandari, and Partha Talukdar. 2019. Submodular Optimization-based Diverse Paraphrasing and its Effectiveness in Data Augmentation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3609-3619.
Wuwei Lan, Siyu Qiu, Hua He, and Wei Xu. 2017. A Continuously Growing Dataset of Sentential Paraphrases. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 1224-1234, Copenhagen, Denmark. Association for Computational Linguistics.
Uri Lerner and Slav Petrov. 2013. Source-side classifier preordering for machine translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 513-523.
Zichao Li, Xin Jiang, Lifeng Shang, and Hang Li. 2018. Paraphrase Generation with Deep Reinforcement Learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3865-3878, Brussels, Belgium. Association for Computational Linguistics.
Zichao Li, Xin Jiang, Lifeng Shang, and Qun Liu. 2019. Decomposable Neural Paraphrase Generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3403-3414, Florence, Italy. Association for Computational Linguistics.
Kathleen R McKeown. 1983. Paraphrasing questions using given and new information. Computational Linguistics, 9(1):1-10.
Sunghyun Park, Seung-won Hwang, Fuxiang Chen, Jaegul Choo, Jung-Woo Ha, Sunghun Kim, and Jinyeong Yim. 2019. Paraphrase Diversification Using Counterfactual Debiasing. In Proceedings of