SlowGuess/ACL_2020_OCR huggingface dataset:
the AAAI Conference on Artificial Intelligence, volume 33, pages 6883-6891.
Ellie Pavlick, Pushpendre Rastogi, Juri Ganitkevitch, Benjamin Van Durme, and Chris Callison-Burch. 2015. PPDB 2.0: Better paraphrase ranking, fine-grained entailment relations, word embeddings, and style classification. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), pages 425-430, Beijing, China. Association for Computational Linguistics.
Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of NAACL-HLT, pages 2227-2237.
Shrimai Prabhumoye, Yulia Tsvetkov, Ruslan Salakhutdinov, and Alan W Black. 2018. Style Transfer Through Back-Translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 866-876.
Lihua Qian, Lin Qiu, Weinan Zhang, Xin Jiang, and Yong Yu. 2019. Exploring Diverse Expressions for Paraphrase Generation. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 3164-3173.
Abigail See, Peter J Liu, and Christopher D Manning. 2017. Get To The Point: Summarization with Pointer-Generator Networks. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1073-1083.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural Machine Translation of Rare Words with Subword Units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725.
Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017. Style transfer from non-parallel text by cross-alignment. In Advances in neural information processing systems, pages 6830-6841.
Roy Tromble and Jason Eisner. 2009. Learning linear ordering problems for better translation. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 2-Volume 2, pages 1007-1016. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in neural information processing systems, pages 5998-6008.
Ashwin K Vijayakumar, Michael Cogswell, Ramprasaath R Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search for improved description of complex scenes. In Thirty-Second AAAI Conference on Artificial Intelligence.
Chao Wang, Michael Collins, and Philipp Koehn. 2007. Chinese syntactic reordering for statistical machine translation. In Proceedings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning (EMNLP-CoNLL), pages 737-745.
Di Wang, Nebojsa Jojic, Chris Brockett, and Eric Nyberg. 2017. Steering Output Style and Topic in Neural Response Generation. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2140-2150.
Su Wang, Rahul Gupta, Nancy Chang, and Jason Baldridge. 2019. A task in a suit and a tie: paraphrase generation with semantic augmentation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 7176-7183.
John Wieting and Kevin Gimpel. 2018. ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 451-462, Melbourne, Australia. Association for Computational Linguistics.
John Wieting, Jonathan Mallinson, and Kevin Gimpel. 2017. Learning Paraphrastic Sentence Embeddings from Back-Translated Bitext. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 274-285.
Dekai Wu. 1997. Stochastic inversion transduction grammars and bilingual parsing of parallel corpora. Computational linguistics, 23(3):377-403.
Fei Xia and Michael McCord. 2004. Improving a statistical MT system with automatically learned rewrite patterns. In Proceedings of the 20th international conference on Computational Linguistics, page 508. Association for Computational Linguistics.
Qiongkai Xu, Juyan Zhang, Lizhen Qu, Lexing Xie, and Richard Nock. 2018. D-PAGE: Diverse Paraphrase Generation. CoRR, abs/1808.04364.
Adams Wei Yu, David Dohan, Quoc Le, Thang Luong, Rui Zhao, and Kai Chen. 2018. QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehension. In International Conference on Learning Representations.
Shiyue Zhang and Mohit Bansal. 2019. Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing
(EMNLP-IJCNLP), pages 2495-2509, Hong Kong, China. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020. BERTScore: Evaluating Text Generation with BERT. In International Conference on Learning Representations.
A SELECTSEGMENTPAIRS: Limiting number of segment pairs
As outlined in Section 2.3, the SELECTSEGMENT-PAIRS subroutine returns a set of non-overlapping sub-phrases  $(A,B)$ . In order to limit the number of sub-phrase pairs during inference, we employ the following heuristics:
1. We compute a score based on number of nonabstracted tokens divided by the total number of tokens in the yield of the parent sub-phrase  $t$ . We reject pairs  $(A, B)$  that have a score of more than 0.6. This reduces spurious ambiguity by encouraging the model to rearrange big constituents hierarchically rather than only abstracting out small pieces.
2. We maintain a list of tags that are never individually selected as sub-phrases. These include constituents that would be trivial to the reordering such as determiners (DT), prepositions (IN), cardinal numbers (CD), modals (MD), etc. However, these may be a part of larger constituents that form  $A$  or  $B$ .
B Training Data for SOW MODEL
In Section 3.2, we outlined our approach for obtaining phrase-level alignments from the PARANMT-50M dataset used to train the SOW MODEL. In the described approach, an alignment score is computed between each pair of phrases  $p, \hat{p}$  belonging to sentences  $s$  and  $\hat{s}$  respectively. We use the exact procedure in Zhang and Bansal (2019) to compute the alignment score, outlined below:
1. First, we compute an inverse document frequency  $(idf)$  score for each token in the training set. Let  $M = \{s^{(i)}\}$  be the total number of sentences. Then  $idf$  of a word  $w$  is computed as:
$$
i d f (w) = - \log {\frac {1}{M}} \sum_ {i = i} ^ {M} \mathbb {1} [ w \in s ^ {(i)} ]
$$
2. Next, we extract a contextual representation of each word in the two phrases  $s$  and  $\hat{s}$ . We use ELMo (Peters et al., 2018) in our approach.
Sow Input	Sow Output
removing the NN from NP	excluding this NN from NP
they might consider VP if NP were imposed	in the case of imposition of NP, they would consider VP
NP lingered in the deserted NNS .	in the abandoned NNS , there was NP .
PP was a black NN archway .	was a black NN passage PP .
there is already a ring NN PP .	PP circular NN exist .
Table 6: Examples of aligned phrase pairs with exactly two sub- phrases abstracted out and replaced with constituent labels. These phrase pairs are used to train the SOW MODEL.
3. In order to compute a similarity score between each pair of phrases  $(p, \hat{p})$ , we use greedy matching to first align each token in the source phrase to its most similar word in the target phrase. To compute phrase-level similarity, these these word-level similarity scores are combined by taking a weighted mean, with weights specified by theidf scores. Formally,
$$
R _ {p, \hat {p}} = \frac {\sum_ {w _ {i} \in p} i d f (w _ {i}) \max _ {\hat {w} _ {j} \in \hat {p}} w _ {i} ^ {T} \hat {w} _ {j}}{\sum_ {w _ {i} \in p} i d f (w _ {i})}
$$
$$
P _ {p, \hat {p}} = \frac {\sum_ {\hat {w} _ {j} \in \hat {p}} i d f (\hat {w} _ {j}) \max _ {w _ {i} \in p} w _ {i} ^ {T} \hat {w} _ {j}}{\sum_ {\hat {w} _ {j} \in \hat {p}} i d f (\hat {w} _ {j})}
$$
$$
F _ {p, \hat {p}} = \frac {2 P _ {p , \hat {p}} R _ {p , \hat {p}}}{P _ {p , \hat {p}} + R _ {p , \hat {p}}}
$$
This scoring procedure is exactly same as the one proposed by Zhang et al. (2020) to evaluate sentence and phrase similarities.
4. Finally, the phrases  $p \in s$  and  $\hat{p} \in \hat{s}$  are aligned if:
$$
p = \operatorname * {a r g m a x} _ {p _ {i} \in s} F _ {p _ {i}, \hat {p}} \& \hat {p} = \operatorname * {a r g m a x} _ {\hat {p} _ {j} \in \hat {s}} F _ {p, \hat {p} _ {j}}
$$
These aligned set of phrase pairs  $(p, \hat{p})$  are used to construct tuples  $(t_A, B, C)$  and  $(t_A', B', C')$ , as outlined in Section 3.2. Table 6 provides examples of such phrase pairs.
C Sow Model Architecture
Figure 6 provides an overview of the SOW seq2seq model. We add POS tag embeddings (or cor
Figure 6: Source Order reWriting (SOW) model. Our model encodes order preference MONOTONE or FLIP through position embeddings added to the encoder output.
responding constituent label embeddings for abstracted X and Y) to the input token embeddings and original order position embeddings. As outlined in Section 2.3, another set of position embeddings corresponding to the order preference, either MONOTONE or FLIP, are further added to the output of the final layer of the encoder. The decoder attends over these augmented encodings during both training and inference.
D Syntactic Exemplars
Table 7 provides an example from the test set of Chen et al. (2019b). The output retains the semantics of the input sentence while following the structure of the exemplar.
I: his teammates eyes got an ugly, hostile expression.
E: the smell of flowers was thick and sweet.
O: the eyes of his teammates had turned ugly and hostile.
Table 7: Example of input (I), syntactic exemplar (E), and the reference output (O) from the evaluation test set of (Chen et al., 2019b).
E Example Generations
In Table 8, we provide examples of paraphrases generated by our system (SOW-REAP) and the baseline SCPN (Iyyer et al., 2018) system. We additionally include the phrase level transductions applied to obtain the sentence level reordering by our system (column 1) and the input template that the corresponding SCPN generation was conditioned on (Column 3).
Rules (SOW)	Output (REAP)	Template (SCPN)	Output (SCPN)
Input: the public tender result message normally contains the following information :
NP normally contains the following NN: → the following NN usually contains in NP :	the following information shall normally be included in the public procurement report :	SBARQ ( WHADVP SQ . )	where is the public procurement report usually contains the following information .
NP normally VP : → usually VP , NPVBZ the following NN → the NNVBZ	normally the following information shall be included in the public procurement result report :	S ( PP , NP VP . )	in the public competition , the report on competition contains the following information .
Input: the story of abi-wan kenobi ends here .
NP VP . → VP is NP the NN of NP → NP NN .	end of the abi-wan kenobi story .	S ( VP . )	tell the story of abi-wan kenobi .
the story PP NNS here . → there NNS a story PP .	here ends the story of abi-wan kenobi .	S ( S , CC S . )	the story of abi-wan kenobi is here , and it ends here .
Input: i leased it before i knew where the money came from .
i VBN it before i VP . → before i VP , i VBN it .	before i knew where the money came from , i rented it .	SBARQ ( WHADVP SQ . )	where did you learn that it was the money ?
NP knew SBAR . → SBAR , S knew .	where the money came from , i lent it to me before i knew .	S ( NP VP . )	i borrowed money before i knew where the money came from .
Input: priority actions should be more clearly specified in future reviews .
NP should be more clearly specified PP . → PP , NP should be clearly specified .	in future reviews , priority measures should be more clearly specified .	S ( S , CC S . )	priority actions should be more clearly specified in future reviews , and they should be informed .
ADVP VBN in future reviews → VBN in future reviews ADVP	priority measures should be specified in future reviews clearly .	SBARQ ( WHADVP SQ . )	where should priority actions are more clearly specified in future reviews ?
Input: okay , well , tonight the occasion is calling .
ADJP , S . → S , ADJP . well , NN the occasion VP → the occasion VP , NN	the occasion is calling today , okay ?	S ( NP VP . )	the opportunity is calling .
ADJP , S . → S , ADJP . well , NP VBZ calling → VBZ calling NP	we 'll call it tonight , okay ?	S ( ADVP NP VP . )	of course , the occasion is calling .
Input: a minor risk considering the number of telephones in new york .
a JJ risk considering NP . → NP is a JJ risk . the NN of NP → NP NN	phones in new york are a minor risk considering .	SBARQ ( WHADVP SQ . )	when do you consider the number of telephones in new york ?
NP1 considering NP2 . → considering NP2 for NP1 NN of NP → NP NN	in new york , the number of phones is a minor risk .	FRAG ( SBAR ) .	that minor risk is the number of telephones in new york .
NP in JJ york → JJ york NP
Input: that dress gets me into anywhere i want .
that S i VBP . → i VBP S .	i want that dress gets me into the place .	NP ( NP . )	that dress gets me in there , i wish .
that S i VBP . → i VBP S . NN gets me PP → PP , NN gets me .	i want a dress in front of me .	S ( VP . )	i want everywhere .
Table 8: Examples of paraphrases generated by our system and the baseline SCPN model. The outputs from our model successfully rearranges the different structural components of the input sentence to obtain meaningful rearrangements. SCPN on the other hand tends to conform to pre-specified templates that are often not aligned with a given input. 251
The hyperparameters values used in REAP (see Table 9) and Sow (see Table 10) models. Note that we do not use coverage loss for the SOW model.
Seq2seq transformer architecture
Hidden size	256
Num layers	2
Num heads	8
Dropout	0.1
Training
Optimizer	Adam, β = (0.9, 0.999), ε = 10-8
Learning rate	0.0001
Batch size	32
Epochs	50 (maximum)
Coverage loss coeff.	1 (first 10 epochs), 0.5 (10 - 20 epochs), 0 (rest)
Inference
k in top-k	20
Beam Size	10
Table 9: Hyperparameters used in the implementation of the REAP model.
Seq2seq transformer architecture
Hidden size	256
Num layers	2
Num heads	8
Dropout	0.1
Training
Optimizer	Adam, β = (0.9, 0.999), ε = 10-8
Learning rate	0.0001
Batch size	32
Epochs	50 (maximum)
Recombination of rules/transductions
Ignored tags	DT, IN, CD, MD, TO, PRP
Max. no. of rules	3
Table 10: Hyperparameters used in the implementation of the Sow model.