SlowGuess/ACL_2020_OCR huggingface dataset:
Tanya Goyal and Greg Durrett
tanyagoyal@utexas.edu, gdurrett@cs.utexas.edu
Paraphrasing natural language sentences is a multifaceted process: it might involve replacing individual words or short phrases, local rearrangement of content, or high-level restructuring like topicalization or passivization. Past approaches struggle to cover this space of paraphrase possibilities in an interpretable manner. Our work, inspired by pre-ordering literature in machine translation, uses syntactic transformations to softly "reorder" the source sentence and guide our neural paraphrasing model. First, given an input sentence, we derive a set of feasible syntactic rearrangements using an encoder-decoder model. This model operates over a partially lexical, partially syntactic view of the sentence and can reorder big chunks. Next, we use each proposed rearrangement to produce a sequence of position embeddings, which encourages our final encoder-decoder paraphrase model to attend to the source words in a particular order. Our evaluation, both automatic and human, shows that the proposed system retains the quality of the baseline approaches while giving a substantial increase in the diversity of the generated paraphrases. $^{1}$
Paraphrase generation (McKeown, 1983; Barzilay and Lee, 2003) has seen a recent surge of interest, both with large-scale dataset collection and curation (Lan et al., 2017; Wieting and Gimpel, 2018) and with modeling advances such as deep generative models (Gupta et al., 2018; Li et al., 2019). Paraphrasing models have proven to be especially useful if they expose control mechanisms that can be manipulated to produce diverse paraphrases (Iyyer et al., 2018; Chen et al., 2019b; Park et al., 2019), which allows these models to be employed for data augmentation (Yu et al., 2018) and
Figure 1: Overview of our paraphrase model. First, we choose various pairs of constituents to abstract away in the source sentence, then use a neural transducer to generate possible reorderings of the abstracted sentences. From these, we construct a guide reordering of the input sentence which then informs the generation of output paraphrases.
adversarial example generation (Iyyer et al., 2018). However, prior methods involving syntactic control mechanisms do not effectively cover the space of paraphrase possibilities. Using syntactic templates covering the top of the parse tree (Iyyer et al., 2018) is inflexible, and using fully-specified exemplar sentences (Chen et al., 2019b) poses the problem of how to effectively retrieve such sentences. For a particular input sentence, it is challenging to use these past approaches to enumerate the set of re-orderings that make sense for that sentence.
In this paper, we propose a two-stage approach to address these limitations, outlined in Figure 1. First, we use an encoder-decoder model (Sow, for Source Order reWriting) to apply transduction operations over various abstracted versions of the input sentence. These transductions yield possible reorderings of the words and constituents, which can be combined to obtain multiple feasible rearrangements of the input sentence. Each rearrangement specifies an order that we should visit words of the source sentence; note that such orderings could encourage a model to passivize (visit the object before the subject), topicalize, or reorder clauses. These orderings are encoded for our encoder-decoder paraphrase model (REAP, for REarrangement Aware Paraphrasing) by way of po
Data and code are available at https://github.com/tagoyal/sow-reap-paraphrasing
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 238-252 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
sition embeddings, which are added to the source sentence encoding to specify the desired order of generation (see Figure 2). This overall workflow is inspired by the pre-ordering literature in machine translation (Xia and McCord, 2004; Collins et al., 2005); however, our setting explicitly requires entertaining a diverse set of possible orderings corresponding to different paraphrasing phenomena.
We train and evaluate our approach on the large-scale English paraphrase dataset PARANMT-50M (Wieting and Gimpel, 2018). Results show that our approach generates considerably more diverse paraphrases while retaining the quality exhibited by strong baseline models. We further demonstrate that the proposed syntax-based transduction procedure generates a feasible set of rearrangements for the input sentence. Finally, we show that position embeddings provide a simple yet effective way to encode reordering information, and that the generated paraphrases exhibit high compliance with the desired reordering input.
Given an input sentence  $\mathbf{x} = \{x_{1}, x_{2}, \ldots, x_{n}\}$ , our goal is to generate a set of structurally distinct paraphrases  $Y = \{\mathbf{y}^{1}, \mathbf{y}^{2}, \ldots, \mathbf{y}^{k}\}$ . We achieve this by first producing  $k$  diverse reorderings for the input sentence,  $R = \{\mathbf{r}^{1}, \mathbf{r}^{2}, \ldots, \mathbf{r}^{k}\}$ , that guide the generation order of each corresponding  $\mathbf{y}$ . Each reordering is represented as a permutation of the source sentence indices.
Our method centers around a sequence-to-sequence model which can generate a paraphrase roughly respecting a particular ordering of the input tokens. Formally, this is a model  $P(\mathbf{y} \mid \mathbf{x}, \mathbf{r})$ . First, we assume access to the set of target reordering  $R$  and describe this rearrangement aware paraphrasing model (REAP) in Section 2.2. Then, in Section 2.3, we outline our reordering approach, including the source order rewriting (SOW) model, which produces the set of reorderings appropriate for a given input sentence  $\mathbf{x}$  during inference  $(\mathbf{x} \rightarrow R)$ .
The models discussed in this work build on a standard sequence-to-sequence transformer model (Vaswani et al., 2017) that uses stacked layers of self-attention to both encode the input tokens  $\mathbf{x}$  and decode the corresponding target sequence  $\mathbf{y}$ . This model is pictured in the gray block of Figure 2. Throughout this work, we use byte pair encoding
Figure 2: Rearrangement aware paraphrasing (REAP) model. The gray area corresponds to the standard transformer encoder-decoder system. Our model adds position embeddings corresponding to the target reordering to encoder outputs. The decoder attends over these augmented encodings during both training and inference.
(BPE) (Sennrich et al., 2016) to tokenize our input and output sentences. These models are trained in the standard way, maximizing the log likelihood of the target sequence using teacher forcing. Additionally, in order to ensure that the decoder does not attend to the same input tokens repeatedly at each step of the decoding process, we include a coverage loss term, as proposed in See et al. (2017).
Note that since the architecture of the transformer model is non-recurrent, it adds position embeddings to the input word embeddings in order to indicate the correct sequence of the words in both  $\mathbf{x}$  and  $\mathbf{y}$  (see Figure 2). In this work, we propose using an additional set of position embeddings to indicate the desired order of words during generation, described next.
2.2 Rearrangement aware Paraphrasing Model (REAP)
Let  $\mathbf{r} = \{r_1, r_2, \dots, r_n\}$  indicate the target reordering corresponding to the input tokens  $\mathbf{x}$ . We want the model to approximately attend to tokens in this specified order when generating the final output paraphrase. For instance, in the example in Figure 1, the reordering specifies that when producing the paraphrase, the model should generate content related to the game before content related to Clippers in the output. In this case, based on the rearrangement being applied, the model will most likely use passivization in its generation, although this is not strictly enforced.
The architecture for our model  $P(\mathbf{y} \mid \mathbf{x}, \mathbf{r})$  is outlined in Figure 2. Consider an encoder-decoder architecture with a stack of  $M$  layers in the encoder
Figure 3: Overview of the source sentence rearrangement workflow for one level of recursion at the root node. First, candidate tree segment pairs contained within the input node are selected. A transduction operation is applied over the abstracted phrase, giving the reordering  $\textbf{45123}$  for the case shown in red, then the process recursively continues for each abstracted node. This results in a reordering for the full source sentence; the reordering indices serve as additional input to the REAP model.
and  $N$  layers in the decoder. We make the target reordering  $\mathbf{r}$  accessible to this transformer model through an additional set of positional embeddings  $PE_{\mathbf{r}}$ . We use the sinusoidal function to construct these following Vaswani et al. (2017).
Let  $E_{M} = \mathrm{encoder}_{M}(\mathbf{x})$  be the output of the  $M^{th}$  (last) layer of the encoder. The special-purpose position embeddings are added to the output of this layer (see Figure 2):  $E = E_{M} + PE_{r}$ . Note that these are separate from standard position embeddings added at the input layer; such embeddings are also used in our model to encode the original order of the source sentence. The transformer decoder model attends over  $E$  while computing attention and the presence of the position embeddings should encourage the generation to obey the desired ordering  $\mathbf{r}$ , while still conforming to the decoder language model. Our experiments in Section 4.3 show that this position embedding method is able to successfully guide the generation of paraphrases, conditioning on both the input sentence semantics as well as the desired ordering.
2.3 Sentence Reordering
We now outline our approach for generating these desired reorderings  $\mathbf{r}$ . We do this by predicting phrasal rearrangements with the Sow model at various levels of syntactic abstraction of the sentence. We combine multiple such phrase-level rearrangements to obtain a set  $R$  of sentence-level rearrangements. This is done using a top-down approach, starting at the root node of the parse tree. The overall recursive procedure is outlined in Algorithm 1.
One step of the recursive algorithm has three
Algorithm 1 REORDER(t)
Input: Sub-tree  $t$  of the input parse tree
Output: Top-  $k$  list of reorderings for  $t$ 's yield
$$
\mathcal {T} = \text {S E L E C T S E G M E N T P A I R S} (t) \quad / / \text {S t e p 1}
$$
$$
\mathcal {R} = \text {I N I T I A L I Z E B E A M} (\text {s i z e} = k)
$$
$$
\textbf {f o r} (A, B) \text {i n} \mathcal {T} \text {d o}
$$
$$
z = \operatorname {R E O R D E R P H R A S E} (t, A, B) \quad / / \text {S t e p 2}
$$
$$
R _ {A} (1, \dots , k) = \operatorname {R E O R D E R} (t _ {A}) / / k \text {o r d e r i n g s}
$$
$$
R _ {B} (1, \dots , k) = \operatorname {R E O R D E R} \left(t _ {B}\right) / / k \text {o r d e r i n g s}
$$
$$
\mathbf {f o r} r _ {a}, r _ {b} \text {i n} R _ {A} \times R _ {B} \mathbf {d o}
$$
$$
r = \operatorname {C o m b i n e} (z, r _ {a}, r _ {b}) \quad / / \text {S t e p 3}
$$
$$
\operatorname {s c o r e} (r) = \operatorname {s c o r e} (z) + \operatorname {s c o r e} \left(r _ {a}\right) + \operatorname {s c o r e} \left(r _ {b}\right)
$$
$$
\mathcal {R}. \operatorname {p u s h} (r, \operatorname {s c o r e} (r))
$$
$$
\mathbf {\Gamma} _ {R}
$$
major steps: Figure 3 shows the overall workflow for one iteration (here, the root node of the sentence is selected for illustration). First, we select sub-phrase pairs of the input phrase that respect parse-tree boundaries, where each pair consists of non-overlapping phrases (Step 1). Since the aim is to learn generic syntax-governed rearrangements, we abstract out the two sub-phrases, and replace them with non-terminal symbols, retaining only the constituent tag information. For example, we show three phrase pairs in Figure 3 that can be abstracted away to yield the reduced forms of the sentences. We then use a seq2seq model to obtain rearrangements for each abstracted phrase (Step 2). Finally, this top-level rearrangement is
combined with recursively-constructed phrase rearrangements within the abstracted phrases to obtain sentence-level rearrangements (Step 3).
Step 1: SELECTSEGMENTPAIRS
We begin by selecting phrase tuples that form the input to our seq2seq model. A phrase tuple  $(t, A, B)$  consists of a sub-tree  $t$  with the constituents  $A$  and  $B$  abstracted out (replaced by their syntactic categories). For instance, in Figure 3, the  $\mathbf{S}_0$ ,  $\mathbf{S}$ , and  $\mathrm{VP}_2$  nodes circled in red form a phrase tuple. Multiple distinct combinations of  $A$  and  $B$  are possible. $^2$
Step 2: REORDERPHRASE
Next, we obtain rearrangements for each phrase tuple  $(t,A,B)$ . We first form an input consisting of the yield of  $t$  with  $A$  and  $B$  abstracted out; e.g. If  $S$  will  $VP$ , shown in red in Figure 3. We use a sequence-to-sequence model (the Sow model) that takes this string as input and produces a corresponding output sequence. We then perform word-level alignment between the input and generated output sequences (using cosine similarity between GloVe embeddings) to obtain the rearrangement that must be applied to the input sequence. The log probability of the output sequence serves as a score for this rearrangement.
Sow model The Sow model is a sequence-to-sequence model  $P(\mathbf{y}' \mid \mathbf{x}', o)$ , following the transformer framework in Section 2.1. Both  $\mathbf{x}'$  and  $\mathbf{y}'$  are encoded using the word pieces vocabulary; additionally, embeddings corresponding to the POS tags and constituent labels (for non-terminals) are added to the input embeddings.
For instance, in Figure 3, If  $S$  will  $VP$  and  $I$  will  $VP$  if  $S$  is an example of an  $(\mathbf{x}', \mathbf{y}')$ , pair. While not formally required, Algorithm 1 ensures that there are always exactly two non-terminal labels in these sequences.  $o$  is a variable that takes values MONOTONE or FLIP. This encodes a preference to keep the two abstracted nodes in the same order or to "flip" them in the output.  $o$  is encoded in the model with additional positional encodings of the form  $\{ \dots 0, 0, 1, 0, \dots, 2, 0 \dots \}$  for monotone and
$\{ \ldots 0, 0, 2, 0, \ldots 1, 0 \ldots \}$  for flipped, wherein the non-zero positions correspond to the positions of the abstracted non-terminals in the phrase. These positional embeddings for the SOW MODEL are handled analogously to the r embeddings for the REAP model. During inference, we use both the monotone rearrangement and flip rearrangement to generate two reorderings, one of each type, for each phrase tuple.
We describe training of this model in Section 3.
Step 3: COMBINE
The previous step gives a rearrangement for the subtree  $t$ . To obtain a sentence-level rearrangement from this, we first recursively apply the RE-ORDER algorithm on subtrees  $t_A$  and  $t_B$  which returns the top-k rearrangements of each subtree. We iterate over each rearrangement pair  $(r_a, r_b)$ , applying these reorderings to the abstracted phrases  $A$  and  $B$ . This is illustrated on the left side of Figure 3. The sentence-level representations, thus obtained, are scored by taking a mean over all the phrase-level rearrangements involved.
3 Data and Training
We train and evaluate our model on the PARANMT-50M paraphrase dataset (Wieting and Gimpel, 2018) constructed by backtranslating the Czech sentences of the CzEng (Bojar et al., 2016) corpus. We filter this dataset to remove shorter sentences (less than 8 tokens), low quality paraphrase pairs (quantified by a translation score included with the dataset) and examples that exhibit low reordering (quantified by a reordering score based on the position of each word in the source and its aligned word in the target sentence). This leaves us with over 350k paired paraphrase pairs.
3.1 Training Data for REAP
To train our REAP model (outlined in Section 2.2), we take existing paraphrase pairs  $(\mathbf{x},\mathbf{y}^{*})$  and derive pseudo-ground truth rearrangements  $\mathbf{r}^*$  of the source sentence tokens based on their alignment with the target sentence. To obtain these rearrangements, we first get contextual embeddings (Devlin et al., 2019) for all tokens in the source and target sentences. We follow the strategy outlined in Lerner and Petrov (2013) and perform reorderings as we traverse down the dependency tree. Starting at the root node of the source sentence, we determine the order between the head and its children (independent of other decisions) based on the order
2In order to limit the number of such pairs, we employ a threshold on the fraction of non-abstracted words remaining in the phrase, outlined in more detail in the Appendix.
3We experimented with a pointer network to predict indices directly; however, the approach of generate and then align post hoc resulted in a much more stable model.
See Appendix for SOW model architecture diagram.
In syntactic translation systems, rules similarly can be divided by whether they preserve order or invert it (Wu, 1997).
Figure 4: Paraphrase sentence pair and its aligned tuples  $A \to B, C$  and  $A' \to B', C'$ . These produce the training data for the SOW MODEL.
of the corresponding aligned words in the target sentence. We continue this traversal recursively to get the sentence level-rearrangement. This mirrors the rearrangement strategy from Section 2.3, which operates over constituency parse tree instead of the dependency parse.
Given triples  $(\mathbf{x},\mathbf{r}^{*},\mathbf{y}^{*})$  , we can train our REAP model to generate the final paraphrases conditioning on the pseudo-ground truth reorderings.
3.2 Training Data for Sow
The PARANMT-50M dataset contains sentence-level paraphrase pairs. However, in order to train our SOW model (outlined in section 2.3), we need to see phrase-level paraphrases with syntactic abstractions in them. We extract these from the PARANMT-50M dataset using the following procedure, shown in Figure 4. We follow Zhang et al. (2020) and compute a phrase alignment score between all pairs of constituents in a sentence and its paraphrase. From this set of phrase alignment scores, we compute a partial one-to-one mapping between phrases (colored shapes in Figure 4); that is, not all phrases get aligned, but the subset that do are aligned one-to-one. Finally, we extract aligned chunks similar to rule alignment in syntactic translation (Galley et al., 2004): when aligned phrases  $A$  and  $A'$  subsume aligned phrase pairs  $(B, C)$  and  $(B', C')$  respectively, we can extract the aligned tuple  $(t_A, B, C)$  and  $(t_{A'}, B', C')$ . The phrases  $(B, C)$  and  $(B', C')$  are abstracted out to construct training data for the phrase-level transducer, including supervision of whether  $o = \text{MONOTONE}$  or FLIP. Using the above alignment strategy, we were able to obtain over 1 million aligned phrase pairs.
Setup As our main goal is to evaluate our model's ability to generate diverse paraphrases, we
obtain a set of paraphrases and compare these to sets of paraphrases produced by other methods. To obtain 10 paraphrases, we first compute a set of 10 distinct reorderings  $\mathbf{r}^1,\dots ,\mathbf{r}^{10}$  with the SOW method from Section 2.3 and then use the REAP to generate a 1-best paraphrase for each. We use top $k$  decoding to generate the final set of paraphrases corresponding to the reorderings. Our evaluation is done over 10k examples from PARANMT-50M.
4.1 Quantitative Evaluation
Baselines We compare our model against the Syntactically Controlled Paraphrase Network (SCPN) model proposed in prior work (Iyyer et al., 2018). It produces 10 distinct paraphrase outputs conditioned on a pre-enumerated list of syntactic templates. This approach has been shown to outperform other paraphrase approaches that condition on interpretable intermediate structures (Chen et al., 2019b). Additionally, we report results on the following baseline models: i) A copy-input model that outputs the input sentence exactly. ii) A vanilla seq2seq model that uses the same transformer encoder-decoder architecture from Section 2.1 but does not condition on any target rearrangement. We use top- $k$  sampling (Fan et al., 2018) to generate 10 paraphrases from this model.<sup>7</sup> iii) A diverse-decoding model that uses the above transformer seq2seq model with diverse decoding (Kumar et al., 2019) during generation. Here, the induced diversity is uncontrolled and aimed at maximizing metrics such as distinct n-grams and edit distance between the generated sentences. iv) A LSTM version of our model where the REAP model uses LSTMs with attention (Bahdanau et al., 2014) and copy (See et al., 2017) instead of transformers. We still use the transformer-based phrase transducer to obtain the source sentence reorderings, and still use positional encodings in the LSTM attention.
Similar to Cho et al. (2019), we report two types of metrics:
1. Quality: Given  $k$  generated paraphrases  $Y = \{\mathbf{y}^1, \mathbf{y}^2 \dots \mathbf{y}^k\}$  for each input sentence in the test set, we select  $\hat{\mathbf{y}}^{best}$  that achieves the best (oracle) sentence-level score with the ground truth paraphrase  $\mathbf{y}$ . The corpus level evaluation is performed using pairs  $(\hat{\mathbf{y}}^{best}, \mathbf{y})$ .
2. Diversity: We calculate BLEU orWER be
If it continues to rain I will carry an umbrella
I will carry an umbrella if rain continues
Prior work (Wang et al., 2019; Li et al., 2019) has shown that such a transformer-based model provides a strong baseline and outperforms previous LSTM-based (Hasan et al., 2016) and VAE-based (Gupta et al., 2018) approaches.
<sup>6</sup>The score is computed using a weighted mean of the contextual similarity between individual words in the phrases, where the weights are determined by the corpus-level inverse-document frequency of the words. Details in the Appendix.
Model	oracle quality (over 10 sentences, no rejection) ↑	pairwise diversity (post-rejection)
BLEU	ROUGE-1	ROUGE-2	ROUGE-L	% rejected	self-BLEU ↓	self-WER ↑
copy-input	18.4	54.4	27.2	49.2	0	-	-
SCPN	21.3	53.2	30.3	51.0	40.6	35.9	63.4
Transformer seq2seq	32.8	63.1	41.4	63.3	12.7	50.7	35.4
+ diverse-decoding	24.8	56.8	33.2	56.4	21.3	34.2	58.1
SOW-REAP (LSTM)	27.0	57.9	34.8	57.5	31.7	46.2	53.9
SOW-REAP	30.9	62.3	40.2	61.7	15.9	38.0	57.9
Table 1: Quality and diversity metrics for the different models. Our proposed approach outperforms other diverse models (SCPN and diverse decoding) in terms of all the quality metrics. These models exhibit higher diversity, but with many more rejected paraphrases, indicating that these models more freely generate bad paraphrases.
tween all pairs  $(\mathbf{y}^i,\mathbf{y}^j)$  generated by a single model on a single sentence, then macro-average these values at a corpus-level.
In addition to these metrics, we use the paraphrase similarity model proposed by Wieting et al. (2017) to compute a paraphrase score for generated outputs with respect to the input. Similar to Iyyer et al. (2018), we use this score to filter out low quality paraphrases. We report on the rejection rate according to this criterion for all models. Note that our diversity metric is computed after filtering as it is easy to get high diversity by including nonsensical paraphrase candidates that differ semantically.
Table 1 outlines the performance of the different models. The results show that our proposed model substantially outperforms the SCPN model across all quality metrics. Furthermore, our LSTM model also beats the performance of the SCPN model, demonstrating that the gain in quality cannot completely be attributed to the use of transformers. The quality of our full model (with rearrangements) is also comparable to the quality of the vanilla seq2seq model (without rearrangements). This demonstrates that the inclusion of rearrangements from the syntax-based neural transducer do not hurt quality, while leading to a substantially improved diversity performance.