SlowGuess/ACL_2020_OCR huggingface dataset:
Amirhossein Kazemnejad†, Mohammadreza Salehi‡, Mahdieh Soleymani Baghshah‡
†Iran University of Science and Technology, ‡Sharif University of Technology a_kazemnejad@comp.iust.ac.ir, mrezasalehi@ce.sharif.edu, soleymani@sharif.edu
Neural sequence to sequence text generation has been proved to be a viable approach to paraphrase generation. Despite promising results, paraphrases generated by these models mostly suffer from lack of quality and diversity. To address these problems, we propose a novel retrieval-based method for paraphrase generation. Our model first retrieves a paraphrase pair similar to the input sentence from a pre-defined index. With its novel editor module, the model then paraphrases the input sequence by editing it using the extracted relations between the retrieved pair of sentences. In order to have fine-grained control over the editing process, our model uses the newly introduced concept of Micro Edit Vectors. It both extracts and exploits these vectors using the attention mechanism in the Transformer architecture. Experimental results show the superiority of our paraphrase generation method in terms of both automatic metrics, and human evaluation of relevance, grammaticality, and diversity of generated paraphrases.
Paraphrases are texts conveying the same meaning while using different words (Bhagat and Hovy, 2013). Paraphrase generation is an important task in Natural Language Processing (NLP) that has many applications in other down-stream tasks, such as text summarization, question answering, semantic parsing, and information retrieval (Cao et al., 2017; Fader et al., 2014; Berant and Liang, 2014).
Early works on paraphrasing mostly investigated rule-based or statistical machine translation approaches to this task (Bannard and Callison-Burch, 2005). With the recent advances of neural sequence-to-sequence (Seq2Seq) framework in different NLP tasks, especially in machine translation, an increasing amount of literature have also applied
Figure 1: An overview of the proposed model. This model retrieves the most similar paraphrase pair to the input  $x$  from the training corpus (Retriever), computes a set of edit vectors  $[M, z]$  based on the retrieved pair (Edit Provider), and applies these edits to the input sequence  $x$  to generate its paraphrase (Edit Performer).
Seq2Seq models to the task of paraphrase generation (Prakash et al., 2016; Gupta et al., 2018; Li et al., 2018).
Although the proposed Seq2Seq methods for paraphrase generation have shown promising results, they are not yet as dominant as their counterparts used in neural machine translation. The main reason is that the available training data for paraphrasing is scarce and domain-specific (Wang et al., 2019). In fact, the necessity to generate sequences from scratch, which is a major drawback of traditional Seq2Seq models (Guu et al., 2018), magnifies itself when dealing with scarce training data. Thus, one can expect that the model would not be trained well and consequently, would not be able to generate diverse outputs.
Although retrieval-based text generation has
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6010-6021 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
been evaluated recently in Guu et al. (2018); Hashimoto et al. (2018); Wu et al. (2019) as a remedy for this problem, to the best of our knowledge, there is no previous study exploring the usage of this approach in paraphrase generation. Moreover, none of the existing works in the realm of retrieval text generation, such as Guu et al. (2018); Wu et al. (2019); Hashimoto et al. (2018), focuses on learning how to extract edits from the retrieved sentences. Indeed, Guu et al. (2018); Wu et al. (2019) computes a single edit vector heuristically through concatenating the weighted sum of the inserted word embeddings and the weighted sum of deleted word embeddings. Moreover, Hashimoto et al. (2018) only focuses on improving the retrieving stage and uses a standard Seq2Seq model to edit the retrieved sentence.
In this paper, we present an effective retrieval-based approach to paraphrase generation by proposing a novel editor module. Our method can be summarized as follows: Given an input sentence  $x$ , the model first retrieves a similar sentence  $p$  and its associated paraphrase  $q$  from the training data. Then, by getting  $x$  and  $(p,q)$ , the editor both learns how to extract the fine-grained relations between  $p$  and  $q$  as a set of edits, and also when and how to use these extracted edits to paraphrase  $x$ . By incorporating the retrieved pairs into the editing process, we invigorate our model with a non-parametric memory, which enables it to produce non-generic and more diverse outputs. Both the retriever and editor components of our method are modeled by deep neural networks. We employ the Transformer architecture (Vaswani et al., 2017) as the backbone of our model, and use its attention mechanism as an effective tool to apply edits in a selective manner. Our main contributions are:
- We propose the Fine-grained Sample-based Editing Transformer (FSET) model. It contains a novel editor that can be used in a retrieval-based framework for paraphrase generation. This editor learns how to discover the relationship between a pair of paraphrase sentences as a set of edits, and transforms the input sentence according to these edits. It is worth noting that the set of edits is learned in an end-to-end manner as opposed to Guu et al. (2018); Wu et al. (2019) that compute the edit vector heuristically.
- For the first time, we utilize the Transformer
as an efficient fully-attentional architecture for the task of retrieval-based text generation.
- Experimentally, we compare our method with the recent paraphrase generation methods, and also with the retrieval-based text generation methods that have been introduced recently. Both of the quantitative and qualitative results show the superiority of our model.
2.1 Neural paraphrase generation
Prakash et al. (2016) was the first work that adapted a neural approach to paraphrase generation with a residual stacked LSTM network. Gupta et al. (2018) combined a variational auto-encoder with a Seq2Seq LSTM model to generate multiple paraphrases for a given sentence. Li et al. (2018) proposed a model in which a generator is first trained on the paraphrasing dataset, and then is fine-tuned by using reinforcement learning techniques. Cao et al. (2017) utilized separate decoders for copying and rewriting as the two main writing modes in paraphrasing. Mallinson et al. (2017) addressed paraphrasing with bilingual pivoting on multiple languages in order to better capture different aspects of the source sentence. Iyyer et al. (2018) proposed a method to generate syntactically controlled paraphrases and use them as adversarial examples. Chen et al. (2019) addressed the same problem, but the syntax is controlled by a sentence exemplar. Kajiwara (2019) proposed a model that first identifies a set of words to be paraphrased, and then generates the output by using a pre-trained paraphrase generation model. Wang et al. (2019) proposed a Transformer-based model that utilizes structured semantic knowledge to improve the quality of paraphrases. Kumar et al. (2019) modified the beam search algorithm with a sub-modular objective function to make the generated set of paraphrases syntactically diverse. Li et al. (2019) decomposed paraphrasing into sentential and phrasal levels and employed separate Transformer-based models for each of these levels. Fu et al. (2019) decomposes paraphrasing into two steps: content planning and surface realization, and improves the interpretability of the first step by incorporating a latent bag of words model.
2.2 Retrieval-based text generation
Retrieval-based text generation has received much attention in the last few years. Song et al. (2016);
Wu et al. (2019) augmented Seq2Seq generation-based models with retrieval frameworks to make the dialog responses more meaningful and nongeneric. Gu et al. (2017) utilized a search engine to retrieve a set of source-translation pairs from the training corpus, both at train and test time, and use them as a guide to translate an input query. Guu et al. (2018) proposed the neural editor model for unconditional text generation, which produces a new sentence by editing a retrieved prototype using an edit vector. Hashimoto et al. (2018) proposed a task-specific retriever using the variational framework to generate complex structured outputs, such as Python code. This work, however, does not have any novelty in the editor's architecture and uses a standard Seq2Seq model with attention and copy mechanism (Hashimoto et al., 2018).
Let  $\mathcal{D} = \{x_n, y_n\}_{n=1}^N$  denotes a dataset where  $x_n$  is a sequence of words, and  $y_n$  is its target paraphrase. In the paraphrasing task, our goal is to find the set of parameters of the model that maximizes  $\prod_{n=1}^{N} p_{\text{model}}(y_n | x_n)$ . Figure 1 illustrates the overview of our proposed model which is composed of a Retriever and an Editor. Given an input sequence  $x$ , the retriever first finds a paraphrase pair  $(p, q)$  from the training corpus based on similarity of  $x$  and  $p$ . Then, the editor utilizes the retrieved pair  $(p, q)$  to paraphrase  $x$ . We discuss the details in the following subsections.
3.1 Retriever
The goal of the retriever module is to select the paraphrase pairs (from the training corpus) that are similar to the input sequence  $x$ . To do that, the retriever finds a neighborhood set  $\mathcal{N}(x)$  consisting of the  $K$  most similar source sentences  $\{p_k\}_{k=1}^K$  to  $x$  and their associated paraphrases  $\{q_k\}_{k=1}^K$  ( $K$  is a hyper-parameter of the model). To measure similarity of sentences, we first embed them employing the pre-trained transformer-based sentence encoder proposed by Cer et al. (2018). The similarity is then calculated using cosine similarity measure in the resulted embedding space. We call this retriever as General Retriever throughout the paper. Note that using a pre-trained retriever can help us to alleviate the scarcity problem of the training data available for paraphrasing<sup>1</sup>.
In order to search for the similar sentences to an input sequence efficiently, we use the FAISS software package (Johnson et al., 2019) to create a fast search index from the sentences in the training corpus. We would also pre-compute the neighborhood set of each source sentence in the training set, so at the training time, our model just needs to sample one of the pairs in the neighborhood set uniformly and feed it as an input to the editor module. The probability of retrieving a pair can thus be stated as
$$
p ((p, q) | x) = \frac {1}{K} {\bf 1} [ (p, q) \in \mathcal {N} (x) ]. \qquad (1)
$$
Note that the same procedure also holds for the test time, and the retriever computes  $\mathcal{N}(x)$  so the model can sample any one of the pairs in  $\mathcal{N}(x)$  to generate the output based on that pair.
3.2 Editor
To edit a sentence according to a retrieved pair, we propose an editor module consisting of two components: 1) Edit Provider and 2) Edit Performer. The Edit Provider computes a set of edit vectors based on the retrieved pair of sentences  $(p,q)$ . After that, the Edit Performer rephrases the input sequence  $x$  by utilizing this prepared set of edits.
3.2.1 Edit Provider
This part of the editor extracts the edits from the retrieved pair as a set of vectors which we call Micro Edit Vectors (MEVs). MEVs are responsible for encoding the information about fine-grained edits that transform  $p$  into  $q$ . Each one of the MEVs represents the most plausible soft alignment between a token in  $p$  and the semantically relevant parts in  $q$ :
$$
M = \left\{m _ {i} := \text {s m a l l e d i t a p p l i e d o n} p _ {i} \mid 1 \leq i \leq l \right\}
$$
where  $l$  is the length of  $p$ .
Figure 2: The general scheme of computing a MEV corresponding to a token of  $p$ .
Figure 2 presents, in schematic form, the procedure of computing one MEV. For each arbitrary
1Pre-trained model is available at https://tfhub.dev/google/universal-sentence-encoder-large/3
6012
token of  $p$ , such as  $p_i$ , we intend to compute a MEV that encodes the edit corresponding to  $p_i$  using attention over  $q$ . Then, given  $p_i$  as the source of the edit, and the attention's result as the target, we concatenate their representations and feed it as the input to a neural network, which calculates  $m_i$  as the corresponding edit vector. To make this process differentiable and parallelizable, we use a fully-attentional architecture consisting of two main sub-modules: 1) Edit Encoder and 2) Target Encoder. Figure 3 shows the overview of the Edit Provider.
In this model, at first, a context-aware representation  $R_{q} = [r_{q}^{1},\dots,r_{q}^{k}]$  of the sequence  $q$  is computed using the Target Encoder which is the encoder sub-graph of the Transformer architecture (Vaswani et al., 2017). The Edit Encoder is also the encoder of the Transformer model, but, with an extra multi-head attention over  $R_{q}$ . This module outputs a vector that encodes the most semantically relevant parts of  $q$  to  $p_i$ . After that, the MEVs, i.e.  $m_{i}\mathrm{s}$ , are computed by feeding these vectors one by one into a single dense layer (with the  $\tanh(\cdot)$  activation function). By setting the output dimension of the dense layer to be smaller than the dimension of the word embeddings, we introduce a bottleneck, which hinders the Edit Encoder from copying  $q$  directly.
Figure 3: Architecture of Edit Provider. The Edit Encoder uses multi-head attention on  $R_{q}$  to select the target of edit for each token of  $p$ . Note that by prepending [AGR] to  $p$ , we can encode all of the MEVS into a single edit vector  $z_{p\rightarrow q}$ .
Finally, all of the MEVs are aggregated into a single vector  $z$  by leveraging a technique inspired by Devlin et al. (2019); we pretend a special token  $[AGR]$  to  $p$  in order to encode all the edits into a single vector  $z_{p\rightarrow q}$ . The intuition behind encoding into a single vector  $z_{p\rightarrow q}$  is to allow the model learn a global edit that can be applied to the whole sentence, in addition to the MEVs as local edits. We run the Edit Performer with the same parameters in the reverse direction, i.e. from  $q$  to  $p$ , to
Figure 4: Illustration of the Edit Performer generating the output token at  $t$ -th time step. Note that only one layer of the decoder is depicted and the layernorms are not shown for simplicity.
compute  $R_{p}$  and  $z_{q\to p}$ . The final edit vector  $z$  is then computed as
$$
z = \operatorname {L i n e a r} (z _ {p \rightarrow q} \oplus z _ {q \rightarrow p}),
$$
where Linear denotes a dense layer without activation and bias.
3.2.2 Edit Performer
The Edit Performer transforms the input sequence  $x = [x_{1}, \dots, x_{s}]$  to the final output  $\hat{y}$  using the edit vectors. We employ a fully-attentional Seq2Seq architecture composed of an encoder and a decoder for this part of the model.
The encoder of the Edit Performer has exactly the same architecture as the original encoder of the Transformer model and outputs a context-aware representation  $R_{x} = \{r_{x}^{i}\}_{i = 1}^{s}$  of the input sequence. For the decoder, we use a slightly modified version of the original Transformer's decoder. Indeed, the Transformer learns to model  $p(y|x)$ , while we would like to model a conditional setting  $p(y|x,(p,q))$ . Moreover, as mentioned in the description of the Edit Provider, the relation between  $p$  and  $q$  is encoded in MEVs  $M$  and the vector  $z$ . Therefore, in order to edit  $x$ , instead of using  $(p,q)$  directly, we only need  $M$  and  $z$  to specify the edits, and the sentence  $p$  to identify the locations in  $x$  to which the edits should be applied. Thus, we aim to model  $p(y|x,p,M,z)$  with the Edit Performer.
Figure 4 depicts the architecture of the Edit Performer. To condition the generation process on
the edit vector  $z$ , we append it to each token of the decoder's input. To apply the edits in a fine-grained manner, we would like the model to attend to the most similar token of  $p$  and select the corresponding edit in MEVs  $M$  to be applied to the input sentence. Therefore, in addition to the input sequence representation  $R_{x}$ , the model also attends to MEVs  $M$  using an extra multi-head attention sub-layer which computes the representation
$$
h ^ {\prime} = _ {\mathrm {M u l t i H e a d A t t}} (\mathbf {Q}: h, \mathbf {K}: R _ {p}, \mathbf {v}: M),
$$
where  $h$  comes from the previous sub-layer and  $R_{p}$  is the context-aware representation of the retrieved sequence  $p$ , which is calculated by the Edit Provider. Hence, this sub-layer allows the model to apply edits only when the current context matches somewhere in  $p$ . Finally, we project  $h'$  (after applying the residual connection and the layernorm) using a fully-connected sub-layer and feed it to the above layer. For the last layer, a softmax activation is employed to predict the next token of the output.
During the training phase, our aim is to maximize the log likelihood objective
$$
\mathcal {L} = \sum_ {(x, y) \in \mathcal {D}} \log p (y | x). \tag {2}
$$
As we decompose the training procedure to two stages of retrieving and editing, we can rewrite  $p(y|x)$  as
$$
p (y | x) = \sum_ {(p, q) \in \mathcal {D}} p (y | x, (p, q)) p ((p, q) | x). \tag {3}
$$
Substituting Eq. 1 into Eq. 3 and then inserting the resulted  $p(y|x)$  into Eq. 2 yields the following formulation for the log likelihood:
$$
\mathcal {L} = \sum_ {(x, y) \in \mathcal {D}} \log (\frac {1}{K} \sum_ {(p, q) \in \mathcal {N} (x)} p (y | x, (p, q))).
$$
We train our model by maximizing the following lower bound of the log likelihood (obtained by Jensen's inequality):
$$
\mathcal {L} \geq \mathcal {L} ^ {\prime} = \frac {1}{K} \sum_ {(x, y) \in \mathcal {D}} \sum_ {(p, q) \in \mathcal {N} (x)} \log p (y | x, (p, q)).
$$
Note that  $p(y|x,(p,q)) = p_{\theta}(y|x,p,m_{\phi}(p,q),z_{\phi}(p,q)),$  where  $\theta$  denotes the parameters of the Edit Performer and  $\phi$
shows the parameters of the Edit Provider. Thus, we solve the following optimization problem:
$$
\theta^ {*}, \phi^ {*} = \operatorname * {a r g m a x} _ {\theta , \phi} \mathcal {L} ^ {\prime} (\theta , \phi).
$$
Except for the retriever which is a pre-trained component of our model, other components are fully coupled and trained together. To prevent the model from ignoring the information coming from the retrieval pathway during the training procedure (i.e. ignoring the edit vectors extracted from the retrieved pair), we use a simple yet effective trick; we manually add extra  $(x,y)$  pairs to  $\mathcal{N}(x)$  proportionate to the number of retrieved pairs  $K$  so the presence of  $y$  as the exact ground-truth paraphrase encourages the model to use the retrieved pairs more. Please refer to A.1 for further details.
In this section, we empirically evaluate the performance of our proposed method in the task of paraphrase generation, and compare it with various other methods, including previous state-of-the-art paraphrasing models.
We conduct experiments on two of the most frequently used datasets for paraphrase generation: the Quora question pair dataset and the Twitter URL paraphrasing corpus. For the Quora dataset, we only consider the paraphrase pairs. Similar to Li et al. (2018), we sample 100k, 30k, 3k instances for train, test, and validation sets, respectively. Twitter URL paraphrasing dataset consists of two subsets, one is labeled by human annotators, and the other is labeled automatically, thus, it is noisier compared to the Quora dataset. Similar to Li et al. (2018), we sample 110k instances from automatically labeled part as our training set and two non-overlapping subsets of 5k and 1k instances from the part annotated by humans for the test and validation sets, respectively. As in Li et al. (2018, 2019), we truncate sentences in both of the datasets to 20 tokens.
Hyperparameter	Edit Performer	Edit Provider
Hidden dimension	64	64
# Layers	6	4
# Heads	8	4
MEV dimension mi	-	40
Edit vector z dimension	-	64
Table 1: Settings of the Model
We compare our method with both the existing paraphrasing methods that are not retrieval-based, and also with the existing or newly created retrieval-based text generation methods which we adapt for paraphrasing:
- Non-retrieval paraphrasing methods:
- Residual LSTM (Prakash et al., 2016) which is the first Seq2Seq model proposed for paraphrase generation,
- RbM (Li et al., 2018) that fine-tunes a paraphrase generation model using reinforcement learning,
- Transformer (Vaswani et al., 2017) which is a Seq2Seq model relying entirely on attention mechanism,
- DNPG (Li et al., 2019) that decomposes paraphrasing to sentential and phrasal levels and utilizes separate Transformers for each level,
- DiPS (Kumar et al., 2019) which aims to generate diverse paraphrases by adopting a novel approach in the decoding stage instead of beam search.
The latter two of the above list have been reported as the state-of-the-art models in paraphrase generation (Kumar et al., 2019; Li et al., 2019).
- Retrieval-based models: We compare our method with one existing retrieval-based text generation model and two other combinational methods that we create by ourselves:
- Seq2Seq+Ret which is an extended version of Seq2Seq Residual LSTM. This model conditions the generation process at each time step on an edit vector encoding the differences between the retrieved sentences  $p$  and  $q$ . To make the comparison fair, we use the General Retriever (introduced in the Retriever subsection of the Proposed Approach Section) to find  $(p,q)$ . The edit vector for this pair is also computed by concatenating the sum of inserted word embeddings with the sum of deleted word embeddings as it is stated by Guu et al. (2018).
- RaE that is proposed by Hashimoto et al. (2018) as a method with an in-domain retriever. The editor of this model is a Seq2Seq LSTM equipped with attention mechanism
over the input  $x$ , and copy mechanism over the retrieved pair  $p$  and  $q$ .
- CopyEditor+Ret which is composed of the editor of Hashimoto et al. (2018), and the General Retriever. We compare FSET with this baseline model to further evaluate the role of our proposed editor.
4.3 Experimental settings
Table 1 shows the settings of our model. We select the hyperparameters suggested by Li et al. (2018) for the LSTM-based Seq2Seq baselines, and the hyperparameters mentioned by Li et al. (2019) for the Transformer-based baselines. It is worth noting that our model's size w.r.t. the number of parameters is approximately  $\frac{1}{2}$  of the baseline LSTM's size and  $\frac{1}{5}$  of the baseline Transformer's size. The newly created retrieval-based baselines have the same hidden size and the same number of layers as the non-retrieval models. For the Seq2Seq+Ret model, we keep the ratio of hidden size to the edit vector dimension same as the reported ratio in Guu et al. (2018). We train all of the models for  $100k$  iterations, and choose the best version based on their validation loss after training. We set the batch size to 128 and the vocabulary size to 8k in all of the experiments. The embeddings are also trained from scratch. In all of the experiments on the retrieval-based methods, the hyper-parameter  $K$  is set to 1. However, results for different values of  $K$  are also reported in A.2. During the decoding stage, we use beam search to generate a set of outputs. In order to select the final output, an approach similar to Gupta et al. (2018) is used which chooses the most lexically similar sentence to the input where the similarity is calculated based on the Jaccard measure.
4.4 Results and analysis
We compare different methods using BLEU (Papineni et al., 2002), ROUGE (Lin, 2004), and METEOR (Banerjee and Lavie, 2005) as the most common metrics for automatic evaluation of paraphrase generation methods. Table 2 summarizes the results of different methods. These results indicate that our model outperforms the previous state-of-the-art models in terms of all of the metrics.