SlowGuess/ACL_2020_OCR huggingface dataset:
Xianggen Liu $^{1}$  Lili Mou $^{2}$  Fandong Meng $^{3}$  Hao Zhou $^{4}$  Jie Zhou $^{3}$  Sen Song $^{1}$
$^{1}$ Laboratory for Brain and Intelligence and Department of Biomedical Engineering, Tsinghua University
$^{2}$ Department of Computing Science, University of Alberta; Alberta Machine Intelligent Institute (Amii)
$^{3}$ Pattern Recognition Center, WeChat AI, Tencent Inc,  ${ }^{4}$ ByteDance AI Lab
liuxg16@mails.tsinghua.edu.cn, doublepower.mou@gmail.com
{fandongmeng,withtomzhou}@tencent.com
zhouhao.nlp@bytedance.com, songsen@tsinghua.edu.cn
We propose UPSA, a novel approach that accomplishes Unsupervised Paraphrasing by Simulated Annealing. We model paraphrase generation as an optimization problem and propose a sophisticated objective function, involving semantic similarity, expression diversity, and language fluency of paraphrases. UPSA searches the sentence space towards this objective by performing a sequence of local edits. We evaluate our approach on various datasets, namely, Quora, Wikianswers, MSCOCO, and Twitter. Extensive results show that UPSA achieves the state-of-the-art performance compared with previous unsupervised methods in terms of both automatic and human evaluations. Further, our approach outperforms most existing domain-adapted supervised models, showing the generalizability of UPSA. $^{1}$
Paraphrasing aims to restate one sentence as another with the same meaning, but different wordings. It constitutes a corner stone in many NLP tasks, such as question answering (Mckeown, 1983), information retrieval (Knight and Marcu, 2000), and dialogue systems (Shah et al., 2018). However, automatically generating accurate and different-appearing paraphrases is a still challenging research problem, due to the complexity of natural language.
Conventional approaches (Prakash et al., 2016; Gupta et al., 2018) model the paraphrase generation as a supervised encoding-decoding problem, inspired by machine translation systems. Usually, such models require massive parallel samples for training. In machine translation, for example, the WMT 2014 English-German dataset contains 4.5M sentence pairs (Neidert et al., 2014).
Figure 1: UPSA generates a paraphrase by a series of editing operations (i.e., insertion, replacement, and deletion). At each step, UPSA proposes a candidate modification of the sentence, which is accepted or rejected according to a certain acceptance rate (only accepted modifications are shown). Although sentences are discrete, we make an analogue in the continuous real  $x$ -axis where the distance of two sentences is roughly given by the number of edits.
However, the training corpora for paraphrasing are usually small. The widely-used Quora dataset<sup>2</sup> only contains 140K pairs of paraphrases; constructing such human-written paraphrase pairs is expensive and labor-intensive. Further, existing paraphrase datasets are domain-specific: the Quora dataset only contains question sentences, and thus, supervised paraphrase models do not generalize well to new domains (Li et al., 2019). On the other hand, researchers synthesize pseudo-paraphrase pairs by clustering news events (Barzilay and Lee, 2003), crawling tweets of the same topic (Lan et al., 2017), or translating bi-lingual datasets (Wieting and Gimpel, 2017), but these methods typically yield noisy training sets, leading to low paraphrasing performance (Li et al., 2018).
As a result, unsupervised methods would largely benefit paraphrase generation as no parallel data are
$^{1}$ Code and data available at: https://github.com/Liuxg16/UPSA
$^{2}$ https://www.kaggle.com/c/quora-question-pairs
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 302-312 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
needed. With the help of deep learning, researchers are able to generate paraphrases by sampling from a neural network-defined probabilistic distribution, either in a continuous latent space (Bowman et al., 2016; Bao et al., 2019) or directly in the word space (Miao et al., 2019). However, the meaning preservation and expression diversity of those generated paraphrases are less "controllable" in such probabilistic sampling procedures.
To this end, we propose a novel approach to Unsupervised Paraphrasing by Simulated Annealing (UPSA). Simulated annealing (SA) is a stochastic searching algorithm towards an objective function, which can be flexibly defined. In our work, we design a sophisticated objective function, considering semantic preservation, expression diversity, and language fluency of paraphrases. SA searches towards this objective by performing a sequence of local editing steps, namely, word replacement, insertion, deletion, and copy. For each step, UPSA first proposes a potential editing, and then accepts or rejects the proposal based on sample quality. In general, a better sentence (higher scored in the objective) is always accepted, while a worse sentence is likely to be rejected, but could also be accepted (controlled by an annealing temperature) to explore the search space in a less greedy fashion. At the beginning, the temperature is usually high, and worse sentences are more likely to be accepted, pushing SA outside a local optimum. The temperature is cooled down as the optimization proceeds, making the model better settle down to some optimum. Figure 1 illustrates how UPSA searches an optimum in unsupervised paraphrase generation.
We evaluate the effectiveness of our model on four paraphrasing datasets, namely, Quora, Wikianswers, MSCOCO, and Twitter. Experimental results show that UPSA achieves a new state-of-the-art unsupervised performance in terms of both automatic metrics and human evaluation.
- We propose the novel UPSA framework that addresses Unsupervised Paraphrasing by Simulated Annealing.
- We design a searching objective function for paraphrasing that not only considers language fluency and semantic similarity, but also explicitly models expression diversity between a paraphrase and the input.
- We propose a copy mechanism as one of our search actions of simulated annealing to address
rare words.
- We achieve the state-of-the-art performance on four benchmark datasets compared with previous unsupervised paraphrase generators, largely reducing the performance gap between unsupervised and supervised paraphrasing. We outperform most domain-adapted paraphrase generators, and even a supervised one on the Wikians dataset.
In early years, paraphrasing was typically accomplished by exploiting linguistic knowledge (Mckeown, 1983; Ellsworth and Janin, 2007; Narayan et al., 2016) and statistical machine translation methods (Quirk et al., 2004; Dolan et al., 2004). Recently, deep neural networks have become a prevailing approach to text generation, where paraphrasing is often formulated as a supervised encoding-decoding problem, for example, using stacked residual LSTM (Prakash et al., 2016) and the Transformer model (Wang et al., 2019).
Unsupervised paraphrasing is an emerging research direction in the field of NLP. The variational autoencoder (VAE) can be intuitively applied to paraphrase generation in an unsupervised fashion, as we can sample sentences from a learned latent space (Bowman et al., 2016; Zhang et al., 2019; Bao et al., 2019). But the generated sentences are less controllable and suffer from the error accumulation problem in VAE's decoding phase (Miao et al., 2019). Roy and Grangier (2019) introduce an unsupervised model based on vector-quantized autoencoders (Van den Oord et al., 2017). But their work mainly focuses on generating sentences for data augmentation instead of paraphrasing itself.
Miao et al. (2019) use Metropolis-Hastings sampling (1953) for constrained sentence generation, achieving the state-of-the-art unsupervised paraphrasing performance. The main difference between their work and ours is that UPSA imposes the annealing temperature into the sampling process for better convergence to an optimum. In addition, we define our searching objective involving not only semantic similarity and language fluency, but also the expression diversity; we further propose a copy mechanism in our searching process.
Recently, a few studies have applied editing-based approaches to sentence generation. Guu et al. (2018) propose a heuristic delete-retrieve-produce component for a supervised sequence-to-sequence
(Seq2Seq) model. Dong et al. (2019) learn the deletion and insertion operations for text simplification in a supervised way, where their groundtruth operations are obtained by some dynamic programming algorithm. Our editing operations (insertion, deletion, and replacement) are the search actions of unsupervised simulated annealing.
Regarding discrete optimization/searching, a naive approach is by hill climbing (Edelkamp and Schroedl, 2011; Schumann et al., 2020; Kumar et al., 2020), which is in fact a greedy algorithm. In NLP, beam search (BS, Tillmann et al. 1997) is widely applied to sentence generation. BS maintains a  $k$ -best list in a partially greedy fashion during left-to-right (or right-to-left) decoding (Anderson et al., 2017; Zhou and Rush, 2019). By contrast, UPSA is local search with distributed edits over the entire sentence. Moreover, UPSA is able to make use of the original sentence as an initial state of searching, whereas BS usually works in the decoder of a Seq2Seq model and is not applicable to unsupervised paraphrasing.
In this section, we present our novel UPSA framework that uses simulated annealing (SA) for unsupervised paraphrasing. In particular, we first present the general SA algorithm and then design our searching objective and searching actions (i.e., candidate sentence generator) for paraphrasing.
3.1 The Simulated Annealing Algorithm
Simulated Annealing (SA) is an effective and general metaheuristic of searching, especially for a large discrete or continuous space (Kirkpatrick et al., 1983).
Let  $\mathcal{X}$  be a (huge) search space of sentences, and  $f(x)$  be an objective function. The goal is to search for a sentence  $x$  that maximizes  $f(x)$ . At a searching step  $t$ , SA keeps a current sentence  $x_{t}$ , and proposes a new candidate  $x_{*}$  by local editing. If the new candidate is better scored by  $f$ , i.e.,  $f(x_{*}) > f(x_{t})$ , then SA accepts the proposal. Otherwise, SA tends to reject the proposal  $x_{*}$ , but may still accept it with a small probability  $e^{\frac{f(x_{*}) - f(x_{t})}{T}}$ , controlled by an annealing temperature  $T$ . In other words, the probability of accepting the proposal is
$$
p (\mathrm {a c c e p t} | \mathrm {x} _ {*}, \mathrm {x} _ {t}, T) = \min  \left(1, e ^ {\frac {f (\mathrm {x} _ {*}) - f (\mathrm {x} _ {t})}{T}}\right). \tag {1}
$$
If the proposal is accepted,  $\mathrm{x}_{t + 1} = \mathrm{x}_*$  , or otherwise,  $\mathrm{x}_{t + 1} = \mathrm{x}_t$
Inspired by the annealing in chemistry, the temperature  $T$  is usually high at the beginning of searching, leading to a high acceptance probability even if  $\mathbf{x}_*$  is worse than  $\mathbf{x}_t$ . Then, the temperature is decreased gradually as the search proceeds. In our work, we adopt the linear annealing schedule, given by  $T = \max(0, T_{\mathrm{init}} - C \cdot t)$ , where  $T_{\mathrm{init}}$  is the initial temperature and  $C$  is the decreasing rate.
The high initial temperature of SA makes the algorithm less greedy compared with hill climbing, whereas the decreasing of temperature enables the algorithm to better settle down to a certain optimum.
Theoretically, simulated annealing is guaranteed to converge to the global optimum in a finite search space if the proposal and the temperature satisfy some mild conditions (Granville et al., 1994). Although such convergence may be slower than exhaustive search and the sentence space is, in fact, potentially infinite, simulated annealing is still a widely applied search algorithm, especially for discrete optimization. Readers may refer to Hwang (1988) for details of the SA algorithm.
3.2 Objective Function
Simulated annealing maximizes an objective function, which can be flexibly specified in different applications. In particular, our UPSA objective  $f(x)$  considers multiple aspects of a candidate paraphrase, including semantic preservation  $f_{\mathrm{sem}}$ , expression diversity  $f_{\mathrm{exp}}$ , and language fluency  $f_{\mathrm{flu}}$ . Thus, our searching objective is to maximize
$$
f (\mathrm {x}) = f _ {\text {s e m}} (\mathrm {x}, \mathrm {x} _ {0}) \cdot f _ {\text {e x p}} (\mathrm {x}, \mathrm {x} _ {0}) \cdot f _ {\text {f l u}} (\mathrm {x}), \tag {2}
$$
where  $\mathbf{x}_0$  is the input sentence.
Semantic Preservation. A paraphrase is expected to capture all the key semantics of the original sentence. Thus, we leverage the cosine function of keyword embeddings to measure if the key focus of the candidate paraphrase is the same as the input. Specifically, we extract the keywords of the input sentence  $\mathbf{x}_0$  by the Rake system (Rose et al., 2010) and embed them by GloVE (Pennington et al., 2014). For each keyword, we find the closest word in the candidate paraphrase  $\mathbf{x}_{*}$  in terms of the cosine similarity. Our keyword-based semantic preservation score is given by the lowest cosine similarity among all the keywords, i.e., the least matched keyword:
$$
f _ {\text {s e m , k e y}} \left(\mathrm {x} _ {*}, \mathrm {x} _ {0}\right) = \min  _ {e \in \text {k e y w o r d s} \left(\mathrm {x} _ {0}\right) j} \max  \left\{\cos \left(\boldsymbol {w} _ {*}, j, \boldsymbol {e}\right) \right\}, \tag {3}
$$
where  $w_{*,j}$  is the  $j$ th word in the sentence  $\mathbf{x}_*$ ;  $e$  is an extracted keyword of  $\mathbf{x}_0$ . Bold letters indicate embedding vectors.
In addition to keyword embeddings, we also adopt a sentence-level similarity function, based on Sent2Vec embeddings (Pagliardini et al., 2017). Sent2Vec learns  $n$ -gram embeddings and computes the average of  $n$ -grams embeddings as the sentence vector. It has been shown to be significant improvements over other unsupervised sentence embedding methods in similarity evaluation tasks (Pagliardini et al., 2017). Let  $\mathbf{x}_*$  and  $\mathbf{x}_0$  be the Sent2Vec embeddings of the candidate paraphrase and the input sentence, respectively. Our sentence-based semantic preservation scoring function is  $f_{\mathrm{sim,sen}}(\mathbf{x}_*, \mathbf{x}_0) = \cos(\mathbf{x}_*, \mathbf{x}_0)$ .
To sum up, the overall semantic preservation scoring function of UPSA is given by
$$
f _ {\text {s e m}} \left(\mathrm {x} _ {*}, \mathrm {x} _ {0}\right) = f _ {\text {s e m , k e y}} \left(\mathrm {x} _ {*}, \mathrm {x} _ {0}\right) ^ {P} \cdot f _ {\text {s e m , s e n}} \left(\mathrm {x} _ {*}, \mathrm {x} _ {0}\right) ^ {Q}, \tag {4}
$$
where  $P$  and  $Q$  are hyperparameters, balancing the importance of the two factors. Here, we use power weights because the scoring functions are multiplicative.
Expression Diversity. The expression diversity scoring function computes the lexical difference of two sentences. We adopt a BLEU-induced function to penalize the repetition of the words and phrases in the input sentence:
$$
f _ {\exp} \left(\mathrm {x} _ {*}, \mathrm {x} _ {0}\right) = \left(1 - \operatorname {B L E U} \left(\mathrm {x} _ {*}, \mathrm {x} _ {0}\right)\right) ^ {S}, \tag {5}
$$
where the BLEU score (Papineni et al., 2002) computes a length-penalized geometric mean of  $n$ -gram precision  $(n = 1, \dots, 4)$ .  $S$  coordinates the importance of  $f_{\mathrm{exp}}(\mathbf{x}_t, \mathbf{x}_0)$  in the objective function (2).
Language Fluency. Despite semantic preservation and expression diversity, the candidate paraphrase should be a fluent sentence by itself. We use a separately trained (forward) language model (denoted as  $\overrightarrow{\mathrm{LM}}$ ) to compute the likelihood of the candidate paraphrase as our fluency scoring function:
$$
f _ {\mathrm {f l u}} \left(\mathrm {x} _ {*}\right) = \prod_ {k = 1} ^ {k = l _ {*}} p _ {\overrightarrow {\mathrm {L M}}} \left(w _ {* , k} \mid w _ {* , 1}, \dots , w _ {* , k - 1}\right), \tag {6}
$$
where  $l_{*}$  is the length of  $\mathbf{x}_{*}$  and  $w_{*,1},\ldots ,w_{*,l}$  are words of  $\mathbf{x}_{*}$ . Here, we use a dataset-specific language model, trained on non-parallel sentences. Notice that a weighting hyperparameter is not
needed for  $f_{\mathrm{flu}}$ , because the relative weights of different factors in Eqn. (2) are given by the powers in  $f_{\mathrm{sem,key}}$ ,  $f_{\mathrm{sem,sen}}$ , and  $f_{\mathrm{exp}}$ .
3.3 Candidate Sentence Generator
As mentioned, simulated annealing proposes a candidate sentence, given by different search actions. Since each action yields a new sentence  $\mathbf{x}_*$  from  $\mathbf{x}_t$ , we call it a candidate sentence generator. While the proposal of candidate sentences does not affect convergence in theory (if some mild conditions are satisfied), it may largely influence the efficiency of SA searching.
In our work, we mostly adopt the word-level editing in Miao et al. (2019) as our searching actions, but we differ in sampling distributions and further propose a copy mechanism for editing.
At each step  $t$ , the candidate sentence generator randomly samples an editing position  $k$  and an editing operation namely, replacement, insertion, and deletion. For replacement and insertion, the candidate sentence generator also samples a candidate word. Let the current sentence be  $\mathbf{x}_t = (w_{t,1},\ldots ,w_{t,k - 1},w_k,w_{t,k + 1}\ldots ,w_{t,l_t})$ . If the replacement operation proposes a candidate word  $w_{*}$  for the  $k$ th step, the resulting candidate sentence becomes  $\mathrm{x}_{*} = (w_{t,1},\dots ,w_{t,k - 1},w_{*},w_{t,k + 1}\dots ,w_{t,l_{t}})$ . The insertion operation works similarly.
Here, the candidate word is sampled from a probabilistic distribution, induced by the objective function (2):
$$
p \left(w _ {*} \mid \cdot\right) = \frac {f _ {\text {s i m}} \left(\mathrm {x} _ {*}, \mathrm {x} _ {0}\right) \cdot f _ {\exp} \left(\mathrm {x} _ {*}, \mathrm {x} _ {0}\right) \cdot f _ {\text {f l u}} \left(\mathrm {x} _ {*}\right)}{Z}, \tag {7}
$$
$$
Z = \sum_ {w _ {*} \in \mathcal {W}} f _ {\text {s i m}} \left(\mathrm {x} _ {*}, \mathrm {x} _ {0}\right) \cdot f _ {\exp} \left(\mathrm {x} _ {*}, \mathrm {x} _ {0}\right) \cdot f _ {\text {f l u}} \left(\mathrm {x} _ {*}\right), \tag {8}
$$
where  $\mathcal{W}$  is the sampling vocabulary;  $Z$  is known as the normalizing factor (noticing our scoring functions are nonnegative). We observe that sampling from such objective-induced distribution typically yields a meaningful candidate sentence, which enables SA to explore the search space more efficiently.
It is also noted that sampling a word from the entire vocabulary involves re-evaluating (2) for each candidate word, and therefore, we also follow Miao et al. (2019) and only sample from the top- $K$  words given by jointly considering a forward language
Algorithm 1 UPSA
1: Input: Original sentence  $\mathbf{x}_0$
2: for  $t \in \{1, \dots, N\}$  do
3:  $T = \max \{T_{\mathrm{init}} - C \cdot t, 0\}$
4: Randomly choose an editing operation and a position  $k$
5: Obtain a candidate  $\mathbf{x}_{*}$  by candidate sentence generator
6: Compute the accepting probability  $p_{\mathrm{accept}}$  by Eqn. (1)
7: With probability  $p_{\mathrm{accept}}$ ,  $\mathrm{x}_{t + 1} = \mathrm{x}_*$
8: With probability  $1 - p_{\mathrm{accept}}$ ,  $\mathbf{x}_{t + 1} = \mathbf{x}_t$
9: end for
10: return  $\mathbf{x}_{\tau}$  s.t.  $\tau = \arg \max_{\tau \in \{1,\dots ,N\}}f(\mathbf{x}_{\tau})$
model and backward language model. The replacement operator, for example, suggests the top-  $K$  words vocabulary by
$$
\begin{array}{l} \mathcal {W} _ {t, \text {r e p l a c e}} = \operatorname {t o p -} K _ {w _ {*}} \left[ p _ {\overrightarrow {\mathrm {L M}}} \left(w _ {t, 1}, \dots , w _ {t, k - 1}, w _ {*}\right) \cdot \right. \\ \left. p _ {\overline {{\mathrm {L M}}}} \left(w _ {*}, w _ {t, k + 1}, \dots , w _ {t, l _ {t}}\right) \right]. \tag {9} \\ \end{array}
$$
For word insertion, the top-  $K$  vocabulary  $\mathcal{W}_{t,\mathrm{insert}}$  is computed in a similar way (except that the position of  $w_{*}$  is slightly different). Details are not repeated. In our experiments,  $K$  is set to 50.
Copy Mechanism. We observe that name entities and rare words are sometimes deleted or replaced during SA stochastic sampling. They are difficult to be recovered because they usually have a low language model-suggested probability.
Therefore, we propose a copy mechanism for SA sampling, inspired by that in Seq2Seq learning (Gu et al., 2016). Specifically, we allow the candidate sentence generator to copy the words from the original sentence  $\mathbf{x}_0$  for word replacement and insertion. This is essentially enlarging the top- $K$  sampling vocabulary with the words in  $\mathbf{x}_0$ , given by
$$
\widetilde {\mathcal {W}} _ {t, \mathrm {o p}} = \mathcal {W} _ {t, \mathrm {o p}} \cup \left\{w _ {0, 1}, \dots , w _ {0, l _ {0}} \right\} \tag {10}
$$
where  $\mathrm{op}\in \{\mathrm{replace,insert}\}$ . Thus,  $\widetilde{\mathcal{W}}_{t,\mathrm{op}}$  is the actual vocabulary from which SA samples the word  $w_{*}$  for replacement and insertion operation.
While such vocabulary reduces the proposal space, it works well empirically because other low-ranked candidate words are either irrelevant or make the sentence disfluent; they usually have low objective scores, and are likely to be rejected even if sampled.
3.4 Overall Optimization Process
We summarize our UPSA algorithm in Algorithm 1.
Given an input  $\mathbf{x}_0$ , UPSA searches from the sentence space to maximize our objective  $f(\mathbf{x})$ , which involves semantic preservation, expression diversity, and language fluency. UPSA starts from  $\mathbf{x}_0$  itself. For each step, it randomly selects a search action (namely, word insertion, deletion, and replacement) at a position  $k$  (Line 4); if insertion or replacement is selected, UPSA also proposes a candidate word, so that a candidate paraphrase  $\mathbf{x}_*$  is formed (Line 5). Then, UPSA computes an acceptance rate  $p_{\mathrm{accept}}$  based on the increment of  $f$  and the temperature  $T$  (Line 6). The candidate sentence  $\mathbf{x}_{t+1}$  for the next step becomes  $\mathbf{x}_t$  if the proposal is accepted, or remains  $\mathbf{x}_t$  if the proposal is rejected. Until the maximum searching iterations, we choose the sentence  $\mathbf{x}_\tau$  that yields the highest score.
Quora. The Quora question pair dataset (Footnote 2) contains 140K parallel paraphrases and additional 260K pairs of non-parallel sentences. We follow the unsupervised setting in Miao et al. (2019), where 3K and 20K pairs are used for validation and test, respectively.
Wikians. The original Wikians dataset (Fader et al., 2013) contains 2.3M pairs of question paraphrases from the Wikians website. Since our model only involves training a language model, we randomly selected 500K non-parallel sentences for training. For evaluation, we followed the same protocol as Li et al. (2019) and randomly sampled 5K for validation and 20K for testing. Although the exact data split in previous work is not available, our results are comparable to previous ones in the statistical sense.
MSCOCO. The MSCOCO dataset contains  $500\mathrm{K}+$  paraphrases pairs for  $\sim 120\mathrm{K}$  image captions (Lin et al., 2014). We follow the standard split (Lin et al., 2014) and the evaluation protocol in Prakash et al. (2016) where only image captions with fewer than 15 words are considered, since some captions are extremely long (e.g., 60 words).
Twitter. The Twitter URL paraphrasing corpus (Lan et al., 2017) is originally constructed for paraphrase identification. We follow the standard train/test split, but take  $10\%$  of the training data as the validation set. The remaining samples are used to train our language model. For the test set, we only consider sentence pairs that are labeled as "paraphrases." This results in 566 test cases.
4.2 Competing Methods and Metrics