SlowGuess/ACL_2020_OCR huggingface dataset:
It is worth noting that the models which have utilized copy mechanism, such as DNPG, RbM, RaE, and CopyEditor+Ret, generally outperform the other baselines. The Seq2Seq+Ret, i.e. the
Models	Quora	Twitter URL Paraphrasing
ROUGE-2	ROUGE-1	BLEU-4	BLEU-2	METEOR	ROUGE-2	ROUGE-1	BLEU-4	BLEU-2	METEOR
Residual LSTM (Prakash et al., 2016)	32.71	59.69	24.56	38.52	29.39	27.94	41.77	25.92	32.13	24.88
Seq2Seq+Ret (Ours)	32.71	60.83	25.23	42.71	32.51	21.56	40.18	20.11	31.58	22.38
DiPS (Kumar et al., 2019)	31.77	59.79	25.37	40.35	29.28	23.67	43.64	27.66	37.92	25.69
Transformer (Vaswani et al., 2017)	34.23	61.25	30.38	42.91	34.65	29.55	44.53	32.14	40.34	28.26
DNPG (Li et al., 2019)2	37.75	63.73	25.03	-	-	-	-	-	-	-
RbM (Li et al., 2018)2	38.11	64.39	-	43.54	32.84	24.23	41.87	-	44.67	19.97
RaE (Hashimoto et al., 2018)	35.07	62.71	29.22	46.21	29.92	31.53	47.55	34.16	44.33	30.09
CopyEditor+Ret (Ours)	35.59	62.93	29.78	46.55	35.56	27.35	45.54	28.06	40.30	26.93
FSET (Ours)	39.55	66.17	33.46	51.03	38.57	32.04	49.53	34.62	46.35	31.67
retrieval-based Residual LSTM, shows an improvement over Residual LSTM on Quora dataset. However, this is not the case on the Twitter dataset and we hypothesize that it is due to uncommon texts in this corpus (i.e. informal text with hashtags and abbreviated words), on which the General Retriever has not been trained. Therefore, a pretrained retriever cannot help in this case. The CopyEditor+Ret model which incorporates a more powerful editor than Seq2Seq+Ret shows better results than both of the Residual LSTM and Seq2Seq+Ret. However, a phenomenon similar to what was stated for Seq2Seq+Ret is also observed for this model on the Twitter dataset. The RaE model with the same editor as CopyEditor but with a supervised (task-specific) retriever leads to near state-of-the-art results. This indicates the role of the supervised task-specific retriever used in RaE, especially in the results on Twitter dataset. The superiority of our method over RaE in all of the metrics could be a sign of the effectiveness of our proposed editor module. Although our model uses the General Retriever, it still outperforms all other methods even on the Twitter dataset. It is worth mentioning that we can replace the General Retriever in our method with other retrievers like supervised task-specific ones to improve the results even more. Moreover, it is worth noting that our model that is only based on the Transformer architecture and the General Retriever (that is not required to be trained in each domain) needs much less training time than RaE.
4.5 Human evaluation
As there is no appropriate automatic metric for evaluating the diversity and novelty of generated sentences, we use human evaluation to assess the performance of our model qualitatively. We
Table 2: Results of the different models on two paraphrasing datasets.
Models	Grammar	Coherency
Score	κ	Score	κ
DiPS (Kumar et al., 2019)	3.97	0.253	2.55	0.476
RaE (Hashimoto et al., 2018)	4.70	0.286	3.90	0.483
FSET (Ours)	4.70	0.394	4.22	0.528
Table 3: Human evaluation on Quora dataset.
Figure 5: Results of the one-on-one human evaluation (second experiment). Annotators decide "Tie" when the outputs of the two models have the same quality in their opinion.
compare our method with two other methods: 1) RaE (Hashimoto et al., 2018) as a retrieval-based method adapted for paraphrasing, and 2) DiPS (Kumar et al., 2019) as a paraphrasing model which generates semantically diverse outputs by adopting a novel approach instead of beam search during the decoding stage. We choose these models as we would like to compare our method both with a state-of-the-art retrieval-based method and with a method that can generate diverse outputs. It must be noted that many of the recent methods in Table 2 are not able to generate diverse outputs.
We first select 100 sentences randomly from the test set of Quora dataset. Then, for each model, three paraphrases are generated for each one of the sentences, and these three outputs are considered as a paraphrase group. We aggregate and shuffle these paraphrase groups and ask six human annotators to
Results are directly reported from Li et al. (2018, 2019) on the same dataset and settings.
evaluate them in two scenarios.
In the first scenario, we ask the human annotators to score the outputs individually based on the following two criteria: 1) Grammar and fluency, 2) Consistency and coherency. Similar to Li et al. (2018), we use a 5-scale rating for each criterion. Table 3 presents the results. As can be seen, our model generally outperforms the other methods. Although RaE and our model can both produce grammatically correct outputs, the consistency and coherency for the outputs of our method is much better. Moreover, the inter-annotator agreement measured by Cohen's kappa  $\kappa$  shows fair or intermediate agreement between raters assessing the models.
Since directly scoring diversity and novelty of one paraphrase group is not simple even for humans, in the second scenario, we ask the annotators to make one-on-one comparisons on the groups of generated paraphrases. In other words, for each pair of the models, they have to decide which model produces better outputs for each one of the sentences (Ties are also allowed). Figure 5 depicts the one-on-one diversity evaluation. Our method and RaE both outperform DiPS, probably due to their retrieval-based nature. Moreover, this figure reveals that our method can generate significantly better outputs compared to RaE. We believe the reason is that RaE's editor is not as properly designed as our editor module. We explicitly inject the paraphrasing patterns found in the neighboring paraphrases into the Edit Performer which helps it to generate more diverse paraphrases. Please refer to the A.3 for some further details on the experiments.
4.6 Case study
Table 4 shows some examples of the paraphrases generated by our model. A common pattern among the output paraphrases is that the model has combined different parts of the input sentence and the retrieved target sentence to create a grammatically correct paraphrase. Furthermore, the model is able to change the syntactic form of words correctly, or replace them with their synonyms, as it is the case for "via" and "by using", and "is" and "are". The fact that the model can apply these changes in different levels of granularity can indicate that the idea of editing based on MEVs works properly.
(1) x	Why do people ask such questions here on Quora which could be easily found on the internet ?
p	Why do people ask questions on Quora that could simply be googled?
q	why do people ask questions on quora that are easily answerable via a quick internet search ?
y'	why do people ask questions on quora that could be easily answered by using a quick internet search ?
(2) x	Who can the Democrats effectively run against Trump in 2020 ?
p	Who will lead Trump 's movement after the election ?
q	Who do you think is most likely to lead the Trump movement after 2016 ?
y'	who do you think is the most likely democrat to run against trump in 2020 ?
(3) x	What should I do when someone marks my question as needing improvement?
p	Even though most of my asked questions are in well written form, they are usually marked as needing improvement. What should I do ?
q	Most of my questions are marked as needing improvement. Where can I seek professional help ?
y'	what should I do if my question is marked as needing improvement ?
(4) x	What is the best way to make money as a 15 year old ?
p	What are the best ways to make money as a teenager ?
q	I 'm a teen , how can I make some money online ?
y'	I 'm a 15 year old , how can I make money ?
Table 4: Example outputs of the model. Red phrases are extracted from the retrieved target sentence  $q$ , and blue parts are extracted from the source sentence  $x$ . Phrases that appear with slight changes in semantics or syntax in the output are made bold. The sentences are annotated manually for better readability.
Model Variant	ROUGE-2	ROUGE-1	BLEU-4	BLEU-2
Jaccard Retriever	38.52	65.47	31.72	48.83
No edit vector z	38.31	65.44	30.40	47.77
No Attention on MEVs M	39.36	65.72	29.73	46.66
Table 5: Ablation study
4.7 Model Ablation
In order to further evaluate the role of each module in our model, we train and assess different variants of it where in each variant, a key component has been replaced by an alternative simpler one:
- Jaccard Retriever: The retriever of our model is replaced by a simple retriever that selects neighbor sentences using the Jaccard similarity metric.
- No edit vector  $z$ : A variant in which we do not condition the Transformer in the Edit Performer on the aggregated edit vector  $z$ , and edit the source sentence merely based on MEVs.
- No Attention on MEVs: In this variant of our model, the Transformer in the Edit Performer is not conditioned on MEVs, and the source sentence is edited based on only  $z$ .
6017
We train all of these variants on the Quora paraphrasing dataset. Table 5 shows the results of these models. As it is seen, the model which uses the Jaccard similarity measure performs worse than the original model with the General Retriever. Nonetheless, the results of this version explains that even the combination of our editor module with this simple retriever outperforms previous state-of-the-art methods. This indicates that our proposed editor can distinguish whether the extracted edits are plausible enough to be applied to the input sentence. Moreover, the results show that both eliminating  $z$  and  $M$  from our editor decrease its performance. In other words, both conditioning on  $z$  as the aggregated edit at each step of generation and the attention on MEVs  $M$  help the proposed editor.
In this paper, we proposed a retrieval-based paraphrase generation model which includes a novel fully-attentional editor. This editor learns how to extract edits from a paraphrase pair and also when and how to apply these edits to a new input sentence. We also introduced the new idea of Micro Edit Vectors, where each one of these vectors represents a small edit that should be applied to the source sentence to get its paraphrase. We incorporated Transformer modules in our editor and augmented them with attention over Micro Edit Vectors. The proposed model outperforms the previous state-of-the-art paraphrase generation models in terms of both automatic metrics and human evaluation. Moreover, the outputs show that our model is able to produce paraphrases by editing sentences in a fine-grained manner using the idea of MEVs. In future work, we intend to adapt our editor module for other learning tasks with both the structured input and structured output.
We thank anonymous reviewers for their detailed feedback and suggestions.
Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pages 65-72, Ann Arbor, Michigan. Association for Computational Linguistics.
Colin Bannard and Chris Callison-Burch. 2005. Paraphrasing with bilingual parallel corpora. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL'05), pages 597-604, Ann Arbor, Michigan. Association for Computational Linguistics.
Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1415-1425, Baltimore, Maryland. Association for Computational Linguistics.
Rahul Bhagat and Eduard Hovy. 2013. Squibs: What is a paraphrase? Computational Linguistics, 39(3):463-472.
Ziqiang Cao, Chuwei Luo, Wenjie Li, and Sujian Li. 2017. Joint copying and restricted generation for paraphrase. In Proceedings of the Thirty-First AAAI Conference on Artificial Intelligence, AAAI17, pages 3152-3158. AAAI Press.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder. CoRR, abs/1803.11175.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019. Controllable paraphrase generation with a syntactic exemplar. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5972-5984, Florence, Italy. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Anthony Fader, Luke Zettlemoyer, and Oren Etzioni. 2014. Open question answering over curated and extracted knowledge bases. In Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '14, pages 1156-1165, New York, NY, USA. ACM.
Yao Fu, Yansong Feng, and John P Cunningham. 2019. Paraphrase generation with latent bag of words. In Advances in Neural Information Processing Systems 32, pages 13645-13656. Curran Associates, Inc.
Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor O. K. Li. 2017. Search engine guided nonparametric neural machine translation. CoRR, abs/1705.07267.
Ankush Gupta, Arvind Agarwal, Prawaan Singh, and Piyush Rai. 2018. A deep generative framework for paraphrase generation. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, AAAI18, pages 5149-5156. AAAI Press.
Kelvin Guu, Tatsunori B. Hashimoto, Yonatan Oren, and Percy Liang. 2018. Generating sentences by editing prototypes. Transactions of the Association for Computational Linguistics, 6:437-450.
Tatsunori B. Hashimoto, Kelvin Guu, Yonatan Oren, and Percy Liang. 2018. A retrieve-and-edit framework for predicting structured outputs. In Proceedings of the 32Nd International Conference on Neural Information Processing Systems, NIPS'18, pages 10073-10083, USA. Curran Associates Inc.
Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1875-1885, New Orleans, Louisiana. Association for Computational Linguistics.
Jeff Johnson, Matthijs Douze, and Herve Jegou. 2019. Billion-scale similarity search with gpus. IEEE Transactions on Big Data, page 11.
Tomoyuki Kajiwara. 2019. Negative lexically constrained decoding for paraphrase generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6047-6052, Florence, Italy. Association for Computational Linguistics.
Ashutosh Kumar, Satwik Bhattachamishra, Manik Bhandari, and Partha Talukdar. 2019. Submodular optimization-based diverse paraphrasing and its effectiveness in data augmentation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3609-3619, Minneapolis, Minnesota. Association for Computational Linguistics.
Zichao Li, Xin Jiang, Lifeng Shang, and Hang Li. 2018. Paraphrase generation with deep reinforcement learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3865-3878, Brussels, Belgium. Association for Computational Linguistics.
Zichao Li, Xin Jiang, Lifeng Shang, and Qun Liu. 2019. Decomposable neural paraphrase generation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3403-3414, Florence, Italy. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2017. Paraphrasing revisited with neural machine translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics: Volume 1, Long Papers, pages 881-893, Valencia, Spain. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Aaditya Prakash, Sadid A. Hasan, Kathy Lee, Vivek Datla, Ashequl Qadir, Joey Liu, and Oladimeji Farri. 2016. Neural paraphrase generation with stacked residual LSTM networks. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 2923-2934, Osaka, Japan. The COLING 2016 Organizing Committee.
Yiping Song, Rui Yan, Xiang Li, Dongyan Zhao, and Ming Zhang. 2016. Two are better than one: An ensemble of retrieval- and generation-based dialog systems. CoRR, abs/1610.07149.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Su Wang, Rahul Gupta, Nancy Chang, and Jason Baldridge. 2019. A task in a suit and a tie: Paraphrase generation with semantic augmentation. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):7176-7183.
Yu Wu, Furu Wei, Shaohan Huang, Yunli Wang, Zhoujun Li, and Ming Zhou. 2019. Response generation by context-aware prototype editing. Proceedings of the AAAI Conference on Artificial Intelligence, 33(01):7281-7288.
A.1 Construction of  $\mathcal{N}(x)$  during training
For each pair of sentences, such as  $(x,y)$ , we augment its neighbourhood set  $\mathcal{N}(x)$  with multiple  $(x,y)$  pairs to get the new neighbourhood set
$$
\mathcal {N} ^ {\prime} (x) = [ (p _ {1}, q _ {1}), \dots , (p _ {K}, q _ {K}), (x, y), \dots , (x, y) ],
$$
where first  $K$  pairs are the  $K$ -most similar pairs (excluding  $(x,y)$  itself), and  $(x,y)$  is repeated  $K' < K$  times ( $K'$  is another hyperparameter of the model). Since the model sees the  $(x,y)$  pair  $\frac{K'}{K + K'}$  times during training as the retrieved pair, and these particular pairs include the output  $y$  themselves, the model is encouraged to use information coming from the retrieved neighboring pairs more often.
A.2 Analysis of Varying  $K$
We conduct an experiment to evaluate the effect of the hyper-parameter  $K$  in the proposed method. For each value of  $K \in \{1, 3, 5\}$ , we train our model once and obtain its results on the Quora dataset. Then, the value of two quality metrics (i.e. BLUE-2 and ROUGE-2) and two diversity metrics (i.e. SelfBLEU-2 And PINC-4) are computed. Figure 6 summarizes the obtained results. According to this figure, increasing the value of  $K$  slightly decreases the quality metrics while highly increases the diversity measures (Note that lower values of SELFf-BLEU and higher values of PINC indicate more diversity in the outputs). It shows that incorporating wider neighborhood in the editing process results in more diversity in the paraphrasing made by the editor.
A.3 Human Evaluation
The form used for the one-by-one experiment contains the following material:
Which set of the outputs do you prefer? Please opt based on the following criteria:
- Novelty: If any one of the outputs has expressed the semantic content of source sentence in a novel way like what we do as humans (e.g. changing the voice from active to passive or vice versa, using different words, phrases, or sentences that are not present in the source sentence but without changing the meaning considerably).
Figure 6: Results of the proposed method with some different values of the hyperparameter  $K$  according to four different metrics
- Diversity: if the three outputs are not expressed in the same way using the same words.
- Quality: If the outputs are paraphrases of the input sentence to a good extent.
- Readability: If the outputs are understandable by humans.
Note: Please select the option 'both(#1 and #2)' if you can not decide which one is better.
A.4 Example outputs
Table 6 shows the paraphrases generated for more sample inputs from the Quora dataset.
Figure 7: The human evaluation form for the first experiment.
Figure 8: The human evaluation form for the second experiment.
Table 6: More examples of our model's outputs
Undo[Ctrl+2] Redo Reset
Please rate the following sentences
What is one incident that changed your life?
Outputs:
- can you describe an incident that changed your life?
- what 's the decision that changed your whole life ?- what is that one incident that changed your life completely ?
Grammatically (Please consider the average score)
1) Non-sense composition of words and not in the form of human language, e.g. how world war iii world war.[1]
2) Can not understand what it means but it is still in the form of human language, e.g. what is the best movie of movie[2]
3) Basically fluent and has two or more minor grammatical errors or one serious grammatical error that does not have strong impact on understanding, e.g. what some good book for read[51]
4) Fluent and has one minor grammatical error that does not affect understanding, e.g. what is the best ways to learn programming[4].
5) Without any grammatical error[6]
Coherency (Please consider the average score)
1) Topic irrelevant or even can not understand what it means.[8]
2) Topic relevant but fail to cover most of the content of source sentence, e.g. what is some tips to learn english  $\rightarrow$  when do you start to learn english?
(3) 3 Cover part of the content of source sentence and has serious information loss, e.g. what is the best love movie by wang ka wai  $\rightarrow$  what is the best movie[8]
4) Basically the same meaning of the source sentence but does not cover some minor content, e.g. what are some good places to visit in hong kong during summer  $\rightarrow$  can you suggest some places to visit in hong kong.[9]
5) Accurate paraphrase with exact the same meaning of the source sentence[0]
Submit [Ctrl+Enter]
Task ID: 6
How do I reset my Gmail password when I do n't remember my recovery information?
Paraphrase sets:
Set #1
- how do i reset my gmail password when i do n't remember my recovery information ?
- I forgot my gmail password and I do n't remember my recovery information . what can i do ?- how do i reset my gmail password when i do n't remember my recovery information ?
Set #2
- what is the best way to reset your gmail without password?
- how can i reset my gmail password if i do n't remember my recovery information ?
I prefer ...
Set  $\# 1^{[1]}$
Set #2[2]
both (#1 and #2) [3]
(1)  $x$  What are great examples of absent mindedness?
$p$  What are the common characteristics of ignorant people?
What are the most common characteristics of stupid, ignorant or uneducated people?
$y^\prime$  what are the most common examples of absent mindedness?
(2)  $x$  What should I know about moving from Pennsylvania to South Carolina?
$p$  What are the pros and cons of living in Richmond, VA?
q What is your review of Richmond, VA?
$y^\prime$  what is your review of moving from pennsylvania to south carolina?
(3)  $x$  How does Quora determine how many views an answer gets?
$p$  How does Quora calculate how many views does an answer have?
$q$  How does Quora count the views of my / your answers?
$y^\prime$  how does quora determine the views of your answers?
(4)  $x$  Do you think India should revisit Indus Water Treaty with Pakistan?
$p$  Should India hold a referendum in Kashmir ?
$q$  Would India ever consider a referendum for Kashmir independence?
$y^\prime$  would india ever consider a indus water treaty with pakistan ?