SlowGuess/NAACL_2022_OCR huggingface dataset:
Implementation. We adopt a pretrained transformer encoder-decoder (sequence-to-sequence) WS-BART (Lewis et al., 2020a) as the backbone of the generator in our proposed method MB-RPG.
<sup>1</sup>The datasets are available in the submitted zip file.
$^{2}$ https://huggingface.co/datasets/quora  
 $^{3}$ https://github.com/lanwuwei/  
Twitter-URL-Corpus  
 $^{4}$ https://arxiv.org/pdf/1504.00325.pdf
Method	Quora-S	Twitter
BLEU-2	BLEU-4	ROUGE-1	ROUGE-2	BLEU-2	BLEU-4	ROUGE-1	ROUGE-2
Supervised	Res-LSTM	38.52	24.56	59.69	32.71	32.13	25.92	41.77	27.94
Transformer	42.91	30.38	61.25	34.23	40.34	32.14	44.53	29.55
RbM	43.54	-	38.11	32.84	44.67	-	41.87	24.23
RaE	40.35	25.37	62.71	31.77	44.33	34.16	47.55	31.53
FSET	51.03	33.46	66.17	39.55	46.35	34.62	49.53	32.04
Weakly-supervised	WS-BART	44.19	31.18	58.69	33.39	45.03	34.00	51.34	35.89
LTL	49.18	36.05	63.06	39.71	49.30	37.94	56.02	40.61
MB-RPG	54.88	41.56	67.66	43.98	51.65	39.58	61.45	44.19
Method	Quora-U	MSCOCO
iBLEU	BLEU	ROUGE-1	ROUGE-2	iBLEU	BLEU	ROUGE-1	ROUGE-2
Unsupervised	VAE	8.16	13.96	44.55	22.64	7.48	11.09	31.78	8.66
CGMH	9.94	15.73	48.73	26.12	7.84	11.45	32.19	8.67
USPA	12.02	18.21	59.51	32.63	9.26	14.16	37.18	11.21
PUP	14.91	19.68	59.77	30.47	10.72	15.81	37.38	13.87
DBlock	20.93	26.76	65.60	42.09	-	-	-	-
Weakly-supervised	WS-BART	29.30	27.63	58.43	33.39	20.11	15.90	40.65	15.62
LTLS	31.20	29.25	62.71	39.21	23.25	18.87	45.18	19.17
MB-RPG	33.56	33.85	66.30	42.48	28.09	19.39	49.42	25.18
Table 1: Performance scores for our method MB-RPG as well as all the baseline methods on four paraphrasing datasets under supervised or unsupervised learning setting.
To improve the efficiency of reinforcement learning, we model the reinforced data valuator model  $M_{\psi}$  as a pretrained BERT followed by two fully-connected trainable layers as the head for policy output. BERT serves as a feature extractor and is kept fixed during policy learning. We present other details for our method in appendix.
Baseline Methods. We compare our method with twelve benchmark approaches including the state-of-the-art method. In general, the baselines come from the following three categories: (i) supervised methods that are trained with high-quality supervised target paraphrases, i.e., Transformer (Vaswani et al., 2017), RbM (Li et al., 2018), Residual LSTM (Prakash et al., 2016) and two retrieval-based methods FSET (Kazemnejad et al., 2020) and RaE (Hashimoto et al., 2018); (ii) unsupervised methods that do not have access to the parallel data, including Constrained sentence Generation with Metropolis-Hastings (CGMH) (Miao et al., 2019) VAE (Bowman et al., 2016), Unsupervised Paraphrase generation with Simulated Annealing (UPSA) (Liu et al., 2020b) and Progressive Unsupervised Paraphrasing (PUP) (Siddique et al., 2020), Dynamic Blocking (DBlock) (Niu et al., 2020); (iii) semi-supervised methods including WS-BART (Lewis et al., 2020a) which corresponds to BART trained upon the weakly-supervised data and Learning To Selectively Learn (LTSL) (Ding et al., 2021) which is the most closely related method to ours. LTSL also adopts reinforcement learning for selective
learning and our approach is a model-based improved version over its vanilla policy gradient formulation. Also note that both LTSL and MB-RPG adopts BART as the pretrained generator.
We present the benchmark results on all the compared methods on the four evaluation datasets in Table 1. Note that our proposed method is denoted as MB-RPG. Overall, we could conclude that MB-RPG outperforms all its baselines with significant margins in terms of BLEU scores and ROUGE scores across the four evaluation datasets.
From the results, we could notice that under the supervised setting, most methods, such as Transformer and FSET, could significantly outperform most of the results obtained from the unsupervised setting (e.g., CGMH and VAE). Compared with the supervised methods, even though our method does not touch the supervised labels, it could still outperform the supervised methods by noticeable margins. This further indicates that our method is promising to be adopted in many real life applications where there is rather limited access to the golden labelled paraphrase pairs.
Compared to the unsupervised approaches, MB-RPG is also superior especially in terms of the iBLEU and BLEU metrics. The main reason might be that the word editing or sampling attempts proposed in the unsupervised baselines yield less desirable target paraphrases and thus makes the performance of the model trained under the unsupervised
Method	Quora-S
BLEU-2	BLEU-4	ROUGE-1	ROUGE-2
WS-BART	44.19	31.18	58.69	33.39
w/o planning	50.67	38.29	64.12	41.28
w/o rew norm	51.26	42.32	66.25	40.23
LTSL	49.18	36.05	63.06	39.71
MB-RPG	54.88	41.56	67.66	43.98
Table 2: Ablation study results for the planning and reward normalization components on Quora-s.
data fall far below our method and various supervised baselines. The inferior performance of the unsupervised methods has also been empirically evaluated and discussed by Niu et al. (2020).
From Table 1, we could notice that the performance of MB-RPG is much better than its closest counterpart LTSL, while LTSL is also a reinforcement learning-based selective learning method. In both supervised and unsupervised scenarios, MB-RPG could outperform LTSL consistently by a noticeable margin. The average improvements on BLEU-2 and BLEU-4 scores are 4.02 and 3.57 respectively. On each BLEU or ROUGE metrics, MB-RPG achieves better scores than LTSL. This shows that our proposed method MB-RPG achieves state-of-the-art method on various paraphrase generation benchmark datasets. It also shows that using model-based planning and refining the noisy reward could bring considerable benefit to the data valuation process. Such result sheds lights to the research of refining the formulation for the Markov Decision Process and coming up with more adequate reinforcement learning frameworks to facilitate better data valuation. The results reveals that the performance of the vanilla WS-BART is inferior than LTSL or MB-RPG, both of which adopt WS-BART as their generators' backbone.
To thoroughly evaluate the effect of the individual components we proposed upon the vanilla policy gradient method, we present an ablation study to evaluate the individual effect of such components. Specifically, we consider three ablated baselines: 1) w/o planning: our model without the model-based planning module; 2) w/o rew norm: our model without the reward normalization module; 3) LTSL: without both planning and reward normalization modules. We present the results in Table 2. From the results, we notice that the baseline w/o rew planning achieves similar results with LTSL which is not comparable to our proposed method. It verifies
the importance of leveraging the model-based planning to reduce the noise among the short-term one-step reward signals. From the results shown in Table 2, we also notice that the baseline w/o rew norm achieves better results than LTSL, but not as good as the full version of our method. This shows that reward normalization is an essential step to train the Markov Decision Process formulated for selective paraphrase generation. The aforementioned results also reveal that the model-based planning module and the reward normalization module are two modules with relatively parallel effects of each other without much conflicting situations. Lastly, we wish to highlight that the WS-BART without reinforcement learning-based selective learning essentially performs very outstanding by itself. However, leveraging reinforcement learning-based selective learning could result in significant boost to the performance of WS-BART. This shows that reinforced selective learning is a promising direction to consider for improving the SOTA performance in paraphrase generation or other generation tasks.
Our work tackles an important problem of leveraging reinforcement learning-based selective learning techniques to effectively deal with the noisy label issue in paraphrase generation tasks. We introduce a model-based framework which performs planning to capture the long-term effects for efficient exploration so as to overcome the noisy short-term reward issue experienced by most of the existing approaches. We also propose an effective reward normalization approach which could normalize the noisy reward signal to a distribution with a fixed zero-mean. We demonstrate that our proposed method could outperform baseline approaches with significant margins on the testified domains. Future work includes refining the state and reward terms in the Markov Decision Process for better data valuation or feature selections. One ongoing work is to integrate MB-RPG into our previous feature exploration work in video recommendation (Li et al., 2020) to improve feature quality. Another one is to employ MB-RPG in our RL-based coreference resolution to filter out irrelevant features (Fei et al., 2019). In addition, we could consider constructing the policy model upon alternative generator backbone. Also, we could consider inferring the reward from different sources, such as incorporating auxiliary language models in the training.
Yu Bao, Hao Zhou, Shujian Huang, Lei Li, Lili Mou, Olga Vechtomova, Xin-Yu Dai, and Jiajun Chen. 2019. Generating sentences from disentangled syntactic and semantic spaces. In Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL), pages 6008-6019, Florence, Italy.
Jonathan Berant and Percy Liang. 2014. Semantic parsing via paraphrasing. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (ACL), pages 1415-1425, Baltimore, MD.
Samuel R. Bowman, Luke Vilnis, Oriol Vinyals, Andrew M. Dai, Rafal Jozefowicz, and Samy Bengio. 2016. Generating sentences from a continuous space. In Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning (CoNLL), pages 10-21, Berlin, Germany.
Mingda Chen, Qingming Tang, Sam Wiseman, and Kevin Gimpel. 2019. Controllable paraphrase generation with a syntactic exemplar. In Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL), pages 5972-5984, Florence, Italy.
Yen-Chun Chen, Zhe Gan, Yu Cheng, Jingzhou Liu, and Jingjing Liu. 2020. Distilling knowledge learned in BERT for text generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 7893-7905, Online.
David A. Cohn, Zoubin Ghahramani, and Michael I. Jordan. 1996. Active learning with statistical models. J. Artif. Intell. Res., 4:129-145.
Mostafa Dehghani, Hamed Zamani, Aliaksei Severyn, Jaap Kamps, and W. Bruce Croft. 2017. Neural ranking models with weak supervision. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 65-74, Shinjuku, Tokyo, Japan.
Kaize Ding, Dingcheng Li, Alexander Hanbo Li, Xing Fan, Chenlei Guo, Yang Liu, and Huan Liu. 2021. Learning to selectively learn for weakly-supervised paraphrase generation. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 5930-5940, Virtual Event / Punta Cana, Dominican Republic.
Bill Dolan, Chris Quirk, and Chris Brockett. 2004. Unsupervised construction of large paraphrase corpora: Exploiting massively parallel news sources. In Proceedings of the 20th International Conference on Computational Linguistics (COLING), Geneva, Switzerland.
Pablo Ariel Duboué and Jennifer Chu-Carroll. 2006. Answering the question you wish they had asked:
The impact of paraphrasing for question answering. In Proceedings of Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics (HLT-NAACL), New York, NY.
Anthony Fader, Luke S. Zettlemoyer, and Oren Etzioni. 2013. Paraphrase-driven learning for open question answering. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (ACL), pages 1608-1618, Sofia, Bulgaria.
Tobias Falke, Markus Boese, Daniil Sorokin, Caglar Tirkaz, and Patrick Lehnen. 2020. Leveraging user paraphrasing behavior in dialog systems to automatically collect annotations for long-tail utterances. In Proceedings of the 28th International Conference on Computational Linguistics (COLING), Industry Track, pages 21-32, Online.
Miao Fan, Chao Feng, Mingming Sun, and Ping Li. 2019. Reinforced product metadata selection for helpfulness assessment of customer reviews. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 1675-1683, Hong Kong, China.
Hongliang Fei, Xu Li, Dingcheng Li, and Ping Li. 2019. End-to-end deep reinforcement learning based coreference resolution. In Proceedings of the 57th Conference of the Association for Computational Linguistics (ACL), pages 660-665, Florence, Italy.
Yao Fu, Yansong Feng, and John P. Cunningham. 2019. Paraphrase generation with latent bag of words. In Advances in Neural Information Processing Systems (NeurIPS), pages 13623-13634, Vancouver, Canada.
Clinton Gormley and Zachary Tong. 2015. Elastic-search: the definitive guide: a distributed real-time search and analytics engine. O'Reilly Media, Inc.
Tatsunori B. Hashimoto, Kelvin Guu, Yonatan Oren, and Percy Liang. 2018. A retrieve-and-edit framework for predicting structured outputs. In Advances in Neural Information Processing Systems (NeurIPS), pages 10073-10083, Montreal, Canada.
Dan Hendrycks, Mantas Mazeika, Duncan Wilson, and Kevin Gimpel. 2018. Using trusted data to train deep networks on labels corrupted by severe noise. In Advances in Neural Information Processing Systems (NeurIPS), pages 10477-10486, Montreal, Canada.
Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech Czarnecki, Simon Schmitt, and Hado van Hasselt. 2019. Multi-task deep reinforcement learning with popart. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI), pages 3796-3803, Honolulu, HI.
Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL-HLT), pages 1875-1885, New Orleans, LA.
Amirhossein Kazemnejad, Mohammadreza Salehi, and Mahdieh Soleymani Baghshah. 2020. Paraphrase generation by learning how to edit from samples. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 6010-6021, Online.
Vijay R. Konda and John N. Tsitsiklis. 1999. Actor-critic algorithms. In Advances in Neural Information Processing Systems (NIPS), pages 1008-1014, Denver, CO.
Wuwei Lan, Siyu Qiu, Hua He, and Wei Xu. 2017. A continuously growing dataset of sentential paraphrases. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1224-1234, Copenhagen, Denmark.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020a. BART: denoising sequence-to-sequence pretraining for natural language generation, translation, and comprehension. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 7871-7880, Online.
Patrick S. H. Lewis, Ethan Perez, Aleksandra Pik-tus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Kuttler, Mike Lewis, Wen-tau Yih, Tim Roektaschel, Sebastian Riedel, and Douwe Kiela. 2020b. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems (NeurIPS), virtual.
Dingcheng Li, Xu Li, Jun Wang, and Ping Li. 2020. Video recommendation with multi-gate mixture of experts soft actor critic. In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR), pages 1553-1556, Virtual Event, China.
Zichao Li, Xin Jiang, Lifeng Shang, and Hang Li. 2018. Paraphrase generation with deep reinforcement learning. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 3865-3878, Brussels, Belgium.
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C. Lawrence Zitnick. 2014. Microsoft COCO: common objects in context. In Proceedings of the 13th European Conference on Computer Vision
(ECCV), Part V, pages 740-755, Zurich Switzerland.
Guiliang Liu, Xu Li, Mingming Sun, and Ping Li. 2020a. An advantage actor-critic algorithm with confidence exploration for open information extraction. In Proceedings of the 2020 SIAM International Conference on Data Mining (SDM), pages 217-225, Cincinnati, OH.
Xianggen Liu, Lili Mou, Fandong Meng, Hao Zhou, Jie Zhou, and Sen Song. 2020b. Unsupervised paraphrasing by simulated annealing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 302-312, Online.
Jonathan Mallinson, Rico Sennrich, and Mirella Lapata. 2017. Paraphrasing revisited with neural machine translation. In Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics (EACL), pages 881-893, Valencia, Spain.
Kathleen R. McKeown. 1979. Paraphrasing using given and new information in a question-answer system. In Proceedings of the 17th Annual Meeting of the Association for Computational Linguistics (ACL), University of California at San Diego, La Jolla, CA.
Ning Miao, Hao Zhou, Lili Mou, Rui Yan, and Lei Li. 2019. CGMH: constrained sentence generation by metropolis-hastings sampling. In Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (AAAI), pages 6834–6842, Honolulu, HI.
Baharan Mirzasoleiman, Kaidi Cao, and Jure Leskovec. 2020. Coresets for robust training of deep neural networks against noisy labels. In Advances in Neural Information Processing Systems (NeurIPS), virtual.
Tong Niu, Semih Yavuz, Yingbo Zhou, Huan Wang, Nitish Shirish Keskar, and Caiming Xiong. 2020. Unsupervised paraphrase generation via dynamic blocking. arXiv preprint arXiv:2010.12885.
Junhyuk Oh, Xiaoxiao Guo, Honglak Lee, Richard L. Lewis, and Satinder P. Singh. 2015. Action-conditional video prediction using deep networks in atari games. In Advances in Neural Information Processing Systems (NIPS), pages 2863–2871, Montreal, Canada.
Aaditya Prakash, Sadid A. Hasan, Kathy Lee, Vivek V. Datla, Ashequl Qadir, Joey Liu, and Oladimeji Farri. 2016. Neural paraphrase generation with stacked residual LSTM networks. In Proceedings of the 26th International Conference on Computational Linguistics (COLING), pages 2923-2934, Osaka, Japan.
Chris Quirk, Chris Brockett, and William B. Dolan. 2004. Monolingual machine translation for paraphrase generation. In Proceedings of the 2004 Conference on Empirical Methods in Natural Language
Processing (EMNLP), pages 142-149, Barcelona, Spain.
Scott E. Reed, Honglak Lee, Dragomir Anguelov, Christian Szegedy, Dumitru Erhan, and Andrew Rabinovich. 2015. Training deep neural networks on noisy labels with bootstrapping. In Proceedings of the 3rd International Conference on Learning Representations (ICLR Workshop), San Diego, CA.
Philip Resnik, Olivia Buzek, Chang Hu, Yakov Kronrod, Alexander J. Quinn, and Benjamin B. Bederson. 2010. Improving translation via targeted paraphrasing. In Proceedings of the 2010 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 127-137, MIT Stata Center, Massachusetts.
Fabio Rinaldi, James Dowdall, Kaarel Kaljurand, Michael Hess, and Diego Mollá Aliod. 2003. Exploiting paraphrases in a question answering system. In Proceedings of the Second International Workshop on Paraphrasing (IWP@ACL), pages 25-32, Sapporo, Japan.
Stephen E. Robertson and Hugo Zaragoza. 2009. The Probabilistic Relevance Framework: BM25 and Beyond, volume 3.
Burr Settles. 2009. Active learning literature survey.
Hajin Shim, Sung Ju Hwang, and Eunho Yang. 2018. Joint active feature acquisition and classification with variable-size set encoding. In Advances in Neural Information Processing Systems (NeurIPS), pages 1375-1385, Montreal, Canada.
A. B. Siddique, Samet Oymak, and Vagelis Hristidis. 2020. Unsupervised paraphrasing via deep reinforcement learning. In Proceedings of the 26th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD), pages 1800-1809, Virtual Event, CA.
Zijun Sun, Chun Fan, Xiaofei Sun, Yuxian Meng, Fei Wu, and Jiwei Li. 2020. Neural semi-supervised learning for text classification under large-scale pretraining. arXiv preprint arXiv:2011.08626.
Richard S. Sutton, David A. McAllester, Satinder P. Singh, and Yishay Mansour. 1999. Policy gradient methods for reinforcement learning with function approximation. In Advances in Neural Information Processing Systems (NIPS), pages 1057-1063, Denver, CO.
Robert Tibshirani. 1996. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Society: Series B (Methodological), 58(1):267-288.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems (NIPS), pages 5998-6008, Long Beach, CA.
Tingwu Wang and Jimmy Ba. 2020. Exploring model-based planning with policy networks. In Proceedings of the 8th International Conference on Learning Representations (ICLR), Addis Ababa, Ethiopia.
Zhuoyi Wang, Yigong Wang, Yu Lin, Evan Delord, and Latifur Khan. 2020. Few-sample and adversarial representation learning for continual stream mining. In Proceedings of the Web Conference (WWW), pages 718-728, Taipei.
Shan Wu, Bo Chen, Chunlei Xin, Xianpei Han, Le Sun, Weipeng Zhang, Jiansong Chen, Fan Yang, and Xunliang Cai. 2021. From paraphrasing to semantic parsing: Unsupervised semantic parsing via synchronous semantic decoding. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (ACL/IJCNLP), pages 5110-5121, Virtual Event.
Sander Wubben, Antal van den Bosch, and Emiel Krahmer. 2010. Paraphrase generation as monolingual translation: Data and evaluation. In Proceedings of the Sixth International Natural Language Generation Conference (INLG), Trim, Co. Meath, Ireland.
Yan Xu, Fuming Sun, and Xue Zhang. 2013. Literature survey of active learning in multimedia annotation and retrieval. In Proceedings of the International Conference on Internet Multimedia Computing and Service (ICIMCS), pages 237-242, Huangshan, China.
Jinsung Yoon, Sercan Ömer Arik, and Tomas Pfister. 2020. Data valuation using reinforcement learning. In Proceedings of the 37th International Conference on Machine Learning (ICML), pages 10842-10851, Virtual Event.
Jinsung Yoon, James Jordon, and Mihaela van der Schaar. 2019. INVASE: instance-wise variable selection using neural networks. In Proceedings of the 7th International Conference on Learning Representations (ICLR), New Orleans, LA.
Sara Zannone, José Miguel Hernández-Lobato, Cheng Zhang, and Konstantina Palla. 2019. Odin: Optimal discovery of high-value information using model-based deep reinforcement learning. In ICML Realworld Sequential Decision Making Workshop.