SlowGuess/NAACL_2022_OCR huggingface dataset:
Haiyan Yin, Dingcheng Li, Ping Li
{haiyanyin18, dingchengl, pingli98}  $@$  gmail.com
Paraphrase generation is an important language generation task attempting to interpret user intents and systematically generate new phrases of identical meanings to the given ones. However, the effectiveness of paraphrase generation is constrained by the access to the golden labeled data pairs where both the amount and the quality of the training data pairs are restricted. In this paper, we propose a new weakly supervised paraphrase generation approach that extends the success of a recent work that leverages reinforcement learning (RL) for effective model training with data selection. While data selection is privileged for the target task which has noisy data, developing a reinforced selective learning regime faces several unresolved challenges. In this paper, we carry on important discussions about the above problem and present a new model that could partially overcome the discussed issues with a model-based planning feature and a reward normalization feature. We perform extensive evaluation on four weakly supervised paraphrase generation tasks where the results show that our method could significantly improve the state-of-the-art performance on the evaluation datasets.
Paraphrase generation is an important natural language generation task which aims to generate a target sentence that encapsulates the meaning of a given source sentence while conforming to the style of some desired exemplar. It plays an essential role in many real-world applications for natural language processing, such as semantic parsing (Berant and Liang, 2014; Wu et al., 2021), machine translation (Resnik et al., 2010; Mallinson et al., 2017), recommend system (Falke et al., 2020) and question answering (Fader et al., 2013; Rinaldi et al., 2003; Duboué and Chu-Carroll, 2006). Different from other controllable text generation tasks where golden labelled data pairs are accessible and often
being readily available, for paraphrase generation tasks, large scale of parallel paraphrase samples are often extremely hard to collect because generating them would often consume extensive domain knowledge or the generation could hardly be standardized. Therefore, the chance of performing supervised learning in real life scenarios would be considerably limited.
To overcome the data unattainable issue, unsupervised and semi-supervised approaches have achieved growing attention in the recent decade. Generally, the unsupervised approaches adopt sampling-based or editing-based techniques (Bowman et al., 2016; Miao et al., 2019) to remedy golden standard knowledge but they generally result in less coherent or controllable target phrases due to their lose of supervision. Therefore, in our paper, we focus on weakly-supervised paraphrase generation which has demonstrated great effectiveness in many major natural language processing tasks (Dehghani et al., 2017; Sun et al., 2020). Although weakly-supervised approaches have successfully pushed forward the state-of-the-art performance standard for the language-based tasks, when employed for paraphrase generation, they still face the challenge of how to acquire high-quality paired paraphrase data and therefore lead to noisy data pairs which might bring negative effect to the downstream task of training paraphrase generation models.
To overcome the aforementioned challenge and effectively train paraphrase generation models from noisy and not equally informative paraphrase pairs, we adopt a learning to selectively learn approach. That is, a meta model is learned to select intuitive paraphrase pairs while eliminating the low quality ones. Thus the paraphrase generation model which is jointly learned with the meta data selector model could achieve better performance through the carefully specified selective learning process. Nonetheless, it is impossible for learning an effec
Human Language Technologies, pages 1385 - 1395
tive meta data selection policy to be a supervised learning task due to the missing of optimal target selection policy. To overcome this issue, we adopt a reinforcement learning-based approach to learn effective selection policy without supervised signal. To this end, we extend the success of previous reinforcement learning-based approach for data selection (Ding et al., 2021). However, formulating a Markov decision process (MDP) for the paraphrase learning process is a non-trivial task. In previous works, several important parts of their MDP formulation, such as the design of reward signal, are in need of further investigation (Yoon et al., 2019; Ding et al., 2021) and there also lacks in depth discussion on the challenge of solving the reinforcement learning problem. In this paper, we are motivated to extend this important line of using reinforcement learning to perform selective learning in weakly-supervised paraphrase generation problems and thus overcoming the data unattainable issue. Overall we present several key insights into formulating the MDP for the selective learning problem as well as developing a model-based reinforcement learning framework to effectively solving the MDP.
This paper has three main contributions:
- We present a novel model-based reinforcement learning approach for effectively training paraphrase generation models under weakly supervised regime, where our proposed reinforcement learning approach could effectively overcome some of the major limitations of the existing works for data selection.
- We present an in-depth discussion on the challenges and the potential ways to formulate the selective weakly supervised paraphrase generation tasks with reinforcement learning, which sheds light on the important direction of developing more sophisticated reinforcement learning frameworks for weakly supervised paraphrase generation.
- We present extensive empirical evaluation results on four evaluation datasets where the weakly supervised datasets are generated from supervised or unsupervised manner. The evaluation results show that our proposed method could lead to substantially better performance than all the considered baseline approaches over all the evaluation datasets.
Paraphrase Generation has long been an important research problem for the natural language processing community. Traditional methods solve this problem by exploiting linguistic knowledge (Wubben et al., 2010; McKeown, 1979) or utilizing statistical machine translation (Quirk et al., 2004; Dolan et al., 2004). As being a sequence generation task, most of recently emerged approaches are framed as instances of the deep neural networks-based sequence-to-sequence (seq2seq) models (Prakash et al., 2016; Chen et al., 2020). Early works are mostly developed under a supervised setting while discarding the noise in the datasets. Two representative examples are the Residual LSTM (Prakash et al., 2016) and BERT (Chen et al., 2020). Later on, researchers start to work on improving the quality of the paraphrases, such as leveraging retrieval augmented (Kazemnejad et al., 2020; Lewis et al., 2020b; Hashimoto et al., 2018) or syntactic structure-based (Iyyer et al., 2018; Chen et al., 2019) approaches to produce better paraphrases. Besides the aforementioned approaches, there are also another lines of methods that attempt to alleviate the labeling cost with attempts like unsupervised learning (Bowman et al., 2016; Fu et al., 2019; Bao et al., 2019; Miao et al., 2019; Wang et al., 2020) as well as simulated annealing (Liu et al., 2020b) and reinforcement learning (Siddique et al., 2020). Compared to the conventional reinforcement learning methods which consider the generators as the policy models, our work models the policy as a meta learner to accomplish a data selection objective. Our work is mostly related to (Ding et al., 2021), but we adopt a very different reinforcement learning approach which is the key for effective selective learning.
Selective Learning refers to the case of selecting items, e.g., features or data points, to learn from among other items. It motivates many important fields in machine learning, such as active learning (Cohn et al., 1996; Settles, 2009; Xu et al., 2013; Fan et al., 2019; Liu et al., 2020a) and robust machine learning (Hendrycks et al., 2018; Reed et al., 2015; Mirzasoleiman et al., 2020). Our work is motivated by the existing instance-wise active data/feature acquisition approaches. One typical example is the conventional linear model that poses sparsity inducing prior distribution to
the model (Tibshirani, 1996) and thus actively selects important features to the model. Recently, there also emerged approaches that adopt reinforcement learning to actively find optimal feature subsets (Yoon et al., 2019; Shim et al., 2018; Zannone et al., 2019). Though such attempts have demonstrated certain efficacy in handling instance-wise feature selection, they only deal with non time-series data in non NLP domains, while the focus of our work is to deal with noisy labeled pairs in paraphrase generation tasks. Our work is mostly related to the instance-level active data acquisition approaches (Yoon et al., 2020; Ding et al., 2021), which are mostly adopted under the circumstances of data efficient or cost-sensitive learning or when dealing with noisy data. Yoon et al. (2020) and Ding et al. (2021) are formulation-wise identical while Yoon et al. (2020) is among the very first model for active data selective learning, whereas Ding et al. (2021) applies it on the task of paraphrase generation. Our work extends this important direction to perform selective learning but we formulate a new model-based reinforcement learning method which aims to overcome partial limitations for the existing work and is empirically proven to be more effective than it on all the experimental domains.
3 Reinforced Selective Learning for Paraphrase Generation
We present the general formulation for reinforcement learning-based selective weakly-supervised paraphrase generation problems.
3.1 Weakly-Supervised Paraphrase Generation Problem
Paraphrase generation is a sequence-to-sequence natural language generation problem. Formally, given a set of  $N$  source sentences  $\mathcal{X} = \{\mathbf{x}_i\}_{i=1}^N$ , where each sentence  $X_i$  is a set of discrete tokens, i.e.,  $\mathbf{x}_i = \{o_j\}_{j=1}^T$ , paraphrase generation aims to obtain a non-parallel output sentences  $\mathcal{Y} = \{\mathbf{y}_i\}_{i=1}^N$ , where each  $y_i$  encapsulates identical meaning to  $x_i$  but comes in the form following some desired exemplars. When training paraphrase generation model, obtaining golden labeled target  $\mathcal{Y}$  is a critical challenge. Therefore, we consider a weakly-supervised paraphrase generation regime, forming a set of pseudo labeled pairs termed as  $\mathcal{D}_{pseudo} = \{\mathbf{x}_i, \mathbf{y}_i\}_{i=1}^N$ . When training models under the weakly-supervised regime,
our work adopts a commonly taken assumption in weakly-supervised learning works. That is, the model has access only to a small set of high-quality parallel sentences  $\mathcal{D}_{dev} = \{\mathbf{x}_i,\mathbf{y}_i\}_{i = 1}^L (L <   <   M)$  which could be considered as golden labeled pairs.
To generate high-quality target for the pseudo labeled pairs, retrieval-based expansion approach is adopted to generate paraphrase  $\{\mathbf{y}_i\}_{i = 1}^N$  which has recently demonstrated great effectiveness in text generation tasks (Kazemnejad et al., 2020; Lewis et al., 2020b). Specifically, for each source sentence  $\mathbf{x}_i$ , BM25 (Robertson and Zaragoza, 2009) is first adopted as an effective retriever. Then we use Elastic Search (Gormley and Tong, 2015) to create search indexes for fast searching similar sentences to  $\mathbf{x}_i$ . The benefit of using such a combination is that the method provides flexibility for weak supervision while being training-free. Though there are considerable possibilities for adopting alternative approaches such as training-based methods for generating paraphrases, we demonstrate that our adopted method already yields promising performance.
Given the training data  $\mathcal{D}_{\text{pseudo}}$  and  $\mathcal{D}_{\text{dev}}$ , the paraphrase generation model is optimized by Maximum Likelihood Estimation (MLE), i.e.,
$$
\mathcal {L} (\psi) = \sum_ {(x, y) \sim \mathcal {D} _ {\text {p s e u d o}}} - \log p _ {\psi} (y | x). \tag {1}
$$
3.2 Markov Decision Process
Reinforcement learning is an area of machine learning concerned with how to learn sequential/non-sequential decision making policies for the problems formulated as Markov Decision Processes.
Formally, a Markov Decision Process is formulated as a tuple  $< S, A, P, R, \gamma >$ , where  $S$  is a set of states,  $A$  is a set of actions,  $P$  is a transition probability matrix,  $R$  is the reward function applied upon a state-action pair, and  $\gamma$  is a discount factor. Among the transition probability matrix, each of its entry determines the probability of transiting from one state to another. In the reinforcement learning environment, actions are executed state by state, forming time sequences. At each time step, the agent observes a state, determines an action to be issued under the state, and receives a reward from the environment suggesting the or optimality of the action given the state. The state is Markovian, which means that the decision could be solely determined by the presented state and not on its preceding states. The objective of reinforcement
learning is to maximize the cumulative rewards received by the agent.
3.3 Reinforced Paraphrase Generation
We now present a detailed discussion on how to technically combine reinforcement learning with paraphrase generation and devise a reinforcement learning-based selective learning paradigm.
Given the training regime with noisy labeled data, evaluating the value of the data would be a fundamental problem. To tackle this problem, we target at utilizing reinforcement learning techniques to learn an adaptive data valuator model  $M_{\phi}(\cdot)$  which could be jointly updated with the paraphrase generator model and intuitively give value evaluation over the pseudo paraphrase pairs. Generally,  $M_{\phi}(\cdot)$  could be considered as a reinforcement learning agent that we train to maximize the reward signal which is quantitatively represented as improvement achieved by the generator throughout the model training period. With this regard, we present how to formulate the Markov Decision Process (MDP) from corresponding state, action and reward for the selective learning of paraphrase generation task in the following section. We also discuss the challenges of MDP formulation in selective learning and potential ways to improve it.
STATE The state refers to what the agent observe for decision making. It is the representation of the informative features for an instance or a group of data instances to be evaluated. Ideally, the information conveyed by a state should dynamically change throughout the learning process. With end-to-end RL, the state could be represented by low-level raw features, such as image pixels. It is privileged to use high-level representations for state which could potentially ease the policy learning. In our case, we adopt a very common representation for thestate sentences, which is extracted from a pretrained language model. Generally it reflects the static importance of data to the task without detailed modeling on the learning process. However, the importance values for the data inferred by our model are dynamic due to the dynamic updates of model parameters.
ACTION In selective learning, the way we could model action is relatively fixed. That is, an action needs to tell the decision about whether a data point or a group of pints need to be selected or not. In this paper, we model the action as a Bernoulli variable over each data point. There is generally less
space to improve the action modeling part in the reinforced paraphrase generation process.
REWARD The reward signal is designated to tell the benefit of selecting a data instance to update the model over its paraphrase generation quality. It is also the most problematic item to model in the MDP. While the data valuator model is jointly trained with the generator model, it is impossible to obtain a golden standard reward signal to tell the importance of data for the dynamic learning environment. Most of the existing selective learning approach model the reward from the improvement of the downstream task performance before and after the model is updated by the data instance placed in a mini batch of samples. However, such performance score-based reward modeling has the following two major limitations: (1) it generates a reward over a group of mini batch samples and thus could not yield precise term over each independent data point; (2) the reward score has a changing distribution whose scale keeps decreasing and eventually converges to 0, which could bring difficulty to the policy learning (upon convergence the performance would no more increase and therefore lead to a mean of 0 over the reward). Most of the existing works consider the performance score-based reward modeling only without compensating its scaling or independence issue. We develop a method with model-based and scaling flavour to partially overcome the aforementioned challenges.
4 MB-RPG: Model-based Reinforced Paraphrase Generation Framework
In this section, we introduce our proposed Model-based Reinforced Paraphrase Generation framework (MB-RPG). The overview of MB-RPG is shown in Figure 1. The essences of MB-RPG can be summarised as two points. One, MB-RPG adopts model-based planning attempting to perform decision making based on multi-step look-ahead. This way, we can address the first limitation, namely, the short-slightness brought by the conventional score-based approach, one-step look-ahead. Two, MB-RPG adopts a reward scaling module to normalize the reward as a fixed distribution to overcome the stochasticity of reward distribution.
4.1 Model-based Planning
We present a sophisticated method where the decision making system is developed to learn the policy based on long term effects, in order to overcome the
Figure 1: Overview to our proposed Model-Based Reinforced Paraphrase Generation (MB-RPG) framework. Our method takes weakly-supervised dataset obtained from a retrieval-based method as its training data. At each time step, model-based planning is performed which expands the forward direction along multiple action directions and then perform multi-step look ahead. Then the generator is updated by the best model after the  $\tau$ -step planning.
limitation of the existing works where the reward is inferred from short-term noisy prediction loss-based metrics. In fact, the model-based planning method has been long studied and been proved effective in the reinforcement learning literature (Oh et al., 2015; Wang and Ba, 2020). Unlike model-free reinforcement learning approaches where the agent experiences single-length trajectory, at each time step the episodes expand along one of the action directions instead. Accordingly, the resulting single chain enables our model-based approach to expand future trajectories through multiple action directions and perform the decision making based on the planned roll-outs. Let  $\tau$  be the planned future steps, i.e., at each time step, the policy network makes decision based on the planned  $\tau$ -step future states along multiple (e.g., all) action directions. Then the multi-step reward for each action direction can be formulated as follows:
$$
\begin{array}{l} \mathcal {R} (s _ {j}, a _ {j}) = \mathbb {E} \Big [ \sum_ {t = j} ^ {j + \tau} r (s _ {t}, a _ {t}) \Big ], \\ \text {w h e r e} s _ {t + 1} \sim \Psi \left(s _ {t + 1} \mid s _ {t}, a _ {t}\right), \tag {2} \\ \end{array}
$$
where  $\Psi (\cdot)$  is the learned model of the environment (e.g., one-step transition model). In our case, as the environment is only determined by the parameter value for the generator model, we specify one independent generator model at each planning direction. That is, over the  $N$  action directions, we will reserve one model to expand each action direction. After planning, the decision of which sample to use is made deterministic by selecting  $\operatorname{argmax}_k R(s_j,a_k)$ . The aggressive planning
scheme enables the decision making to consider prioritizing the actions with better long-term effect. This would typically significantly accelerate the exploration efficiency for the policy training. Such planning would only take effect on the action sampling part, i.e., which samples to be activated to choose would be determined by planning. With this design, the learning part is unaffected and conveniently inherits a model-free nature.
4.2 Reward Scaling
One problem in our prediction-loss based reward modeling is that the reward signal would have a noisy distribution whose mean would gradually decay and eventually converge to 0. In order to resolve it, inspired by many principled reinforcement learning methods such as Actor-critic (Konda and Tsitsiklis, 1999) and PopArt (Hessel et al., 2019), we propose a simple yet effective reward scaling approach to normalize the reward signal to a static distribution. To this end, our normalization method stores the scalar reward signals within a recent window, which would typically consume very minimal amount of memory. Then when we update the policy model, we normalize the reward signal for each transition in the following manner,
$$
\hat {R} _ {j} = \frac {R _ {j} - R ^ {\min}}{R ^ {\max} - R ^ {\min}} \tag {3}
$$
where  $R_{j}$  and  $\hat{R}_j$  are the raw and normalized reward signals,  $R^{min}$  and  $R^{max}$  are the minimal and maximal values obtained from the window of rewards. With the normalized reward signal, the data
selection policy could be optimized by the REINFORCE algorithm (Sutton et al., 1999). We present the pseudocode for MB-RPG in Algorithm 1.
In this section, we present extensive empirical evaluation results on comparing our method with its various counterparts on four commonly used paraphrase generation datasets<sup>1</sup>.
Evaluation Datasets. For comparison, we consider to adopt both supervised and unsupervised datasets. Note that our method adopts a semi-supervised setting where the target paraphrases are
generated following a retrieval-based method introduced in Section 3.1 and thus alleviates the need for golden labeled target data. Overall, we experiment with the following four datasets. The first two datasets for supervised setting and the last two for unsupervised scenario:
- Quora-s: corresponds to the Quora Question Pairs (QQP) dataset<sup>2</sup> which consists of 400,000 question pairs and each pair comes with a binary tag telling whether that pair is paraphrase or not. To split the dataset for training and testing, we follow the existing works (Li et al., 2018; Kazemnejad et al., 2020; Ding et al., 2021) and use randomly sampled non-overlapping parallel pairs with sizes 100K, 3K and 30K for training, validation and testing, respectively.
- Twitter: is the Twitter News URL Corpus<sup>3</sup> proposed by Lan et al. (2017). The dataset is created by large-scale sentential paraphrases from Twitter by linking tweets through shared URL. Following (Li et al., 2018; Kazemnejad et al., 2020; Ding et al., 2021), we randomly sample 110K instances from automatically labelled data as our training dataset and two non-overlapping datasets of sizes 1K and 5K from the human-annotated data to form the validation set and the testing set, respectively.
- Quora-U: is the unsupervised version of Quoras. To make a fair comparison, we follow the settings of the works (Miao et al., 2019; Liu et al., 2020b) and use two sets of 3K and 20K non-overlapping pairs as the validation set and the testing set, respectively.
- MSCOCO: is the COCO image captioning dataset<sup>4</sup>. It consists of over 500K captioning paraphrase pairs for more than 120K images. To create the datasets for training and testing we follow Lin et al. (2014) to split the dataset and adopt an identical evaluation protocol presented in Liu et al. (2020b).