SlowGuess/NAACL_2022_OCR huggingface dataset:
Natural Language Generation with Truncated Reinforcement Learning
Samovar, Télecom SudParis, Institut Polytechnique de Paris, Palaiseau
CMAP, École Polytechnique, Institut Polytechnique de Paris, Palaiseau
Guillaume Quispe and Charles Ollion
Sylvain Le Corff
Deep Mind
Olivier Pietquin
This paper introduces TRUncated ReinForcement Learning for Language (TrufLL), an original approach to train conditional language models without a supervised learning phase, by only using reinforcement learning (RL). As RL methods unsuccessfully scale to large action spaces, we dynamically truncate the vocabulary space using a generic language model. TrufLL thus enables to train a language agent by solely interacting with its environment without any task-specific prior knowledge; it is only guided with a task-agnostic language model. Interestingly, this approach avoids the dependency to labelled datasets and inherently reduces pretrained policy flaws such as language or exposure biases. We evaluate TrufLL on two visual question generation tasks, for which we report positive results over performance and language metrics, which we then corroborate with a human evaluation. To our knowledge, it is the first approach that successfully learns a language generation policy without pre-training, using only reinforcement learning.
Since the development of generic language models trained on massive unlabelled text corpora (Radford et al., 2019; Brown et al., 2020), state-of-the-art language processing systems rely on sequential transfer learning (Ruder, 2019). The pretrained Language Model (LM) is fine-tuned on the downstream task using a standard supervised learning (SL)
Figure 1: (left) In a conditional language generation task as VQG, TruffLL truncates the vocabulary space by using a language model. Here, 'run,' and 'the' are syntactically incorrect and thus truncated. Yet, 'car' is not trimmed as the LM is not visually grounded. (right) In a VQG training loop, the agent generates a question given an image-answer pair, which is then fed to a VQA model predicting an expected answer. If both answers match, the agent is rewarded.
objective (Wu et al., 2019; Peters et al., 2019). Yet, such an approach suffers from several issues (Chen et al., 2020): (i) catastrophic forgetting when a model forgets previously learned knowledge and overfits to target domains, (ii) computational inefficiency from fine-tuning billion-parameters networks, and (iii) the need of supervised datasets. Moreover, task-specific language models learned with SL suffer from well-studied text degeneration issues (Holtzman et al., 2019), such as the exposure bias (Bengio et al., 2015), language biases (Saleh et al., 2020; Jaques et al., 2020), or a lack of diversity (Li et al., 2015).
On the other hand, text generation can be naturally framed as a sequential decision making problem, with the sequence of words seen as successive actions over a vocabulary. Thus, some researchers have recently focused on learning language models using instead
$^{1}$ Code is available at https://github.com/AMDonati/RL-NLP
Human Language Technologies, pages 12 - 37
Reinforcement Learning (RL) (Strub et al., 2017; Das et al., 2017; Narasimhan et al., 2015). RL methods allow acquiring language through interactions within rich and diverse environments (Luketina et al., 2019), help understanding language acquisition and language pragmatics (Lazaridou et al., 2016; Bisk et al., 2020). "Reward is enough" (Silver et al., 2021) highlights the necessity of using RL for AI systems to acquire language in its full richness. Indeed, (i) language may be intertwined with other modalities of action and observation, (ii) the utility of language varies according to situations and behaviours, (iii) it is consequential and purposeful, and (iv) some linguistic problems are better solved dynamically, through experience (such as using a diplomatic tone in a speech.) In addition, RL allows optimizing a non-differentiable learning signal, hence handles more diverse objective functions, and also avoids some of the text degeneration issues previously mentioned. So far, RL-based text-generation tasks have relied on a pre-training phase to ease learning: the policy language model is trained with SL on the task dataset, before being fine-tuned with policy gradient methods (Sutton et al., 1999) on the task at hand. Those approaches often require human-labelled datasets. Besides, combining pre-training and fine-tuning phases either barely change the policy distribution, or induces language drift (Lazaridou et al., 2020; Lu et al., 2020b), i.e the generated language drifts semantically or syntactically from natural language.
In this paper, we aim at learning a conditional language model using RL without a pre-training phase, so that (i) we get free from datasets with human annotations, and (ii) we avoid the text generation flaws induced by the common methods. While appealing, such an approach requires overcoming the hurdle of the combinatorial language action space, a vocabulary usually containing more than 10,000 words. Yet, while large and discrete, a language action space contains a specific structure, made of all the syntactical and semantics rules of a given language. TrufLL leverages such structure to drive the exploration of the RL-based language agent during training. At each time step of the text generation process, TrufLL truncates its effective action space to a small subset of words provided by a pretrained task-agnostic language model. Such an approach injects a generic prior linguistic knowledge into the RL algorithm, is usable on tasks lacking in-domain labeled data, and can be easily transferred to new RL-based text generation tasks. Thus, TrufLL can
be applied to any language generation task given a generic LM and a reward. We here evaluate it on two Visual Question Generation (VQG) tasks, the synthetic CLEVR dataset (Johnson et al., 2017), and the natural language VQAv2 dataset (Goyal et al., 2017). Unlike alternative RL without pre-training approaches, TrufLL manages to ask meaningful and valid questions on large vocabularies, exhibiting success rate and language metrics close to pretrain models with labeled data, while producing more original language.
Language Generation as an RL Problem. We cast the word-based text generation task as a Markov Decision Process to apply RL methods (Sutton et al., 1998). In this setting, a language model agent generates a sequence of words  $w_{<t} = (w_0, w_1, \dots, w_{t-1})$  drawn from a vocabulary  $\mathcal{V}$ , given an initial context  $c$  associated with a reward  $r_t$ . Translation, text summarization or image captioning are examples of such tasks respectively using a source sentence, a text article, or an image as a context ( $c$ ). During this process, the agent may be rewarded with language scores (Ranzato et al., 2016), human preferences (Stiennon et al., 2020) or task completion scores (Strub et al., 2017).
Formally, a language generation agent is defined by a policy  $\pi_{\theta}$  (a distribution over  $\mathcal{V}$ ) parametrized by  $\theta$ , first initialized with the context  $c$ . At each time step  $t$ , the agent samples a new word  $w_{t}$  from its policy  $\pi_{\theta}(w_t | w_{<t}, c)$ . It moves to a new state  $(w_{<t+1}, c)$  and receives a reward  $r_t = r(w_{<t}, c, w_t)$ , where  $r$  is a reward function relative to the language task. The RL language agent aims to learn a policy that maximizes  $\mathbb{E}_{\pi_{\theta}}[\sum_{t=0}^{T} r_t]^2$  while generating the sequence of words  $w_{<T}$ , where  $\mathbb{E}_{\pi_{\theta}}$  is the expectation under  $\pi_{\theta}$ , and  $T$  the maximal length of the words sequence.
Policy Gradient This optimization process may be performed through Policy Gradient (PG) algorithms (Sutton et al., 1999). In the language literature, REINFORCE (Williams, 1992) has been used as a simple Monte Carlo approximation of this gradient (Strub et al., 2017; Li et al., 2016). Yet, in this paper, we use a Proximal Policy Optimization approach (PPO) (Schulman et al., 2017) to have a lower variance and better convergence rate; PPO clips the gradient estimate to have smooth policy updates. For all  $0 \leq t \leq T$ , let  $s_t = (w_{<t}, c)$  and  $a_t = w_t$  be the state and action at time  $t$ . Policy gradient methods
2We cast the language modelling as an episodic problem with  $\gamma = 1$  and omit the discount factor in the paper for clarity.
minimize the objective:
$$
L _ {\mathrm {p g}} (\theta) = \mathbb {E} _ {\pi_ {\theta}} \left[ \sum_ {t = 0} ^ {T} \log \pi_ {\theta} (a _ {t} | s _ {t}) \hat {A} _ {t} \right],
$$
where  $\hat{A}_t$  is an estimator of the advantage function, here defined as  $\hat{A}_t = \sum_{u=t}^T r_u - V_\phi(s_t)$  with  $V_\phi(s)$  an estimator of the value function  $V_{\pi_\theta}(s) = \mathbb{E}_{\pi_\theta}[\sum_{u=t}^T r(s_u, a_u)|s_t = s]$ . PPO then keeps track of the previous policy  $\pi_{\theta_{old}}$  before the PG update to compute the training objective:
$$
L _ {\mathrm {p p o}} (\theta) = \mathbb {E} _ {\pi_ {\theta_ {o l d}}} \left[ \sum_ {t = 0} ^ {T} \rho_ {t} ^ {\theta} \hat {A} _ {t} \wedge \mathrm {c l i p} (1 - \epsilon , \rho_ {t} ^ {\theta}, 1 + \epsilon) \hat {A} _ {t} \right],
$$
where for all real numbers  $a, b, a \wedge b = \min(a, b)$ ,  $\rho_t^\theta = \pi_\theta(a_t|s_t) / \pi_{\theta_{old}}(a_t|s_t)$ ,  $\epsilon$  is a hyper-parameter controlling the magnitude of the policy updates, and  $\mathrm{clip}(a,x,b)$  is the function that clips  $x$  in interval  $[a,b]$ . The expectation is estimated in practice using a Monte Carlo approach, with an empirical average over a finite batch of episodes, i.e. a succession of transitions  $(s_t,a_t \sim \pi_{\theta_{old}}(.|s_t),r_t)$  from an initial state  $s_0$  to a terminal state  $s_T$ . Finally, the training loss is completed first with a value-based loss to learn the baseline  $V_\phi$  that reduces the gradient variance; it computes for each timestep  $t$  of an episode the mean squared error  $|\sum_{u=t}^{T}r_u - V_\phi(s_t)|^2$ . Secondly, the loss is completed with an entropy term to soften the policy distribution, which computes for each timestep  $t$  of an episode  $H(\pi_\theta(a_t|s_t))$ , where  $H$  is the entropy function.
3 TrufLL
We here aim at making RL methods feasible in the language setting by dynamically reducing the action space, i.e., by restricting the language agent to select a word within a subset of the vocabulary at each time step. We detail below the action space's truncation model and the associated RL algorithm to learn the language agent.
3.1 Dynamic Vocabulary Truncation
TruFLL combines two distinct language models, which share the same vocabulary  $\mathcal{V}$ : a RL language agent  $\pi_{\theta}$  and a pretrained language model  $f_{LM}$ . At each timestep  $t$ , TruFLL restricts the vocabulary space of the RL language agent with:
$$
\mathcal {V} _ {t} ^ {-} = \{w | w \in \mathcal {V}, g _ {\text {t r u n c}} (w | w _ {<   t}) = 1 \},
$$
where  $g_{trunc}$  is a truncation function based on  $f_{LM}$  which either associates 0 or 1 with each word in the vocabulary given the past words  $w_{<t}$ . From a language modelling perspective, the vocabulary space of the language agent is reduced from  $\mathcal{V}$  to  $\mathcal{V}^{-}$  where  $|\mathcal{V}^{-}| \ll |\mathcal{V}|$ , with  $| \cdot |$  the cardinal of a finite set. From a RL perspective, the RL agent follows a truncated policy  $\pi_{\theta}^{-}$  which only samples actions over the subset  $\mathcal{V}^{-}$ . In practice, such a policy is computed using a masked softmax function over the truncated vocabulary  $\mathcal{V}_t^{-}$ :  $\pi_{\theta}^{-}(.|w_{<t},c) = \text{softmax}(m * \text{logits}_{\pi_{\theta}}(w_{<t},c))$  where  $m = 1$  when  $g_{trunc}(w|w_{<t}) = 1$  otherwise  $m = -\infty$ .
3.2 Truncation Functions
We here list the different truncation functions  $g_{trunc}$  explored through the paper.
Top-k words: This function selects the  $k$  words with the highest probability given by  $f_{LM}(|w_{<t}|)$ :
$$
g _ {\mathrm {t o p} (\mathrm {k})} (w _ {t} | w _ {<   t}; k) = \mathbb {1} _ {w _ {t} \in \mathrm {t o p} (\mathrm {k}) (f _ {L M} (. | w _ {<   t})).}
$$
Probability threshold  $(\alpha)$ : This function only keeps words having a probability  $f_{LM}(.|w_{< t})$  greater than  $\alpha$ :
$$
g _ {\mathrm {p} _ {\mathrm {t h}} (\alpha)} \left(w _ {t} \mid w _ {<   t}; \alpha\right) = \mathbb {1} _ {f _ {L M} \left(w _ {t} \mid w _ {<   t}\right) > \alpha}.
$$
Top-p: This function is based on nucleus sampling (Holtzman et al., 2019), and it keeps the most likely words contained in a probability mass  $p$  of  $f_{LM}(.|w_{< t})$ . Formally, we define  $\mathcal{V}_t^p$  as:
$$
\mathcal{V}_{t}^{p} = \operatorname *{argmin}_{|V_{t}|,V_{t}\subset \mathcal{V}}\left\{w|w\in V_{t},\sum_{w\in V_{t}}f_{LM}(w|w_{<  t}) > p\right\} ,
$$
and readily,  $g_{\mathrm{top(p)}}(w_t|w_{< t};p) = \mathbb{1}_{w_t\in \mathcal{V}_t^p}$ .
Sample  $(k)$ : This function randomly samples  $k$  words from the language model with replacement to directly build the truncated vocabulary:
$$
g _ {\text {s a m p l e} (\mathrm {k})} \left(w _ {t} \mid w _ {<   t}; k\right) = \mathbb {1} _ {w _ {t} \in \left\{w _ {i} \sim f _ {L M} (., | w _ {<   t}) i \in [ 1, \dots , k ] \right\}}.
$$
Only  $\mathrm{top}(\mathbf{k})$  provides a fixed number of words at each time step.  $\mathrm{p_{th}}(\alpha)$ ,  $\mathrm{top}(\mathfrak{p})$ , and sample(k) have a dynamic truncation, whose size at  $t$  depends on the language model entropy.
3.3 Task-Specific vs. Generic LM
We benchmark two types of language models for truncation. On the one hand, we use an external language model pretrained on a large task-agnostic language corpora. Such a model provides a generic linguistic prior to the RL agent exploration process, solely encoding syntactic and semantic information. On the other hand,
3Note that other TD-based losses are applicable (Sutton et al., 1998; Schulman et al., 2016; Espeholt et al., 2018).
we use a task-related language model pretrained on the supervised dataset associated with the task. Such a model provides a task-specific linguistic prior to the RL language agent, and captures language pragmatics. We emphasize that this paper aims at leveraging task-agnostic language models as they discard the need for task-specific data. For the sake of completeness, we also study the truncation with the task-related LM as an additional benchmark to assess our approach.
We here list the experimental setting and detail the network and hyperparameters in Appendix A.4.
4.1 Visual Question Generation
We showcase TrufLL on the task of Visual Question Generation (VQG) (Mostafazadeh et al., 2016), which is a form of Visual Jeopardy!™ (Ferrucci, 2012). There, the language agent observes an image-answer pair and has to generate a question that results in a similar answer, as illustrated in Figure 1. Such a task presents multiple advantages. First, by combining vision, scene understanding and language generation, it requires high-level reasoning and exhibits a large spectrum of language difficulties. Secondly, the success criterion is naturally non-differentiable, hence a natural fit for RL methods. Such a criterion, unlike metrics based on ground-truth sentences, allows generating diverse grounded questions given an image-answer pair.
Formally, the initial context  $c$  is composed of the image-answer pair  $(\mathcal{I},\mathcal{A})$ . The RL agent then generates a sequence of words  $w_{<t}$  of maximum length  $T$ . We then provide the generated question to a pretrained VQA model. This model takes as inputs the image  $\mathcal{I}$ , the generated question  $w_{<t}$  and outputs a predicted answer  $\hat{\mathcal{A}}$ . Finally, the agent receives a reward  $r(w_t,w_{<t},c)$  based on  $\mathcal{A}$  and  $\hat{\mathcal{A}}$ .
We evaluate TrufLL on the CLEVR and VQAv2 datasets to simulate large-scale VQG datasets. The two datasets have been originally created for the task of Visual Question Answering (VQA), i.e. for multi-modal classification algorithms predicting an answer given an image-question pair.
CLEVR The CLEVR VQA dataset (Johnson et al., 2017) is made of template questions on synthetic images, which contain simple objects with four distinct properties (shape, material, color, size). The vocabulary contains 86 words and 28 potential answers, making it a valuable proof of concept for assessing
TrufLL. Both language models are single-layer LSTMs (Hochreiter and Schmidhuber, 1997) with 512 units, and 512 word embedding dimension. The task-specific LM is trained over the full train dataset of CLEVR questions. The external language model is trained on the mixture of CLOSURE (Bahdanau et al., 2019) and CLEVR-Dialog (Kottur et al., 2019) datasets. Although those two datasets share the CLEVR vocabulary, their language distribution differs from vanilla CLEVR. Finally, we use a pretrained GT-Vector-NMN (Bahdanau et al., 2019) to compute the reward  $r(w_{t},w_{< t},c) = \mathbb{1}_{\mathcal{A} = \hat{\mathcal{A}},t = T - 1}$ , where  $\mathbb{1}$  is the indicator function.
VQAv2 The VQAv2 dataset (Goyal et al., 2017) is made of natural language and open-formed questions on images from the MS-Coco Dataset (Lin et al., 2014). It has a vocabulary of 14,810 words and 3,149 answers. The task-specific language model is a one-layer LSTM with 512 units and a 512 word embedding dimension, pretrained over the full training dataset of VQAv2 questions. The External Language Model is Open-AI's GPT-2 (Radford et al., 2019). The original language model outputs a probability distribution over 50,257 tokens, but we use a masked softmax function to restrict the probability distribution to the 14,810 tokens of the VQAv2 dataset. Unlike most NLP tasks relying on pretrained generic language models, we do not fine-tune it on the task dataset. Instead, we leverage the few-shot generalization capabilities of GPT-2, by feeding the language model with the prompt "Here are a few examples:" followed by 100 random questions  $q_{<100}$  from the dataset. The truncation is then based on the probability distribution  $f_{LM}^{gpt2}(\cdot|q_{<100}, w_{<t})$ . Finally, we used a pretrained ViBERT to compute the reward (Lu et al., 2020a). Given the large number of answers, we use as reward a decreasing function of the rank of the reference answer  $\mathrm{rk}(\mathcal{A})$ :  $r(w_t, w_{<t}, c) = \mathbb{1}_{\mathrm{rk}(\mathcal{A}) \leq 10, t=T-1} e^{-\mathrm{rk}(\mathcal{A})/2}$ , as further explained in Appendix A.5.
In these two settings, we acknowledge that the task dataset is still used to train the VQA models. Please note that the VQA modules are only used to model the environment, i.e. to provide a positive/negative feedback to the agent. In other settings, TrufLL would still work if we replace the VQA model by any language interface: text-game (e.g. Zork), expert-systems, or humans. Here, we only use the VQG framework as a proof of concept that natural language can be learned through pure interaction given any task reward. Other language generation applications are discussed in Section 5.3.
In this paper, we aim to show that a RL language agent can be trained from scratch, i.e. without the usual pre-training phase by solely interacting with another language system, the VQA model, when supported by truncation methods. The truncation with the task-related LM is referred to as TrufLL (Task-LM), while the one with the External LM is referred as TrufLL (Ext-LM). We first emphasize the difficulty of training an RL language agent without a supervised pre-training phase through two baselines. We trained a simple on-policy PPO algorithm without any action space pruning, and refer to it as scratch. Then, we added a Kullback-Leibler (KL) regularization term to the loss,  $\lambda_{\mathrm{KL}}\mathrm{KL}(\pi_{\theta}\| f_{LM})$  with  $\lambda_{\mathrm{KL}} > 0$ , to incorporate language prior to the agent as in (Jaques et al., 2017, 2019). We refer to it as scratch + KL-task when distilling the task-specific language model, and scratch + KL-ext with the external language model. Finally, we include two baselines with a pre-training phase. We trained a language agent on the task-dataset with a log-likelihood objective, and refer to it as pretrain. Then, we fine-tune the pretrained language agent with PPO without truncation, and refer to it as pretrain + RL fine-tune. These two baselines should be viewed as gold standards as they rely on task-related data; additionally, pretrain + RL fine-tune is today the state-of-the-art method for learning RL-based LM.
4.4 Metrics and Evaluation Methods
Evaluating text generation is an open-research problem in language literature. We decompose automatic language evaluation into three categories to assess different facets of language, and perform as well a human evaluation study.
Performance metrics. We measure the task-completion score or recall @ 1 which states whether the target answer  $\mathcal{A}$  is the top answer of the VQA models, and the recall @ 5 (R@5), which assesses whether  $\mathcal{A}$  is in the 5 top answers. These scores measure the task-solving abilities of the agent, but they are also conditioned by the VQA model abilities.
Language Metrics. First, we used n-grams metrics, BLEU (Papineni et al., 2002), METEOR (Banerjee and Lavie, 2005) and CIDEr (Vedantam et al., 2015), to measure the similarity between the generated question and the reference questions in the evaluation set. While those scores can capture syntactic and semantic properties of language, they also fall short when dealing with open-form language, e.g. an
identical answer may arise from two non-overlapping but syntactically correct questions. Thus, we also compute two metrics assessing the quality of the language independently of reference questions, the perplexity of the question given an external LM (ppl-e), and its perplexity given the task-related LM (ppl-t).
Diversity Metrics. We here estimate a self-BLEU (sBLEU) score (Zhang et al., 2017) over 10 questions generated on the same image-answer pair. Although such score detects potential mode collapse, i.e., when the language utters identical sequences of words, it also values babbling, i.e., outputting random words. We thus also measure the probability mass of the ten most frequent words (Choshen et al., 2020), and refer to it as peakiness (peak).
Human Evaluation. On the VQAv2 task, we also performed human evaluation by surveying 53 participants on the first 50 questions produced by some of the models at test time. The study (further detailed in Appendix C) is based on pairwise comparison of question samples produced by the concurrent algorithms according to four criteria. First, we evaluated the language quality of the question samples, by asking the participants to select the most syntactically and semantically correct question among the two samples of the questions pair. Secondly, we evaluated language grounding, i.e. adequacy of the sample to the image-answer pair, by asking the participants to select the question most suitable given the two elements. Thirdly, we evaluated the language originality and diversity, by asking participants to select the question the most different from the dataset reference question. Finally, we evaluated the number of syntax errors by asking participants to tick the question if it is grammatically incorrect. Examples of questions asked during the study are included in the Appendix C.
4.5 Sampling methods for text generation
When generating text from a trained language model, the quality and diversity of samples depend on the decoding algorithm (Zhang et al., 2020). We consider three text generation methods. greedy uses the argmax of the policy, while sampling uses the multinomial distribution. Finally, we sampled ten text sequences from the policy, and selected the one with the lowest perplexity according to the external language model, and refer to it as  $lm$ -ranking. This process has been used recently in Text-to-Image Generation tasks (Ramesh et al., 2021).