Venue: NAACL 2022

# Learning

Natural Language Generation with Truncated Reinforcement Learning

Alice Martin

Samovar, Telecom SudParis, Institut Polytechnique de Paris, Palaiseau

CMAP, Ecole Polytechnique, Institut Polytechnique de Paris, Palaiseau

Guillaume Quispe and Charles Ollion

CMAP, Ecole Polytechnique, Institut Polytechnique de Paris, Palaiseau

Sylvain Le Corff

Samovar, Telecom SudParis, Institut Polytechnique de Paris, Palaiseau Florian Strub

Deep Mind &Olivier Pietquin

Google Brain

###### Abstract

This paper introduces TRUncated ReinForement Learning for Language (TrufLL), an original approach to train conditional language models without a supervised learning phase, by only using reinforcement learning (RL). As RL methods unsuccessfully scale to large action spaces, we dynamically truncate the vocabulary space using a generic language model. Trufall thus enables to train a language agent by solely interacting with its environment without any task-specific prior knowledge; it is only guided with a task-agnostic language model. Interestingly, this approach avoids the dependency to labelled datasets and inherently reduces pretrained policy flaws such as language or exposure biases. We evaluate Trufall on two visual question generation tasks, for which we report positive results over performance and language metrics, which we then corroborate with a human evaluation. To our knowledge, it is the first approach that successfully learns a language generation policy without pre-training, using only reinforcement learning. 1

Footnote 1: Code is available at [https://github.com/AMDonati/RL-NLP](https://github.com/AMDonati/RL-NLP)

_Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 12 - 37

July 10-15, 2022 G2022 Association for Computational Linguistics

## 1 Introduction

Since the development of generic language models trained on massive unlabelled text corpora Radford et al. (2019); Brown et al. (2020), state-of-the art language processing systems rely on _sequential transfer learning_Ruder (2019). The pretrained Language Model (LM) is fine-tuned on the downstream task using a standard supervised learning (SL) objective Wu et al. (2019); Peters et al. (2019). Yet, such an approach suffers from several issues Chen et al. (2020): (i) catastrophic forgetting when a model forgets previously learned knowledge and overfits to target domains, (ii) computational inefficiency from fine-tuning billion-parameters networks, and (iii) the need of supervised datasets. Moreover, task-specific language models learned with SL suffer from well-studied text degeneration issues Holtzman et al. (2019), such as the exposure bias Bengio et al. (2015), language biases Saleh et al. (2020); Jaques et al. (2020), or a lack of diversity Li et al. (2015).

On the other hand, text generation can be naturally framed as a sequential decision making problem, with the sequence of words seen as successive actions over a vocabulary. Thus, some researchers have recently focused on learning language models using instead

Figure 1: (left) In a conditional language generation task as VGG, TruthL truncates the vocabulary space by using a language model. Here, ’run’, ‘and ‘the’ are syntactically incorrect and thus truncated. Yet, ‘car’ is not trimmed as the LM is not visually grounded. (right) In a VQG training loop, the agent generates a question given an image-answer pair, which is then fed to a VQA model predicting an expected answer. If both answers match, the agent is rewarded.



Reinforcement Learning (RL) (Strub et al., 2017; Das et al., 2017; Narasimhan et al., 2015). RL methods allow acquiring language through interactions within rich and diverse environments (Luketina et al., 2019), help understanding language acquisition and language pragmatics (Lazaridou et al., 2016; Bisk et al., 2020). "Reward is enough" (Silver et al., 2021) highlights the necessity of using RL for AI systems to acquire language in its full richness. Indeed, (i) language may be intertwined with other modalities of action and observation, (ii) the utility of language varies according to situations and behaviours, (iii) it is consequential and purposeful, and (iv) some linguistic problems are better solved dynamically, through experience (such as using a diplomatic tone in a speech.) In addition, RL allows optimizing a non-differentiable learning signal, hence handles more diverse objective functions, and also avoids some of the text degeneration issues previously mentioned. So far, RL-based text-generation tasks have relied on a pre-training phase to ease learning: the policy language model is trained with SL on the task dataset, before being fine-tuned with policy gradient methods (Sutton et al., 1999) on the task at hand. Those approaches often require human-labelled datasets. Besides, combining pre-training and fine-tuning phases either barely change the policy distribution, or induces _language drift_(Lazaridou et al., 2020; Lu et al., 2020b), i.e the generated language drifts semantically or syntactically from natural language.

In this paper, we aim at learning a conditional language model using _RL without a pre-training phase_, so that (i) we get free from datasets with human annotations, and (ii) we avoid the text generation flaws induced by the common methods. While appealing, such an approach requires overcoming the hurdle of the combinatorial language action space, a vocabulary usually containing more than 10,000 words. Yet, while large and discrete, a language action space contains a specific structure, made of all the syntactical and semantics rules of a given language. TrufLL leverages such structure to drive the exploration of the RL-based language agent during training. At each time step of the text generation process, TrufLL truncates its effective action space to a small subset of words provided by a pretrained task-agnostic language model. Such an approach injects a generic prior linguistic knowledge into the RL algorithm, is usable on tasks lacking in-domain labeled data, and can be easily transferred to new RL-based text generation tasks. Thus, TrufLL can be applied to any language generation task given a generic LM and a reward. We here evaluate it on two Visual Question Generation (VQG) tasks, the synthetic CLEVR dataset (Johnson et al., 2017), and the natural language VQAv2 dataset (Goyal et al., 2017). Unlike alternative RL without pre-training approaches, TrufLL manages to ask meaningful and valid questions on large vocabularies, exhibiting success rate and language metrics close to pretrain models with labeled data, while producing more original language.

## 2 Background

Language Generation as an RL Problem.We cast the word-based text generation task as a Markov Decision Process to apply RL methods (Sutton et al., 1998). In this setting, a language model agent generates a sequence of words \(w_{<t}=(w_{0},w_{1},...,w_{t-1})\) drawn from a vocabulary \(\mathcal{V}\), given an initial context \(c\) associated with a reward \(r_{t}\). Translation, text summarization or image captioning are examples of such tasks respectively using a source sentence, a text article, or an image as a context (\(c\)). During this process, the agent may be rewarded with language scores (Ramzato et al., 2016), human preferences (Siennon et al., 2020) or task completion scores (Strub et al., 2017).

Formally, a language generation agent is defined by a policy \(\pi_{\theta}\) (a distribution over \(\mathcal{V}\)) parametrized by \(\theta\), first initialized with the context \(c\). At each time step \(t\), the agent samples a new word \(w_{t}\) from its policy \(\pi_{\theta}(w_{t}|w_{<l},c)\). It moves to a new state \((w_{<l+1},c)\) and receives a reward \(r_{t}=r\left(w_{<t},c,w_{t}\right)\), where \(r\) is a reward function relative to the language task. The RL language agent aims to learn a policy that maximizes \(\mathbb{E}_{\pi_{\theta}}[\sum_{t=0}^{T}r_{t}]\),2 while generating the sequence of words \(w_{<T}\), where \(\mathbb{E}_{\pi_{\theta}}\) is the expectation under \(\pi_{\theta}\), and \(T\) the maximal length of the words sequence.

Footnote 2: We cast the language modelling as an episodic problem with \(\gamma=1\) and omit the discount factor in the paper for clarity.

Policy GradientThis optimization process may be performed through Policy Gradient (PG) algorithms (Sutton et al., 1999). In the language literature, REINFORCE (Williams, 1992) has been used as a simple Monte Carlo approximation of this gradient (Strub et al., 2017; Li et al., 2016).Yet, in this paper, we use a Proximal Policy Optimization approach (PPO) (Schulman et al., 2017) to have a lower variance and better convergence rate; PPO clips the gradient estimate to have smooth policy updates. For all \(0\leq t\leq T\), let \(s_{t}=(w_{<t},c)\) and \(a_{t}=w_{t}\) be the state and action at time \(t\). Pol 

minimize the objective:

\[L_{\rm pg}(\theta) = \mathbb{E}_{\pi_{\theta}}\Bigg{[}\sum_{t=0}^{T}\!\log\!\pi_{\theta}( a_{t}|s_{t})\hat{A}_{t}\Bigg{]},\]

where \(\hat{A}_{t}\) is an estimator of the advantage function, here defined as \(\hat{A}_{t}=\sum_{u=t}^{T}r_{u}-V_{\phi}(s_{t})\) with \(V_{\phi}(s)\) an estimator of the value function \(V_{\pi_{\theta}}(s)=\mathbb{E}_{\pi_{\theta}}[\sum_{u=t}^{T}r(s_{u},a_{u})|s_{ t}=s]\). PPO then keeps track of the previous policy \(\pi_{\theta_{old}}\) before the PG update to compute the training objective:

\[L_{\rm ppo}(\theta) = \mathbb{E}_{\pi_{\theta_{old}}}\Bigg{[}\sum_{t=0}^{T}\rho_{t}^{ \theta}\hat{A}_{t}\wedge\mathrm{clip}(1-\epsilon,\rho_{t}^{\theta},1+\epsilon )\hat{A}_{t}\Bigg{]},\]

where for all real numbers \(a\), \(b\), \(a\wedge b=\min(a,b)\), \(\rho_{t}^{\theta}=\pi_{\theta}(a_{t}|s_{t})/\pi_{\theta_{old}}(a_{t}|s_{t})\), \(\epsilon\) is a hyper-parameter controlling the magnitude of the policy updates, and \(\mathrm{clip}(a,x,b)\) is the function that clips \(x\) in interval \([a,b]\). The expectation is estimated in practice using a Monte Carlo approach, with an empirical average over a finite batch of episodes, i.e a succession of transitions \(\big{(}s_{t},a_{t}\sim\pi_{\theta_{old}}(.|s_{t}),r_{t}\big{)}\) from an initial state \(s_{0}\) to a terminal state \(s_{T}\). Finally, the training loss is completed first with a value-based loss to learn the baseline \(V_{\phi}\) that reduces the gradient variance; it computes for each timestep \(t\) of an episode the mean squared error \(|\sum_{u=t}^{T}r_{u}-V_{\phi}(s_{t})|^{2}\).3 Secondly, the loss is completed with an entropy term to soften the policy distribution, which computes for each timestep \(t\) of an episode \(H(\pi_{\theta}(a_{t}|s_{t}))\), where \(H\) is the entropy function.

Footnote 3: Note that other TD-based losses are applicable (Sutton et al., 1998; Schulman et al., 2016; Espeholt et al., 2018).

## 3 TrufLL

We here aim at making RL methods feasible in the language setting by dynamically reducing the action space, i.e., by restricting the language agent to select a word within a subset of the vocabulary at each time step. We detail below the action space's truncation model and the associated RL algorithm to learn the language agent.

### Dynamic Vocabulary Truncation

TrufLL combines two distinct language models, which share the same vocabulary \(\mathcal{V}\): a RL language agent \(\pi_{\theta}\) and a pretrained language model \(f_{LM}\). At each timestep \(t\), TrufLL restricts the vocabulary space of the RL language agent with:

\[\mathcal{V}_{t}^{-} = \big{\{}w|w\in\mathcal{V},g_{trunc}(w|w_{<t})=1\big{\}},\]

where \(g_{trunc}\) is a truncation function based on \(f_{LM}\) which either associates 0 or 1 with each word in the vocabulary given the past words \(w_{<l}\). From a language modelling perspective, the vocabulary space of the language agent is reduced from \(\mathcal{V}\) to \(\mathcal{V}^{-}\) where \(|\mathcal{V}^{-}|\ll|\mathcal{V}|\), with \(|\cdot|\) the cardinal of a finite set. From a RL perspective, the RL agent follows a truncated policy \(\pi_{\theta}^{-}\) which only samples actions over the subset \(\mathcal{V}^{-}\). In practice, such a policy is computed using a masked softmax function over the truncated vocabulary \(\mathcal{V}_{t}^{-}\): \(\pi_{\theta}^{-}(.|w_{<t},c)=\operatorname*{softmax}(m*logits_{\pi_{\theta}}( w_{<t},c))\) where \(m=1\) when \(g_{trunc}(w|w_{<t})=1\) otherwise \(m=-\infty\).

### Truncation Functions

We here list the different truncation functions \(g_{trunc}\) explored through the paper.

Top-k words:This function selects the \(k\) words with the highest probability given by \(f_{LM}(.|w_{<t})\):

\[g_{\rm top(k)}(w_{t}|w_{<t};k) = \mathds{1}_{w_{t}\in\text{top(k)}(f_{LM}(.|w_{<t}))}.\]

Probability threshold (\(\alpha\)):This function only keeps words having a probability \(f_{LM}(.|w_{<t})\) greater than \(\alpha\):

\[g_{p_{\rm th}(\alpha)}(w_{t}|w_{<t};\alpha) = \mathds{1}_{f_{LM}(w_{t}|w_{<t})>\alpha}.\]

Top-p:This function is based on nucleus sampling (Holtzman et al., 2019), and it keeps the most likely words contained in a probability mass \(p\) of \(f_{LM}(.|w_{<t})\). Formally, we define \(\mathcal{V}_{t}^{p}\) as:

\[\mathcal{Y}_{t}^{p} = \mathop{\arg\!\min}_{|V_{t}|,V_{t}\subset\mathcal{V}}\{w|w\in V_{t },\sum_{w\in V_{t}}f_{LM}(w|w_{<t})>p\},\]

and readily, \(g_{\rm top(p)}(w_{t}|w_{<t};p)=\mathds{1}_{w_{t}\in\mathcal{V}_{t}^{p}}\).

Sample (\(k\)):This function randomly samples \(k\) words from the language model with replacement to directly build the truncated vocabulary:

\[g_{\rm sample(k)}(w_{t}|w_{<t};k) = \mathds{1}_{w_{t}\in\{w_{i}\sim f_{LM}(.|w_{<t})\in[1,...,k]\}}.\]

Only \(\rm top(k)\) provides a fixed number of words at each time step. \(\rm p_{th}(\alpha)\), \(\rm top(p)\), and \(\rm sample(k)\) have a dynamic truncation, whose size at \(t\) depends on the language model entropy.

### Task-Specific vs. Generic LM

We benchmark two types of language models for truncation. On the one hand, we use an _external language model_ pretrained on a large task-agnostic language corpora. Such a model provides a generic linguistic prior to the RL agent exploration process, solely encoding syntactic and semantic information. On the other hand, 

we use a _task-related language model_ pretrained on the supervised dataset associated with the task. Such a model provides a task-specific linguistic prior to the RL language agent, and captures language pragmatics. We emphasize that this paper aims at leveraging task-agnostic language models as they discard the need for task-specific data. For the sake of completeness, we also study the truncation with the task-related LM as an additional benchmark to assess our approach.

## 4 Experimental Setting

We here list the experimental setting and detail the network and hyperparameters in Appendix A.4.

### Visual Question Generation

We showcase TrufLL on the task of Visual Question Generation (VQG) Mostafazadeh et al. (2016), which is a form of Visual Jeopardy!(rerrucci, 2012). There, the language agent observes an image-answer pair and has to generate a question that results in a similar answer, as illustrated in Figure 1. Such a task presents multiple advantages. First, by combining vision, scene understanding and language generation, it requires high-level reasoning and exhibits a large spectrum of language difficulties. Secondly, the success criterion is naturally non-differentiable, hence a natural fit for RL methods. Such a criterion, unlike metrics based on ground-truth sentences, allows generating diverse grounded questions given an image-answer pair.

Formally, the initial context \(c\) is composed of the image-answer pair \(({\mathcal{I}},{\mathcal{A}})\). The RL agent then generates a sequence of words \(w_{<t}\) of maximum length \(T\). We then provide the generated question to a pretrained VQA model. This model takes as inputs the image \({\mathcal{I}}\), the generated question \(w_{<t}\) and outputs a predicted answer \(\hat{\mathcal{A}}\). Finally, the agent receives a reward \(r(w_{t},w_{<t},c)\) based on \({\mathcal{A}}\) and \(\hat{\mathcal{A}}\).

### Datasets

We evaluate TrufLL on the CLEVR and VQAv2 datasets to simulate large-scale VQG datasets. The two datasets have been originally created for the task of Visual Question Answering (VQA), i.e. for multi-modal classification algorithms predicting an answer given an image-question pair.

ClevrThe CLEVR VQA dataset Johnson et al. (2017) is made of template questions on synthetic images, which contain simple objects with four distinct properties (shape, material, color, size). The vocabulary contains 86 words and 28 potential answers, making it a valuable proof of concept for assessing TrufLL. Both language models are single-layer LSTMs Hochreiter and Schmidhuber (1997) with 512 units, and 512 word embedding dimension. The task-specific LM is trained over the full train dataset of CLEVR questions. The external language model is trained on the mixture of CLOSURE Bahdanau et al. (2019) and CLEVR-Dialog Kottur et al. (2019) datasets. Although those two datasets share the CLEVR vocabulary, their language distribution differs from vanilla CLEVR. Finally, we use a pretrained GT-Vector-NMh Bahdanau et al. (2019) to compute the reward \(r(w_{t},w_{<t},c)=\mathds{1}_{{\mathcal{A}}=\hat{\mathcal{A}},t=T-1}\), where \(\mathds{1}\) is the indicator function.

Vqav2The VQAv2 dataset Goyal et al. (2017) is made of natural language and open-formed questions on images from the MS-Coco Dataset Lin et al. (2014). It has a vocabulary of 14,810 words and 3,149 answers. The task-specific language model is a one-layer LSTM with 512 units and a 512 word embedding dimension, pretrained over the full training dataset of VQAV2 questions. The External Language Model is Open-AI's GPT-2 Radford et al. (2019). The original language model outputs a probability distribution over \(50,257\) tokens, but we use a masked softmax function to restrict the probability distribution to the \(14,810\) tokens of the VQAv2 dataset. Unlike most NLP tasks relying on pretrained generic language models, we do not fine-tune it on the task dataset. Instead, we leverage the few-shot generalization capabilities of GPT-2, by feeding the language model with the prompt "Here are a few examples:" followed by 100 random questions \(q_{<100}\) from the dataset. The truncation is then based on the probability distribution \(f^{gpL2}_{LM}(\cdot|q_{<100},w_{<l})\). Finally, we used a pretrained VISERT to compute the reward Lu et al. (2020a). Given the large number of answers, we use as reward a decreasing function of the rank of the reference answer \(\mathrm{rk}({\mathcal{A}})\): \(r(w_{t},w_{<t},c)=\mathds{1}_{{\rm rk}({\mathcal{A}})\leq 10,t=T-1}e^{- \mathrm{rk}({\mathcal{A}})/2}\), as further explained in Appendix A.5.

In these two settings, we acknowledge that the task dataset is still used to train the VQA models. Please note that the VQA modules are only used to model the environment, i.e. to provide a positive/negative feedback to the agent. In other settings, TrufLL would still work if we replace the VQA model by any language interface: text-game (e.g. Zork), expert-systems, or humans. Here, we only use the VQG framework as a proof of concept that natural language can be learned _through pure interaction_ given any task reward. Other language generation applications are discussed in Section 5.3 



### Baselines

In this paper, we aim to show that a RL language agent can be trained from scratch, i.e. without the usual pre-training phase by solely interacting with another language system, the VQA model, when supported by truncation methods. The truncation with the task-related LM is referred to as TrufLL (Task-LM), while the one with the External LM is referred as TrufLL (Ext-LM). We first emphasize the difficulty of training an RL language agent without a supervised pre-training phase through two baselines. We trained a simple on-policy PPO algorithm without any action space pruning, and refer to it as _scratch_. Then, we added a Kullback-Leibler (KL) regularization term to the loss, \(\lambda_{\mathrm{KLI}}\mathrm{KL}(\pi_{\theta}||f_{LM})\), with \(\lambda_{\mathrm{KL}}>0\), to incorporate language prior to the agent as in [19, 2]. We refer to it as _scratch + KL-task_ when distilling the task-specific language model, and _scratch + KL-ext_ with the external language model. Finally, we include two baselines with a pre-training phase. We trained a language agent on the task-dataset with a log-likelihood objective, and refer to it as _pretrain_. Then, we fine-tune the pretrained language agent with PPO without truncation, and refer to it as _pretrain + RL fine-tune_. These two baselines should be viewed as gold standards as they rely on task-related data; additionally, _pretrain + RL fine-tune_ is today the state-of-the-art method for learning RL-based LM.

### Metrics and Evaluation Methods

Evaluating text generation is an open-research problem in language literature. We decompose automatic language evaluation into three categories to assess different facets of language, and perform as well a human evaluation study.

Performance metrics.We measure the task-completion score or recall @ 1 which states whether the target answer \(\mathcal{A}\) is the top answer of the VQA models, and the recall @ 5 (R@5), which assesses whether \(\mathcal{A}\) is in the 5 top answers. These scores measure the task-solving abilities of the agent, but they are also conditioned by the VQA model abilities.

Language Metrics.First, we used n-grams metrics, BLEU Papineni et al. (2002), METEOR Banerjee and Lavie (2005) and CIDErVedantam et al. (2015), to measure the similarity between the generated question and the reference questions in the evaluation set. While those scores can capture syntactic and semantic properties of language, they also fall short when dealing with open-form language, e.g. an identical answer may arise from two non-overlapping but syntactically correct questions. Thus, we also compute two metrics assessing the quality of the language independently of reference questions, the perplexity of the question given an external LM (ppl-e), and its perplexity given the task-related LM (ppl-t).

Diversity Metrics.We here estimate a self-BLEU (sBLEU) score Zhang et al. (2017) over 10 questions generated on the same image-answer pair. Although such score detects potential mode collapse, i.e., when the language utters identical sequences of words, it also values babbling, i.e., outputting random words. We thus also measure the probability mass of the ten most frequent words Choshen et al. (2020), and refer to it as peakiness (peak).

Human Evaluation.On the VQAv2 task, we also performed human evaluation by surveying 53 participants on the first 50 questions produced by some of the models at test time. The study (further detailed in Appendix C) is based on pairwise comparison of question samples produced by the concurrent algorithms according to four criteria. First, we evaluated the language quality of the question samples, by asking the participants to select the most syntactically and semantically correct question among the two samples of the questions pair. Secondly, we evaluated language grounding, i.e adequacy of the sample to the image-answer pair, by asking the participants to select the question most suitable given the two elements. Thirdly, we evaluated the language originality and diversity, by asking participants to select the question the most different from the dataset reference question. Finally, we evaluated the number of syntax errors by asking participants to tick the question if it is grammatically incorrect. Examples of questions asked during the study are included in the Appendix C.

### Sampling methods for text generation

When generating text from a trained language model, the quality and diversity of samples depend on the decoding algorithm Zhang et al. (2020). We consider three text generation methods. _greedy_ uses the \(\operatorname*{argmax}\) of the policy, while _sampling_ uses the multinomial distribution. Finally, we sampled ten text sequences from the policy, and selected the one with the lowest perplexity according to the external language model, and refer to it as _lm-ranking_. This process has been used recently in Text-to-Image Generation tasks Ramesh et al. (2021)



## 5 Results

### CLEVR results

Quantitative performance:In Table 1, vanilla RL from scratch fails to have a decent performance even with synthetic language. Besides, adding a KL regularisation term does kick-start the learning process. Yet, as soon as we apply the dynamic truncation, TrufLL matches the pretrained baselines performance when using the external LM, and even outperforms them with the task-specific LM. In this synthetic VQG setting, TrufLL seems to be a viable and promising procedure to learn a RL language agent without a supervised training phase. Pretrained baselines have high language scores when assessed with dataset-based metrics, e.g BLEU or task-perplexity. Yet, they also remain close to the original dataset distribution with a medium external perplexity. Noticeably, TrufLL with the task-specific LM follows the same pattern. On the other hand, TrufLL with the external LM reports poor dataset-based language scores, while maintaining a low external perplexity. Therefore, TrufLL seems to correctly capture the language distribution of the initial LM. As the performance score is high when using an external LM, it suggests that our approach can learn a policy on a language task without the need of a task-related dataset. Less positively, TrufLL diversity metrics suggest potential mode collapse, with a high peakiness and self-BLEU score.

Qualitative performance:We display qualitative samples in Figure 2 and Appendix D. On the one hand, the pretrained baselines generate either a question inconsistent with the visual context, or which fails to answer the expected answer. They inaccurately capture the pragmatics of the task. On the other hand, TrufLL generate adequate questions, resulting in the expected answer. Interestingly, they are often grounded with different objects of the image. It is remarkable that TrufLL with a generic LM still manages to capture the necessary subtleties of VQG, without any prior task knowledge. Despite a peaky distribution, TrufLL has moderate repetitions across images, and is mostly overconfident. As for the scratch+KL samples, they are either not grounded, or showcase degenerated language.

Truncation function in CLEVR:In Table 2, we evaluate the different truncation functions defined in Section 3. While all truncation methods report similar task performance, the dynamic truncation functions, i.e. \({\rm p_{th}}(\alpha)\), \({\rm top(p)}\) and \({\rm sample(k)}\), outperform the \({\rm top(k)}\) regarding language metrics. Interestingly, the \({\rm sample(k)}\) one, which generates a stochastic truncated

\begin{table}
\begin{tabular}{l|c c|c c c c c|c c} Method & Score & R0.5 & BLEU & Meteor & CIDEr & ppl4(↓) & pple-le (↓) & sBLEU (↓) & psk(↓) \\ \hline Pretrain & 0.30 & 0.71 & 0.19 & 0.38 & 0.83 & 3.11 & 31 & 0.44 & 0.96 \\ Pretrain + RL fine-tune & 0.44 & 0.36 & 0.17 & 0.34 & 0.70 & 4.0 & 35 & 0.46 & 0.95 \\ \hline \hline Scratch & 0.17 & 0.47 & 0.05 & 0.08 & 0.10 & \(10^{9}\) & \(10^{6}\) & **0.14** & **0.26** \\ Scratch + KL-task & 0.14 & 0.38 & 0.15 & 0.30 & 0.53 & 92 & \(10^{2}\) & 0.34 & 0.94 \\ Scratch + KL-ext & 0.17 & 0.44 & 0.14 & 0.27 & 0.43 & \(10^{4}\) & 28 & 0.37 & 0.95 \\ \hline TrufLL (Task-LM) & **0.56** & 0.90 & **0.17** & **0.32** & **0.66** & **3.4** & 23 & 0.95 & 1.00 \\ TrufLL (Ex-LM) & 0.48 & **0.93** & 0.08 & 0.18 & 0.34(\pm 0.10) & \(10^{3}\) & **3.0** & 0.95 & 1.00 \\ \end{tabular}
\end{table}
Table 1: **CLEVR metrics** on 5K test episodes with 50k train episodes on 20k Images. Scores are averaged over the three decoding procedures mentioned in Section 4.5 and over 5 seeds; standard deviations are displayed when greater than \(0.01\) for accuracy metrics. We here report the models with the highest task-success::, i.e. the _scratch+KL_ baselines with \(\lambda_{KL}=0.1\), and the truncation model with a probability threshold, \({\rm p_{th}}(\alpha=0.05)\). Best values are underlined, best values without task-data (from scratch) are in bold.

Figure 2: Samples on CLEVR and VQA: the checkbox indicates that the question generates the correct answer.



action space, while having a lower performance, yields to the most correct and diverse language, with higher language scores and a lower self-BLEU. A stochastic action space might be harder to explore efficiently for reaching good task-solving abilities, but might strengthen the agent language generation properties.

### VQAv2 task

In CLEVR, we observe that TrufLL seems a promising approach to learn a language policy without a supervised training phase, by solely interacting with another language system. We scale our approach to natural language with large vocabulary (15k tokens) through the VQAv2 dataset.

Quantitative performance:Table3 reports the VQAv2 results, for which TrufLL and the baselines present a similar trend than on CLEVR. First, the scratch baselines keep failing to learn a valuable policy, with performance scores and n-grams metrics close to zero. Although TrufLL does not outperform the performance of the pretrained baselines anymore, it still leads to similar performances, and satisfactory language scores. The similarity between TrufLL (Task-LM) and TrufLL (Ext-LM) results suggests that the truncation approach is viable when using a generic LM whose original vocabulary distribution differs from the task. Interestingly, TrufLL displays a self-BLEU score similar to the pretrained baselines. This suggests that the poor diversity behavior observed on CLEVR is likely attributable to the small vocabulary and synthetic language distribution.

Qualitative performance:In Figure2 and Appendix D, we display question samples for all models. TrufLL and the pretrained baselines successfully generate a question giving the expected answer ("Black"), while the RL from scratch baselines fail, and even showcase degenerated language. Pretrained baselines tend to output a question closer to the reference question whereas TrufLL outputs original questions which differs from the VQA distribution, yet consistent with the context.

Human Evaluation:Figure3 details the Human Evaluation results. Among the RL from scratch baselines, we selected scratch+KL-task as the only model producing sometimes meaningful questions. Yet, it fails to generate correct and grounded language; it is thus not a viable approach despite its diverse output. In line with the automatic metrics, the supervised baselines produce the best language, while being accurately grounded. Yet, they exhibit significantly less diversity with the reference language; this suggests in particular that pretrain+RL fails to go beyond the initial task-data distribution. Finally, unlike TrufLL (Task-LM) which suffers from syntactic errors, TrufLL (Ext-LM) produces language that qualitatively competes with pretrain models (53%), with a similar ratio of syntactic uncorrect samples. Although its questions are less grounded, they are diverse, which suggests that they follow a different distribution from the initial VQA dataset. It confirms that TrufLL (Ext-LM) could be an alternative approach as it has an excellent trade-off between language quality, diversity, and grounding.

Decoding procedure:In Table4, we evaluate the text sampling procedures described in Section4.5. While greedy decoding produces the best outcome for pretrained models, lm-ranking provides an excellent trade-off between task performance and language quality with RL-based methods. As PG solely optimizes the task success ratio, this may reduce overall language quality, the re-ranking thus retrieves the best syntactically sentences a posteriori.

### Discussion

Removing the truncation at evaluation with off-policy RLSo far, TrufLL directly learns the truncated policy over the truncated vocabulary \({\cal V}_{t}^{-}\) in an on-policy scheme. Hence, the algorithm requires the truncation, and a fortiori the language model, at test time. In this section, we investigate if we can directly learn a policy over the full vocabulary, and thus removing the truncation at test time. In such a setting, we adopt an off-policy training scheme, where the trajectories used to learn the behavior \(\pi_{\theta}\) at training time are sampled under a different policy, the truncated policy \(\pi_{\theta}^{-}\). Thus, we need to unbiased the PG by using an importance sampling term between the exploratory policy \(\pi_{\theta}^{-}\) and the behavior policy \(\pi_{\theta}\)Degris et al. [20].

\begin{table}
\begin{tabular}{l|c c c c|c} \hline \multicolumn{1}{c|}{Tune.} & Score & BLEU & CIDEr & ppl-\(\downarrow_{(L)}\) & sBLEU(\downarrow_{(L)}\) \\ \hline \multicolumn{7}{l}{**TriML (Task-LM)**} \\ \hline top(\(k\)) & 0.50 & 0.12 & 0.32 & 100 & 0.93 \\ \(\mathrm{p_{th}}(\alpha)\) & **0.54** & 0.17 & 0.65 & 24 & 0.95 \\ top(\(\mathrm{p}\)) & 0.51 & 0.17 & 0.69 & **12** & 0.96 \\ sample(\(\mathrm{k}\)) & 0.50 & **0.18** & **0.73** & 16 & **0.89** \\ \hline \multicolumn{7}{l}{**TrfLL (Est-LM)**} \\ \hline top(\(k\)) & **0.52** & 0.06 & 0.15 & 151 & 0.94 \\ \(\mathrm{p_{th}}(\alpha)\) & 0.48 & 0.08 & 0.34\(\pm 0.10\) & 3.0 & 0.95 \\ top(\(\mathrm{p}\)) & 0.45 & 0.10 & 0.40\(\pm 0.17\) & 3.3 & **0.92** \\ sample(\(\mathrm{k}\)) & 0.41 & **0.13** & **0.46\(\pm 0.16\)** & 2.7 & **0.92** \\ \hline \end{tabular}
\end{table}
Table 2: **CLEVR task: Truncation functions with parameters: \(\mathrm{top(k\!=\!10)}\), \(\mathrm{p_{th}}(\alpha\!=\!0.05)\)\(\mathrm{top(p\!=\!0.85)}\), \(\mathrm{sample(k\!=\!20)}\). Best values are underlined, best values for each TrufLL algorithms are in bold.** 

2012). Formally, the off-policy PPO loss is defined by:

\[L_{\mathrm{pop}}^{off}(\theta) = \mathbb{E}_{\pi_{\theta}^{-}}\big{[}\min(\bar{p}_{t}^{\theta}A_{ t},\text{clip}(1-\epsilon,\bar{p}_{t}^{\theta},1+\epsilon)A_{t})\big{]},\]

where \(\bar{p}_{t}^{\theta} =\frac{\pi_{\theta}(a_{i}|s_{t})}{\pi_{\theta d}(a_{i}|s_{t})} \frac{\pi_{\theta d}(a_{i}|s_{t})}{\pi_{\theta d}^{-}(a_{i}|s_{t})}\) is the new ratio.4

Footnote 4: Note that we did not simplify the expression to highlight the importance sampling ratio.

Table 5 displays the on-policy and off-policy results on both VQG tasks for TrufLL (task-LM), and is further detailed in Appendix B.3. We also monitor the probability mass of the policy attributed to the truncated action space (\(\rm sumVA\)). The policy only samples words within the truncated action space when \(\rm sumVA=1\), without needing the truncation. On CLEVR, the \(\rm TrufLL_{off}\) has lower - yet close - performance on language and task scores than TrufLL. As its \(\rm sumVA\) ratios are very close to 1, the agent has learned to generalize over the full vocabulary. However, the approach does not manage to sufficiently scale to VQAv2. It could be improved with regularisation techniques and the use of TruFLL within state-of-the-art off-policy RL algorithms. We leave such possibilities to future works.

## Appendix C

Figure 3: **VQAv2** results for Human Evaluation study detailed in Section 4.4. The three matrices on top are pairwise comparisons: each cell displays the proportion of questions chosen for the models in the row (bold) when compared to the concurrent model in the column. The table at the bottom displays the proportion of incorrect questions coming from each model among all incorrect samples. In all figures, bracket numbers indicates the model rank per criteria, from 1=”best” to 5="worst”.

\begin{table}
\begin{tabular}{c c|c|c c c} Method & Text-gen & Scene & BLEU & CIDEr & ppl+e \\ \hline \multirow{3}{*}{pretrain} & greedy & 0.40 & 0.32 & 1.01 & 51 \\  & sampling & 0.37 & 0.30 & 0.88 & 62 \\  & lm-ranking & 0.37 & 0.14 & 0.87 & 54 \\ \hline \multirow{3}{*}{pretrain + RL} & greedy & 0.42 & 0.32 & 1.05 & 55 \\  & sampling & 0.40 & 0.30 & 0.92 & 71 \\  & lm-ranking & 0.40 & 0.31 & 0.99 & 26 \\ \hline \hline \multirow{3}{*}{Trufl LL (Task-LM)} & greedy & **0.36** & 0.20 & 0.11 & 366 \\  & sampling & 0.35 & 0.20 & 0.11 & 337 \\  & lm-ranking & 0.34 & **0.21** & 0.11 & 95 \\ \hline \multirow{3}{*}{Trufl LL (Ext-LM)} & greedy & **0.36** & 0.18 & 0.04 & 25 \\  & sampling & 0.34 & 0.18 & 0.04 & 28 \\ \cline{1-1}  & lm-ranking & 0.33 & 0.19 & **0.15** & **20** \\ \end{tabular}
\end{table}
Table 4: **VQAv2: Ablation on the sampling methods. Overall best values are underlined, TrufLL best values are in bold.** 

Additional experiments.We sweep over truncation hyper-parameters in Table 6 of Appendix B. In Table 8, we observe that rewarding an agent with a BLEU score is sub-optimal in both language and task scores on CLEVR. In VQA, we apply temperature scheduling on the LM to perform fine-grained truncations in Table 9 of B.2. Finally, we explore TrufLL with a pre-training phase in Table 10.

Generalization of the approach.TrufallL learns conditional language models able to solve specific Natural Language Generation tasks given a context \(c\). For solving such tasks, it only requires the context, a reward function that scores the language generated by the RL agent with respect to the task, and eventually a few natural language demonstrations fed as input prompt to the generic language model used in the truncation algorithm. Hence, the method is transferable to a wide variety of NLG tasks, without requiring upfront large-scale labelled datasets. Additionally, the RL framework allows to optimize non-differentiable objectives, making TrufLL a natural choice to learn end-to-end task-oriented dialogs, such as De Vries et al. (2017); Das et al. (2017). Other interesting tasks for TrufLL include the ones typically found in Vision and Language Representation Learning Lu et al. (2020), such as Image Captioning, Grounding Referring Expressions (generation of a referring expression over a specific bounding box of an image), Caption-based Image Retrieval (generation of a caption that discriminates an image between a set of images). Reward functions for such tasks can be based on similarity scores between the generated language and the associated image or image region, which can be computed using pretrained language representations such as BERT Devlin et al. (2019) or multi-modal pretrained systems such as ViLBERT Lu et al. (2019). The context can be any kind of data structure (natural language, database, video, etc): if it is a linguistic input, TrufLL can be applied for instance to text summarization, paraphrase generation (with reward functions based on similarity scores between the context and the generated language) or text-based games Ammanabrolu and Riedl (2018).

## 6 Related work

RL and NLP Tasks.Following Singh et al. (2002); Lemon and Pietquin (2007), recent RL-based task-oriented dialogues De Vries et al. (2017); Das et al. (2017); Lewis et al. (2017); Narasimhan et al. (2015) have been developed, where the policy language model is generally pretrained with SL followed RL fine-tuning. Yang et al. (2018); Fan et al. (2018) focused on tackling VGG tasks with RL, respectively on CLEVR and on the VQG dataset. Yet, the former uses slot filling with template questions, while the later computes a mixed objective with a MLE loss using ground-truth sentences. Bahdanau et al. (2016); Rennie et al. (2017) use RL to train language models as an alternative to SL to prevent typical text degeneration issues, but within training algorithms relying on ground-truth examples from labelled datasets.

RL methods for Language Action Spaces.Several RL algorithms have been developed to tackle large discrete action spaces. Hence, Dulac-Arnold et al. (2015); Tennenholtz and Mannor (2019); Chandak et al. (2019) embed the actions into a continuous action space, and then use classic RL algorithms to learn a policy over this continuous space. Zahavy et al. (2018); Seurin et al. (2020) proposes Q-learning algorithms with an elimination signal to eliminate forbidden actions. Closer to our work, a few algorithms Ammanabrolu and Riedl (2018) use the structure of language to prune the action space of text-based games, but within value-based algorithms, which are less scalable to large vocabularies. Similarly to TrufLL, CALM Yao et al. (2020) combines a pretrained language model to prune the action space with a DeepQ network, aka DRNN He et al. (2016). Yet, its truncation language model remains fine-tuned on the RL dataset. Besides, CALM is only evaluated on a vocabulary of 697 tokens, and on 4-words action sequences.

Learning Language Models from scratch.Ziegler et al. (2019); Garg et al. (2021) finetune pretrained GPT-2 models with RL for language generation tasks without task-related data, only using reward signals. Yet, they still face optimization and computational challenges Parisotto et al. (2020).

## 7 Conclusion

We proposed TrufLL, an original approach to learn a natural language generation (NLG) task using RL, without the usual pre-training phase requiring supervised datasets. To our knowledge, this is the first RL-based algorithm dedicated to learning a word-based text-generation task, which does not rely on a pre-training phase while scaling to large vocabularies. Although it comes with its limitations, the truncated RL algorithm provided by TrufLL gets free from labelled data in task-oriented language models, presents interesting language generation properties, and provides a generic and transferable method to learn any NLG problem 



## Acknowledgements

This action benefited from the support of the Chair New Gen RetAll led by l'X - Ecole Polytechnique and the Fondation de l'Ecole Polytechnique, sponsored by CARREFOUR.

## References

* P. Ammanabrolu and M. O. Riedl (2018)Playing text-adventure games with graph-based deep reinforcement learning. In Proc. of the North American Chapter of the Association for Computational Linguistics (NAACL), Cited by: SS3.4. D. Bahdanau, W. Brakel, K. Xu, A. Goyal, R. Lowe, J. Pineau, A. Courville, and Y. Bengio (2016)An actor-critic algorithm for sequence prediction. In Proc. of the International Conference on Learning Representations (ICLR), Cited by: SS5.1. D. Bahdanau, H. de Vries, T. J. O'Donnell, S. Murty, P. Beaudoin, Y. Bengio, and A. Courville (2019)Closure: assessing systematic generalization of clevr models. Visually Grounded Interaction and Language (VGIL), pp. . External Links: Document Cited by: SS1, SS2.1.
* S. Banerjee and A. Lavie (2005)Meteor: an automatic metric for mt evaluation with improved correlation with human judgments. In Proc. of ACL Workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, Cited by: SS1, SS2.1.
* S. Bengio, O. Vinyals, N. Jaitly, and N. Shazeer (2015)Scheduled sampling for sequence prediction with recurrent neural networks. In Proc. of Neural Information Processing Systems (NIPS), Cited by: SS1, SS2.1.
* Y. Bisk, A. Holtzman, J. Thomason, J. Andreas, Y. Bengio, J. Chai, M. Lapata, A. Lazaridou, J. May, A. Nisnevich, N. Pinto, and J. Turian (2020)Experience grounds language. External Links: Link Cited by: SS1, SS2.1.
* T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sabry, A. Askell, et al. (2020)Language models are few-shot learners. In Proc. of Neural Information Processing Systems (NeurIPS), Cited by: SS1, SS2.1.
* Y. Chandak, G. Theocharous, J. Kostas, S. Jordan, and P. S. Thomas (2019)Learning action representations for reinforcement learning. In Proc. of International Conference on Machine Learning (ICML), Cited by: SS1, SS2.1.
* S. Chen, Y. Hou, Y. Cui, W. Che, T. Liu, and X. Yu (2020)Recall and learn: fine-tuning deep pretrained language models with less forgetting. In Proc. of Empirical Methods in Natural Language Processing (EMNLP), Cited by: SS1, SS2.1.
* L. Choshen, L. Fox, Z. Aizenbud, and O. Abend (2020)On the weaknesses of reinforcement learning for neural machine translation. In Proc. of the International Conference on Learning Representations (ICLR),

Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, Jose MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In _Proc. of Conference on Computer Vision and Pattern Recognition (CVPR)_.
* De Vries et al. (2017) Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. 2017. Guesswhat?! visual object discovery through multi-modal dialogue. In _Proc. of Conference on Computer Vision and Pattern Recognition (CVPR)_.
* Degris et al. (2012) Thomas Degris, Martha White, and Richard S Sutton. 2012. Off-policy actor-critic. _arXiv preprint arXiv:1205.4839_.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
* Dulac-Arnold et al. (2015) Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, Peter Sunehaeg, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. 2015. Deep reinforcement learning in large discrete action spaces. _arXiv preprint arXiv:1512.07679_.
* Espeholt et al. (2018) Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. 2018. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In _Proc. of International Conference on Machine Learning (ICML)_.
* Fan et al. (2018) Zhihao Fan, Zhongyu Wei, Siyuan Wang, Yang Liu, and Xuan-Jing Huang. 2018. A reinforcement learning framework for natural question generation using bi-discriminators. In _Proc. of the International Conference on Computational Linguistics (ICCL)_.
* Ferrucci (2012) David A Ferrucci. 2012. Introduction to "this is watson". _IBM Journal of Research and Development_, 56(3.4):1-1.
* Garg et al. (2021) Sonal Garg, Sumanth Prabhu, Hemant Misra, and G Srinivasaraghavan. 2021. Unsupervised contextual paraphrase generation using lexical control and reinforcement learning. _arXiv preprint arXiv:2103.12777_.
* Goyal et al. (2017) Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In _Proc. of Conference on Computer Vision and Pattern Recognition (CVPR)_.
* He et al. (2016) Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. 2016. Deep reinforcement learning with a natural language action space. In _Proc. of Association for Computational Linguistics (ACL)_.
* Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jurgen Schmidhuber. 1997. Long short-term memory. _Neural computation_, 9(8):1735-1780.
* Holtzman et al. (2019) Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In _Proceedings of the International Conference on Learning Representations (ICLR)_.
* Jang et al. (2016) Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. In _Proc. of the International Conference on Learning Representations (ICLR)_.
* Jaques et al. (2019) Natasha Jaques, Asma Ghandehariou, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. 2019. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. _arXiv preprint arXiv:1907.00456_.
* Jaques et al. (2017) Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, Jose Miguel Hernandez-Lobato, Richard E Turner, and Douglas Eck. 2017. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In _Proc. of International Conference on Machine Learning (ICML)_.
* Jaques et al. (2020) Natasha Jaques, Judy Hanwen Shen, Asma Ghandehariou, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Shane Gu, and Rosslind Picard. 2020. Human-centric dialog training via offline reinforcement learning. In _Proc. of Empirical Methods in Natural Language Processing (EMNLP)_.
* Johnson et al. (2017) Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In _Proc. of Conference on Computer Vision and Pattern Recognition (CVPR)_.
* Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In _Proceedings of the International Conference on Learning Representations (ICLR)_.
* Kottur et al. (2019) Satwik Kottur, Jose M. F. Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. 2019. CLEVR-dialog: A diagnostic dataset for multi-round reasoning in visual dialog. In _Proc. of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
* Lazaridou et al. (2016) Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. 2016. Multi-agent cooperation and the emergence of (natural) language. _CoRR_, abs/1612.07182.
* Lazaridou et al. (2020) Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. 2020. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In _Proc. of Association for Computational Linguistics (ACL)_.
* Lemon and Pietquin (2007) Oliver Lemon and Olivier Pietquin. 2007. Machine learning for spoken dialogue systems. In _Proc. of Conference of the International Speech Communication Association (ISCA)_.
* Lewis et al. (2017) Mike Lewis, Denis Yarats, Yann N Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning for negotiation dialogues. In _Proc. of Empirical Methods in Natural Language Processing (EMNLP)_ 

Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objective function for neural conversation models. In _Proc. of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
* Li et al. [2016] Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016. Deep reinforcement learning for dialogue generation. In _Proc. of Empirical Methods in Natural Language Processing (EMNLP)_.
* Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In _Proc. of European Conference on Computer Vision (ECCV)_.
* Lu et al. [2019] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_.
* Lu et al. [2020a] Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2020a. 12-in-1: Multi-task vision and language representation learning. In _Proc. of Conference on Computer Vision and Pattern Recognition (CVPR)_.
* Lu et al. [2020b] Yuchen Lu, Soumye Singhal, Florian Strub, Olivier Pietquin, and Aaron Courville. 2020b. Countering language drift with seeded iterated learning. In _Proc. of International Conference on Machine Learning (ICML)_.
* Luketina et al. [2019] Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Rocktaschel. 2019. A survey of reinforcement learning informed by natural language. In _Proc. of International Joint Conferences on Artificial Intelligence (IJCAI)_.
* Mostafazadeh et al. [2016] Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwerde. 2016. Generating natural questions about an image. In _Proc. of Association for Computational Linguistics (ACL)_.
* Narasimhan et al. [2015] Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. 2015. Language understanding for text-based games using deep reinforcement learning. In _Proc. of Empirical Methods in Natural Language Processing (EMNLP)_.
* Papineni et al. [2002] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In _Proc. of the Association for Computational Linguistics (ACL)_.
* Parisotto et al. [2020] Emilio Parisotto, Francis Song, Jack Bae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. 2020. Stabilizing transformers for reinforcement learning. In _Proc. of International Conference on Machine Learning (ICML)_.
* Peters et al. [2019] Matthew E Peters, Sebastian Ruder, and Noah A Smith. 2019. To tune or not to tune? adapting pretrained representations to diverse tasks. In _Proc. of Association for Computational Linguistics (ACL)_.
* Radford et al. [2019] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. _OpenAI blog_, 1(8):9.
* Ramesh et al. [2021] Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image generation. _arXiv preprint arXiv:2102.12092_.
* Ranzato et al. [2016] Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In _Proceedings of the International Conference on Learning Representations (ICLR)_.
* Ren et al. [2015] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In _Proc. of Neural Information Processing Systems (NIPS)_.
* Rennie et al. [2017] Steven J Rennie, Etienne Marcheret, Youssef Morueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In _Proc. of Conference on Computer Vision and Pattern Recognition (CVPR)_.
* Ruder [2019] Sebastian Ruder. 2019. _Neural transfer learning for natural language processing_. Ph.D. thesis, NUI Galway.
* Saleh et al. [2020] Abdelrhiman Saleh, Natasha Jaques, Asma Ghandehaioun, Judy Shen, and Rosalind Picard. 2020. Hierarchical reinforcement learning for open-domain dialog. In _Proc. of Conference on Artificial Intelligence (AAAI)_.
* Schulman et al. [2016] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2016. High-dimensional continuous control using generalized advantage estimation. In _Proceedings of the International Conference on Learning Representations (ICLR)_.
* Schulman et al. [2017] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. _arXiv preprint arXiv:1707.06347_.
* Seurin et al. [2020] Mathieu Seurin, Philippe Preux, and Olivier Pietquin. 2020. " i'm sorry dave, i'm afraid i can't do that" deep q-learning from forbidden actions. In _Proc. of the International Joint Conference on Neural Networks (IJCNN)_.
* Silver et al. [2021] David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. 2021. Reward is enough. _Artificial Intelligence_, page 103535.
* Singh et al. [2002] Satinder Singh, Diane Litman, Michael Kearns, and Marilyn Walker. 2002. Optimizing dialogue management with reinforcement learning: Experiments with the nifun system. _Journal of Artificial Intelligence Research_, 16:105-133.
* Stiennon et al. [2020] Nisan Stiennon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback. _arXiv preprint arXiv:2009.01325_ 

Florian Strub, Harm De Vries, Jeremie Mary, Bilal Piot, Aaron Courville, and Olivier Pietquin. 2017. End-to-end optimization of goal-driven and visually grounded dialogue systems. In _Proc. of International Joint Conferences on Artificial Intelligence (IJCAI)_.
* Sutton et al. (1998) Richard S Sutton, Andrew G Barto, et al. 1998. _Introduction to reinforcement learning_, volume 135. MIT press Cambridge.
* Sutton et al. (1999) Richard S Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, et al. 1999. Policy gradient methods for reinforcement learning with function approximation. In _Proc. of Neural Information Processing Systems (NIPS)_.
* Tennenholtz and Mannor (2019) Guy Tennenholtz and Shie Mannor. 2019. The natural language of actions. In _Proc. of International Conference on Machine Learning (ICML)_.
* Vedantam et al. (2015) Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In _Proc. of Conference on Computer Vision and Pattern Recognition (CVPR)_.
* Wang et al. (2020) Pei-Hsin Wang, Sheng-Iou Hsieh, Shih-Chieh Chang, Yu-Ting Chen, Jia-Yu Pan, Wei Wei, and Da-Chang Juan. 2020. Contextual temperature for language modeling. _arXiv preprint arXiv:2012.13575_.
* Williams (1992) Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. _Machine learning_, 8(3-4):229-256.
* Wu et al. (2019) Qingyang Wu, Yichi Zhang, Yu Li, and Zhou Yu. 2019. Alternating roles dialog model with large-scale pre-trained language models. In _Proc. of European Association for Computational Linguistics (EACL)_.
* Yang et al. (2018) Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. 2018. Visual curiosity: Learning to ask questions to learn visual recognition. In _Proc. of International Conference on Machine Learning (ICML)_.
* Yao et al. (2020) Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. 2020. Keep calm and explore: Language models for action generation in text-based games. In _Proc. of Empirical Methods in Natural Language Processing (EMNLP)_.
* Zahavy et al. (2018) Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J Mankowitz, and Shie Mannor. 2018. Learn what not to learn: Action elimination with deep reinforcement learning. In _Proc. of Neural Information Processing Systems (NeurIPS)_.
* Zhang et al. (2020) Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. 2020. Trading off diversity and quality in natural language generation. In _Proc. of European Association for Computational Linguistics (EACL)_.
* Zhang et al. (2018) Xu Zhang, Felix Xinnan Yu, Swebor Karaman, Wei Zhang, and Shih-Fu Chang. 2018. Heated-up softmax embedding. _arXiv preprint arXiv:1809.04157_.
* Zhang et al. (2017) Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. 2017. Adversarial feature matching for text generation. In _Proc. of International Conference on Machine Learning (ICML)_.
* Ziegler et al. (2019) Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. _arXiv preprint arXiv:1909.08593_.

 

Dataset and training details

### Evaluation Metrics

For the BLEUU and METEOR scores, we used the NLTK5 implementations with the smoothing function number 2 for the BLEU score. For the CIDEr score, we used the nlg-eval implementation6.

Footnote 5: [https://www.nltk.org/](https://www.nltk.org/)

Footnote 6: [https://github.com/Maluuba/nlg-eval](https://github.com/Maluuba/nlg-eval)

### Answer filtering

For each dataset, we remove \(yes\) and \(no\) question-answer pairs which frequency largely exceeds other answers, to avoid any bias in the question generation process, as usually done in the VQG litterature [14].

### Dataset split

For CLEVR (resp. VQAv2), the RL language agent is trained for 50k (resp. 100k) episodes over the first 20k images (resp. all the images) of the training dataset, and is then evaluated on the first 5k (resp. 20k) images of the validation set. Besides, we uniformly sample the answer in the set of reference answers for each image to reduce the bias in the distribution of answers. Finally, questions are limited to 20 (resp. 10) words.

### Language Agent Networks and Training

For CLEVR (resp. VQAv2), we used a single-layer LSTM with 64 (resp. 256) units for the policy network. At every time step, the LSTM input is then the concatenation of the word embedding of dimension 32 (resp. 128), the answer embedding of dimension 32 (resp. 128), and the image representation. For CLEVR, the image representation is extracted from a pretrained ResNet50 and projected into a tensor of size (32,7,7) before being flattened. For VQAv2, the image representation is the average of 200 bounding box features of dimension 1048, extracted from a faster R-CNN [9].

We optimize the full loss \(L=L_{PPO}+\alpha L_{VF}+\beta L_{E}\) with \(\alpha=0.5\), \(\beta=0.01\) and a PPO clipping ratio \(\epsilon=0.02\) (resp. 0.01) for CLEVR (resp. VQAv2). We use Adam optimizer [10] with a learning rate (lr) of \(10^{-3}\) for Trufl LL and the scratch baseline, \(10^{-5}\) (resp. \(10^{-6}\)) for RL algorithms with a pre-training phase on CLEVR (resp. VQAv2), and \(5*10^{-4}\) for models including a KL regularization term. We use a batch size (bs) of 128 for all models except the ones with KL regularization, for which we use a batch size of 64. Finally, for the RL from scratch baselines, we perform gradient clipping (gladclip) of \(1\) (resp. \(5\)) for CLEVR and VQAv2.

Such hyper-parameters were selected, after conducting an extensive hyper-parameter search. The following values were tested: \(\beta\in\{0.01,\,0.02,\,0.05,\,0.1\}\), \(\epsilon\in\{0.01,\,0.02,\,0.05,\,0.1,\,0.5,\,0.9\}\), lr \(\in\{10^{-6},10^{-5},10^{-4},5*10^{-4},10^{-3},5*10^{-3},10^{-2},5*10^{-2}\}\), gradclip \(\in\{\text{None},1,5,10,100\}\), bs \(\in\{\)32,64,128\(\}\).

Additionally, we also tested for VQAv2 policy networks with 64, 256 and 1024 units, with respectively 32, 128 and 512 word embedding dimensions. We kept the network size giving the best performances, i.e. policy network of 256 units and 128 word embedding dimension.

### Reward formula for VQAv2

In this section, we detail the reward function used for the VQAv2 task. \(r(w_{t},w_{<t},c)=\mathds{1}_{\mathrm{rk}(\mathcal{A})\leq 10,t=T-1}e^{- \mathrm{rk}(\mathcal{A})/2}\), with \(\mathrm{rk}(\mathcal{A})\) the rank of the ground-truth answer given by the VQA model, when predicting the actual answer from the terminal state \((c,w_{<T})\). Formally, it is defined as:

\[\mathrm{rk}(\mathcal{A})=\mathrm{rank}(\mathrm{VQA}(c,w_{<T})[\mathcal{A}]),\]

with \(\mathrm{VQA}(c,w_{<T})\) the probability distribution given by the VQA model over the set of answers, and \(\mathrm{rank}\) the function which ranks the probability of answer \(\mathcal{A}\) within \(\mathrm{VQA}(c,w_{<T})\) probability distribution.

## Appendix B Additional experiments

### Clevr

Table 6 displays the complete ablation on the truncation functions with parameters sweep. The 'sizeVA' variable indicates the average size of the truncated action space for each truncation function. Table 7 displays the



ablation over the three decoding procedures defined in Section 4.5. Such an ablation presents a similar pattern than VQAv2 results described in section 5.2.

Finally, Table 8 reports CLEVR metrics when using the BLEU score as the reward. While on such a task TrufLL still exhibits promising language scores, the n-grams metrics remain lower than the pretrained baselines. This illustrates that using a language similarity score as a reward signal is much less interesting than a reward based on a task completion score.

### VQav2

Temperature scheduling:On the CLEVR task, we observed that dynamic truncations outperform static ones such as \({\rm top}({\rm k})\): indeed, they better take into account the inherent variability of the language structure at the sentence-level. When scaling up to the 15k words of the VQAv2 task, we also dynamically decrease the truncation size through training, by applying a decreasing temperature schedule on the language model. While temperature scaling [3] is usually used at test time to control the smoothness of the language model distribution, temperature schedules during training of language models have been used in several settings [7; 6; 9; 10]. Formally, \(f_{LM}(w_{i}|w_{<t})\) distribution is computed as \({\rm softmax}(x_{i})\!=\!e^{-s_{i}/\tau}\!/\!\sum_{j}\!e^{-x_{j}/\tau}\), with \(x_{j}\) the LM logits and \(\tau\) the temperature, which decreases from \(\tau_{max}\) to \(\tau_{min}\) by a factor \(T_{F}\) every \(T_{u}\) training step. In Table 9, both TrufLL (Task-LM) and TrufLL

\begin{table}
\begin{tabular}{l|c|c c c|c c} trunc. & Score & BLEU & CIDEr & ppl-e(\(\downarrow\)) & sBLEU(\(\downarrow\)) & Size VA \\ \hline \multicolumn{2}{l|}{**TrufLL (Task-LM)**} & & & & & \\ \hline \({\rm top}({\rm k}\!=\!10)\) & 0.50 & 0.12 & 0.32 & \(10^{2}\) & 0.93 & 10 \\ \({\rm top}({\rm k}\!=\!20)\) & 0.45 & 0.10 & 0.24 & \(10^{3}\) & 0.87 & 20 \\ \({\rm p}_{\rm th}(\alpha\!=\!0.05)\) & **0.55** & **0.18** & 0.63 & 25 & 0.96 & 4.4 \\ \({\rm p}_{\rm th}(\alpha\!=\!0.1)\) & 0.47 & 0.18 & **0.87** & 6.7 & 0.98 & 2.4 \\ \({\rm ptn}(\alpha\!=\!1/{\rm V})\) & 0.50 & 0.16 & 0.49 & 41 & 0.97 & 6.6 \\ \({\rm top}({\rm p}\!=\!0.85)\) & 0.52 & 0.17 & 0.69 & 10.4 & 0.96 & 4.6 \\ \({\rm top}({\rm p}\!=\!0.9)\) & 0.51 & 0.17 & 0.69 & 11.5 & 0.96 & 5.1 \\ sample(\({\rm k}\!=\!20)\) & 0.50 & 0.18 & 0.73 & 18.9 & **0.86** & 5.4 \\ sample(\({\rm k}\!=\!30\)) & 0.50 & 0.18 & 0.73 & 16.1 & 0.89 & 6.1 \\ \hline \multicolumn{2}{l|}{**TriML (Ext-LM)**} & & & & & \\ \hline \({\rm top}({\rm k}\!=\!10)\) & 0.52 & 0.06 & 0.15 & \(10^{2}\) & 0.94 & 10 \\ \({\rm top}({\rm k}\!=\!20)\) & 0.48 & 0.05 & 0.12 & \(10^{2}\) & 0.89 & 20 \\ \({\rm p}_{\rm th}(\alpha\!=\!0.05)\) & 0.48 & 0.08 & 0.34 & 3.03 & 0.95 & 3.3 \\ \({\rm pn}(\alpha\!=\!0.1)\) & 0.45 & 0.17 & 0.74 & **2.2** & 0.99 & 2.1 \\ \({\rm p}_{\rm th}(\alpha\!=\!1/{\rm V})\) & 0.44 & 0.11 & 0.37 & 3.7 & 0.96 & 5.7 \\ \({\rm top}({\rm p}\!=\!0.85)\) & 0.45 & 0.10 & 0.39 & 3.2 & 0.92 & 4.1 \\ top(p\({\rm p}\!=\!0.9)\) & 0.48 & 0.15 & 0.57 & 2.8 & 0.97 & 4.3 \\ sample(\({\rm k}\!=\!20)\) & 0.45 & 0.14 & 0.50 & 2.4 & 0.92 & 4.1 \\ sample(\({\rm k}\!=\!30)\) & 0.43 & 0.13 & 0.46 & 2.7 & 0.92 & 4.6 \\ \hline \end{tabular}
\end{table}
Table 6: **CLEVR task: Ablation on the truncation functions with parameters sweep. Best values are in bold.**

\begin{table}
\begin{tabular}{c c|c|c c c} method & text-gen & score & BLEU & CIDEr & ppl-e \\ \hline \multirow{3}{*}{pretrain} & greedy & 0.32 & 0.22 & 1.01 & 14 \\  & sampling & 0.29 & 0.17 & 0.76 & 58 \\  & lm-ranking & 0.28 & 0.18 & 0.73 & 20 \\ \hline \multirow{3}{*}{pretrain + RL} & greedy & 0.53 & 0.18 & 0.73 & 24 \\  & sampling & 0.40 & 0.16 & 0.68 & 39 \\  & lm-ranking & 0.40 & 0.17 & 0.68 & 5 \\ \hline \multirow{3}{*}{Task-LM} & greedy & **0.57** & 0.17 & 0.65 & 39 \\  & sampling & 0.55 & **0.17** & **0.66** & 24 \\  & lm-ranking & 0.51 & 0.16 & 0.65 & 9 \\ \hline \multirow{3}{*}{Ext-LM} & greedy & 0.48 & 0.09 & 0.34(\(\pm 0.11\)) & 3.0 \\  & sampling & 0.48 & 0.10 & 0.35(\(\pm 0.11\)) & 3.1 \\ \cline{1-1}  & lm-ranking & 0.48 & 0.06 & 0.34(\(\pm 0.11\)) & **2.9** \\ \hline \hline \end{tabular}
\end{table}
Table 7: **CLEVR task: Ablation on sampling methods. Best overall values are underlined, while best values for TrufLL are in bold.** 

(Ext-LM) benefit slightly from truncation with a temperature schedule compared to a vanilla truncation. The former displays the best performance/language scores trade-off for the schedule "\(\tau\): 3 > 1. & \(T_{u}\)=5,000", while the latter has the best metrics trade-off for "\(\tau\): 1.5 > 1. & \(T_{u}\)=5,000".

Finally, Figure 4 displays the evolution of the training return for TrufLL and the baselines. As expected, the pretrain+RL fine-tune baseline return does not evolve much, confirming that the policy distribution almost does not shift through the fine-tuning phase. The training curves of TrufLL present a steady increase in the return until reaching convergence, confirming that our approach, by guiding the exploration of the action space, provides a sufficient learning signal. On the other hand, the scratch+KL baselines stay stuck to a low training return. This suggests that the KL regularization term, while encouraging the policy distribution to resemble the language model distribution, fails to capture the task pragmatics, which requires generating a language that is visually grounded.

### Additional discussion

TruffLL with a pre-training phase.Although TrufLL aims at providing a robust method to learn a language model (almost) from scratch, we investigate whether such algorithm can be complementary to RL algorithms with a pre-training phase. Therefore, when using the task-related dataset, we evaluate TrufLL from a pretrained policy, and we refer to it as \(\mathrm{TriufLL}_{\mathrm{pretrain}}\).

In table 10, while on CLEVR, \(\mathrm{TriufLL}_{\mathrm{pretrain}}\) marginally improves the results of the pretrain+RL fine-tune baseline, the combination of TrufLL with a pre-training phase leads to performance degradation on VQAv2. This suggests that on a large vocabulary task, the language distribution learned by the SL pretrained policy is significantly different from the one learned with TrufLL.

On-policy TrufLL versus off-policy TrufLL.To ease off-policy learning, we propose to add a KL-regularization term in the RL loss (Jaques et al., 2017; Jaques et al., 2019; Wu et al., 2019), and refer to it as \(\mathrm{TriufLL}_{\mathrm{off},\mathrm{KL}}\). Intuitively, it encourages the policy to stay close to the language model's distribution, with a distribution support attributing negligible probabilities to words outside the truncated action space.

\begin{table}
\begin{tabular}{l|c c|c c c c c|c c} Method & Score & R@5 & BLEU & Meteor & CIDEr & ppl+ (↓) & ppl+ (↓) & sBLEEU (↓) & peak.(↓) \\ \hline \hline pretrain & 0.30 & 0.71 & 0.19 & 0.38 & 0.83 & 3.1 & 31 & 0.44 & 0.96 \\ pretrain + RL fine-tune & 0.34 & 0.80 & 0.20 & 0.38 & 0.83 & 3.8 & 12 & 0.56 & 0.96 \\ \hline scratch & 0.03 & 0.19 & 0.06 & 0.09 & 0.09 & \(10^{8}\) & \(10^{6}\) & **0.13** & **0.14** \\ scratch + KL-task & 0.09 & 0.33(\(\pm 0.15\)) & 0.15 & 0.31 & 0.58(\(\pm 0.23\)) & 3.8 & 63 & 0.34 & 0.95 \\ scratch + KL-ext & 0.06 & 0.30(\(\pm 0.23\)) & 0.13 & 0.25 & 0.42 & \(10^{3}\) & 3.6 & 0.37 & 0.96 \\ \hline scratch + Truncation-task & **0.17** & **0.51** & **0.18** & **0.37** & **0.80** & **2.6** & 17 & 0.63 & 1.0 \\ scratch + Truncation-ext & 0.07 & 0.36 & 0.16 & 0.29 & 0.49 & \(10^{2}\) & **2.3** & 0.60 & 1.0 \\ \hline \end{tabular}
\end{table}
Table 8: **CLEVR, BLEU reward. Scores are averaged over the three decoding procedures detailed in Section 4.5 and over 5 seeds, standard deviation are displayed whenever greater than \(0.01\) for accuracy metrics. We here report the models with the highest task-success, i.e. the _scratch with KL regularization_ baseline with \(\lambda_{KL}=0.1\), and the truncation model with a probability threshold, \(p_{\rm th}(\alpha\!=\!0.05)\). Baseline and Metrics are respectively detailed in Section 4.4 and 4.3. Best overall values are underlined, while best values for models without task-data (i.e RL from scratch algorithms) are in bold.** 

Table 11 displays the full results of on-policy versus off-policy scores for TrufLL (Task-LM) and TrufLL (Ext-LM) on the two tasks. The full results emphasize the challenges of the approach for the large vocabulary of VQAv2. Indeed, on the off-policy setting for such a task, the exploding values for e-ppl suggest that the optimized language agent samples incoherent words taken outside the truncated action space, as corroborated by the low values of the \({\rm sumVA}\) ratio.

Interestingly, while on CLEVR, \({\rm TrufLL_{off,KL}}\) trades off task performance for language quality when compared to \({\rm TrufLL_{off}}\), on VQAv2, it mainly provides a better learning signal for the complete (large) vocabulary. In such a setting, it hence improves the global scores of the off-policy version of TrufLL, and enables a much better generalization at test time of the global policy over the full vocabulary. Yet, keeping truncation at test time remains crucial with large vocabulary. Note that for VQAv2, the poor performances of \({\rm TrufLL_{off,KL}}\) on the external LM is mainly due to numerical instability challenges when using GPT-2 as the target policy of the KL regularization term.

Additionally, on-policy versus off-policy scores split per sampling procedure are displayed in table 12: unsurprisingly, greedy decoding for \({\rm TrufLL_{off}}\) outperforms the two sampling-based methods, that are more penalized by the imperfect generalization of the optimized policy over the full vocabulary.

Figure 4: **VQAv2: Training curves. Reward is a rolling average over 5000 timesteps.**





\begin{table}
\begin{tabular}{l|c|c c c|c c} Algo & Score & BLEU & CIDEr ppl-e & sBLEU & sumVA \\ \hline \multicolumn{6}{l}{**CLEVR**} \\ \hline \hline
**TrufLL (Task-LM)** & & & & & \\ \hline TrufLL & **0.56** & **0.17** & 0.06 & \(10^{3}\) & 0.78 & N.A \\ TrufLL\({}_{\rm off}\) & 0.50 & 0.14 & 0.43 & \(10^{4}\) & 0.88 & 0.96 \\ TrufLL\({}_{\rm off,KL}\) & 0.39 & 0.17 & **0.71** & 69 & **0.48** & 0.95 \\ \hline \multicolumn{6}{l}{**Truff LL (Ext-LM)**} \\ \hline TrufLL & 0.48 & 0.08 & 0.34 & **3.03** & 0.95 & N.A \\ TrufLL\({}_{\rm off}\) & 0.41 & 0.10 & 0.35 & \(10^{5}\) & 0.88 & 0.95 \\ TrufLL\({}_{\rm off,KL}\) & 0.35 & 0.15 & 0.60 & 20 & 0.55 & 0.96 \\ \hline \multicolumn{6}{l}{**VQAv2**} \\ \hline \hline
**TrufallL (Task-LM)** & & & & & \\ \hline TrufLL & **0.35** & 0.21 & 0.11 & \(10^{4}\) & 0.36 & N.A \\ TrufLL\({}_{\rm off}\) & 0.07 & 0.03 & 0.01 & \(10^{4}\) & 0.05 & 0.08 \\ TrufLL\({}_{\rm off,KL}\) & 0.12 & **0.24** & **0.25** & \(10^{3}\) & 0.26 & 0.71 \\ \hline \multicolumn{6}{l}{**TriuffLL (Ext-LM)**} \\ \hline TrufLL & 0.34 & 0.18 & 0.04 & **24** & 0.83 & N.A \\ TrufLL\({}_{\rm off}\) & 0.09 & 0.04 & 0.01 & \(10^{4}\) & **0.05** & 0.07 \\ TrufLL\({}_{\rm off,KL}\) & 0.0 & 0.15 & 0.02 & \(10^{3}\) & 0.19 & 0.47 \\ \hline \end{tabular}
\end{table}
Table 11: On-policy vs. off-policy scores for different variants of TrufLL: when training with an off-policy loss, we remove the truncation at test time. \(\mathrm{TrufLL}_{\rm off,KL}\) is evaluated with \(\lambda_{KL}=0.05\). Best values are in bold.





\begin{table}
\begin{tabular}{c c|c|c c c} method & text-gen & score & BLEU & CIDEr & e-ppl \\ \hline \multicolumn{5}{l}{**CLEVR**} \\ \hline
**TrufLL (Task-LM)** & & & & \\ \hline  & greedy & **0.57** & 0.17 & 0.65 & 39 \\ \multirow{2}{*}{\(\mathrm{TrufLL}_{\mathrm{off}}\)} & sampling & 0.55 & 0.17 & 0.66 & 24 \\  & lm-ranking & 0.51 & 0.16 & 0.65 & 8.8 \\ \hline \hline  & greedy & 0.52 & 0.17 & 0.58 & 71 \\ \multirow{2}{*}{\(\mathrm{TrufLL}_{\mathrm{off}}\)} & sampling & 0.49 & 0.16 & 0.59 & \(10^{5}\) \\  & lm-ranking & 0.48 & 0.17 & 0.58 & 19 \\ \hline  & greedy & 0.56 & **0.18** & **0.78** & 24 \\ \multirow{2}{*}{\(\mathrm{TrufLL}_{\mathrm{off,KL}}\)} & sampling & 0.31 & 0.16 & 0.62 & \(10^{2}\) \\  & lm-ranking & 0.31 & 0.18 & 0.74 & 5.8 \\ \hline \multicolumn{5}{l}{**TrufLL (Ext-LM)**} \\ \hline  & greedy & 0.48 & 0.09 & 0.34 & 3.1 \\ \multirow{2}{*}{\(\mathrm{TrufLL}_{\mathrm{allr}}\)} & sampling & 0.48 & 0.10 & 0.35 & 3.1 \\  & lm-ranking & 0.48 & 0.06 & 0.34 & 2.9 \\ \hline \hline  & greedy & 0.42 & 0.10 & 0.38 & 4.4 \\ \multirow{2}{*}{\(\mathrm{TrufLL}_{\mathrm{off}}\)} & sampling & 0.40 & 0.10 & 0.35 & \(10^{6}\) \\  & lm-ranking & 0.40 & 0.10 & 0.34 & 15 \\ \hline  & greedy & 0.48 & 0.16 & 0.70 & 2.1 \\ \multirow{2}{*}{\(\mathrm{TrufLL}_{\mathrm{off},\mathrm{KL}}\)} & sampling & 0.27 & 0.13 & 0.48 & 55 \\  & lm-ranking & 0.30 & 0.16 & 0.61 & **2.0** \\ \hline \multicolumn{5}{l}{**VQAv2**} \\ \hline
**TrufLL (Task-LM)** & & & & & \\ \hline  & greedy & 0.36 & 0.20 & 0.11 & 366 \\ \multirow{2}{*}{\(\mathrm{TrufLL}_{\mathrm{allr}}\)} & sampling & 0.35 & 0.20 & 0.11 & 337 \\  & lm-ranking & 0.34 & 0.21 & 0.11 & 95 \\ \hline \hline  & greedy & 0.09 & 0.04 & 0.02 & \(10^{3}\) \\ \multirow{2}{*}{\(\mathrm{TrufL}_{\mathrm{off}}\)} & sampling & 0.05 & 0.03 & 0.01 & \(10^{6}\) \\  & lm-ranking & 0.06 & 0.03 & 0.01 & \(10^{4}\) \\ \hline  & greedy & 0.16 & 0.29 & 0.46 & 38 \\ \multirow{2}{*}{\(\mathrm{TrufLL}_{\mathrm{eff},\mathrm{KL}}\)} & sampling & 0.08 & 0.19 & 0.09 & \(10^{4}\) \\  & lm-ranking & 0.12 & 0.24 & 0.22 & \(10^{2}\) \\ \hline \multicolumn{5}{l}{**TrufLL (Ext-LM)**} \\ \hline  & greedy & **0.48** & 0.09 & 0.34 & 3.1 \\ \multirow{2}{*}{\(\mathrm{TrufLL}_{\mathrm{allr}}\)} & sampling & 0.48 & 0.10 & **0.35** & 3.1 \\  & lm-ranking & 0.48 & 0.06 & 0.34 & **2.9** \\ \hline \hline  & greedy & 0.11 & 0.05 & 0.01 & \(10^{2}\) \\ \multirow{2}{*}{\(\mathrm{TrufLL}_{\mathrm{off}}\)} & sampling & 0.07 & 0.03 & 0.01 & \(10^{5}\) \\  & lm-ranking & 0.08 & 0.04 & 0.01 & \(10^{4}\) \\ \hline  & greedy & 0.00 & **0.18** & 0.05 & 27 \\ \multirow{2}{*}{\(\mathrm{TrufLL}_{\mathrm{off},\mathrm{KL}}\)} & sampling & 0.00 & 0.13 & 0.01 & \(10^{3}\) \\  & lm-ranking & 0.00 & 0.16 & 0.02 & \(10^{2}\) \\ \end{tabular}
\end{table}
Table 12: On-policy vs. off-policy scores per decoding procedure: when training with an off-policy loss, we remove the truncation at test time. \(\mathrm{TrufLL}_{\mathrm{off},\mathrm{KL}}\) is evaluated with \(\lambda_{KL}=0.05\). Best values are in bold.



Human Evaluation details

For the Human Evaluation study, we designed one form per participant, with three sections evaluating respectively the language quality, language grounding and diversity criteria. Given the five evaluated models, there are ten different model pairs: each section of the form contains 10 pairwise comparison covering all the possible model pairs for the criteria. Each pairwise comparison is sampled uniformly over the 50 first question samples generated by the algorithms at test time. The evaluation of syntax errors was made within the diversity section: for each questions pair, we asked participants to tick the questions if they are grammatically incorrect. Figure 5 displays one pairwise comparison example for the three sections, and a full form example is available at the following url: [https://forms.gle/kkL38x3lwF7A9YKx5](https://forms.gle/kkL38x3lwF7A9YKx5).

Figure 5: Examples of pairwise comparison for each evaluated criteria.

 



## Appendix D Additional VQG Samples

Figure 6 and Figure 7 display the 10 first dialog samples produced at test time on CLEVR, while figures 8, 9, and 10 display the 15 first dialog samples produced at test time on VQAv2.



Figure 6: Samples on CLEVR.

 

Figure 7: Samples on Clevr.

 



## Appendix B

Figure 8: Samples on VQA.



Figure 9: Samples on VQA.

 

Figure 10: Samples on VQA.