SlowGuess/NAACL_2022_OCR huggingface dataset:
Method	Score	R@5	BLEU	Meteor	CIDEr	ppl-t (↓)	ppl-e (↓)	sBLEU (↓)	peak.(↓)
Pretrain	0.30	0.71	0.19	0.38	0.83	3.1	31	0.44	0.96
Pretrain + RL fine-tune	0.44	0.86	0.17	0.34	0.70	4.0	35	0.46	0.95
Scratch	0.17	0.47	0.05	0.08	0.10	10^9	10^6	0.14	0.26
Scratch + KL-task	0.14	0.38	0.15	0.30	0.53	92	10^2	0.34	0.94
Scratch + KL-ext	0.17	0.44	0.14	0.27	0.43	10^4	28	0.37	0.95
TruffLL (Task-LM)	0.56	0.90	0.17	0.32	0.66	3.4	23	0.95	1.00
TruffLL (Ext-LM)	0.48	0.93	0.08	0.18	0.34(±0.10)	10^3	3.0	0.95	1.00
Figure 2: Samples on CLEVR and VQA: the checkbox indicates that the question generates the correct answer.
Table 1: CLEVR metrics on 5k test episodes with 50k train episodes on 20k Images. Scores are averaged over the three decoding procedures mentioned in Section 4.5 and over 5 seeds; standard deviations are displayed when greater than 0.01 for accuracy metrics. We here report the models with the highest task-success:, i.e. the scratch+KL baselines with  $\lambda_{KL} = 0.1$ , and the truncation model with a probability threshold,  $\mathrm{p_{th}}(\alpha = 0.05)$ . Best values are underlined, best values without task-data (from scratch) are in bold.
Human	There is a blue thing that is the same shape as the big cyan metallic object ; what is its size? A:Small
pretrain	There is a red metallic object that is the same size as the yellow rubber block ; what is its size?
pretrain + RL	What size is the thing that is the same color as the matte cube ? ✔
scratch	size sphere small blue or a yellow green large else in cylinders cubes color and how matte objects cube
scratch+KL-task	How big is the shiny cylinder ?
scratch+KL-ext	How many other objects in the are of same color as that shiny object ?
TruffLL (Task-LM)	How big is the thing that is to the right of the big matte thing ? ✔
TruffLL (Ext-LM)	What is the size of the thing that is right of the big cyan thing and is the same shape? ✔
Human	What color is the cat A:Black
pretrain	What color is the cat's collar?☑
pretrain + RL	What color is the cat?☑
scratch	AmazingAmazingAmazingAmazingAmazingAmazingAmazing
scratch+KL-task	What color is their hat of the fingers of this?
scratch+KL-ext	The first time is a bit of the way
TruLL (Task-LM)	What color is her outfit?☑
TruLL (Ext-LM)	What color can these cats look like in real life?☑
5.1 CLEVR results
Quantitative performance: In Table 1, vanilla RL from scratch fails to have a decent performance even with synthetic language. Besides, adding a KL regularisation term does kick-start the learning process. Yet, as soon as we apply the dynamic truncation, TrufLL matches the pretrained baselines performance when using the external LM, and even outperforms them with the task-specific LM. In this synthetic VQG setting, TrufLL seems to be a viable and promising procedure to learn a RL language agent without a supervised training phase. Pretrained baselines have high language scores when assessed with dataset-based metrics, e.g BLEU or task-perplexity. Yet, they also remain close to the original dataset distribution with a medium external perplexity. Noticeably, TrufLL with the task-specific LM follows the same pattern. On the other hand, TrufLL with the external LM reports poor dataset-based language scores, while maintaining a low external perplexity. Therefore, TrufLL seems to correctly capture the language distribution of the initial LM. As the performance score is high when using an external LM, it suggests that our approach can learn a policy on a language task with
out the need of a task-related dataset. Less positively, TrufLL diversity metrics suggest potential mode collapse, with a high peakiness and self-BLEU score.
Qualitative performance: We display qualitative samples in Figure 2 and Appendix D. On the one hand, the pretrained baselines generate either a question inconsistent with the visual context, or which fails to answer the expected answer. They inaccurately capture the pragmatics of the task. On the other hand, TrufLL generate adequate questions, resulting in the expected answer. Interestingly, they are often grounded with different objects of the image. It is remarkable that TrufLL with a generic LM still manages to capture the necessary subtleties of VQG, without any prior task knowledge. Despite a peaky distribution, TrufLL has moderate repetitions across images, and is mostly overconfident. As for the scratch+KL samples, they are either not grounded, or showcase degenerated language.
Truncation function in CLEVR: In Table 2, we evaluate the different truncation functions defined in Section 3. While all truncation methods report similar task performance, the dynamic truncation functions, i.e.  $\mathrm{p_{th}(\alpha)}$ , top(p) and sample(k), outperform the top(k) regarding language metrics. Interestingly, the sample(k) one, which generates a stochastic truncated
Trunc.	Score	BLEU	CIDEr	ppl-e(↓)	sBLEU(↓)
TruffL (Task-LM)
top(k)	0.50	0.12	0.32	100	0.93
\( p_{th}(\alpha) \)	0.54	0.17	0.65	24	0.95
top(p)	0.51	0.17	0.69	12	0.96
sample(k)	0.50	0.18	0.73	16	0.89
TruffL (Ext-LM)
top(k)	0.52	0.06	0.15	151	0.94
\( p_{th}(\alpha) \)	0.48	0.08	0.34(±0.10)	3.0	0.95
top(p)	0.45	0.10	0.40(±0.17)	3.3	0.92
sample(k)	0.41	0.13	0.46(±0.16)	2.7	0.92
Table 2: CLEVR task: Truncation functions with parameters: top(k=10),  $p_{th}(\alpha = 0.05)$  top(p=0.85), sample(k=20). Best values are underlined, best values for each TrufLL algorithms are in bold.
action space, while having a lower performance, yields to the most correct and diverse language, with higher language scores and a lower self-BLEU. A stochastic action space might be harder to explore efficiently for reaching good task-solving abilities, but might strengthen the agent language generation properties.
5.2 VQAv2 task
In CLEVR, we observe that TrufLL seems a promising approach to learn a language policy without a supervised training phase, by solely interacting with another language system. We scale our approach to natural language with large vocabulary (15k tokens) through the VQAv2 dataset.
Quantitative performance: Table 3 reports the VQAv2 results, for which TrufLL and the baselines present a similar trend than on CLEVR. First, the scratch baselines keep failing to learn a valuable policy, with performance scores and n-grams metrics close to zero. Although TrufLL does not outperform the performance of the pretrained baselines anymore, it still leads to similar performances, and satisfactory language scores. The similarity between TrufLL (Task-LM) and TrufLL (Ext-LM) results suggests that the truncation approach is viable when using a generic LM whose original vocabulary distribution differs from the task. Interestingly, TrufLL displays a self-BLEU score similar to the pretrained baselines. This suggests that the poor diversity behavior observed on CLEVR is likely attributable to the small vocabulary and synthetic language distribution.
Qualitative performance: In Figure 2 and Appendix D, we display question samples for all models. TrufLL and the pretrained baselines successfully generate a question giving the expected answer ("Black"), while the RL from scratch baselines fail, and even showcase degenerated language. Pretrained baselines tend to output a question closer to the reference question whereas TrufLL outputs original
questions which differs from the VQA distribution, yet consistent with the context.
Human Evaluation: Figure 3 details the Human Evaluation results. Among the RL from scratch baselines, we selected scratch+KL-task as the only model producing sometimes meaningful questions. Yet, it fails to generate correct and grounded language; it is thus not a viable approach despite its diverse output. In line with the automatic metrics, the supervised baselines produce the best language, while being accurately grounded. Yet, they exhibit significantly less diversity with the reference language; this suggests in particular that pretrain+RL fails to go beyond the initial task-data distribution. Finally, unlike TrufLL (Task-LM) which suffers from syntactic errors, TrufLL (Ext-LM) produces language that qualitatively competes with pretrain models  $(53\%)$ , with a similar ratio of syntactic uncorrect samples. Although its questions are less grounded, they are diverse, which suggests that they follow a different distribution from the initial VQA dataset. It confirms that TrufLL (Ext-LM) could be an alternative approach as it has an excellent trade-off between language quality, diversity, and grounding.
Decoding procedure: In Table 4, we evaluate the text sampling procedures described in Section 4.5. While greedy decoding produces the best outcome for pretrained models, lm-ranking provides an excellent trade-off between task performance and language quality with RL-based methods. As PG solely optimizes the task success ratio, this may reduce overall language quality, the re-ranking thus retrieves the best syntactically sentences a posteriori.
Removing the truncation at evaluation with off-policy RL. So far, TrufLL directly learns the truncated policy over the truncated vocabulary  $\mathcal{V}_t^-$  in an on-policy scheme. Hence, the algorithm requires the truncation, and a fortiori the language model, at test time. In this section, we investigate if we can directly learn a policy over the full vocabulary, and thus removing the truncation at test time. In such a setting, we adopt an off-policy training scheme, where the trajectories used to learn the behavior  $\pi_{\theta}$  at training time are sampled under a different policy, the truncated policy  $\pi_{\theta}^{-}$ . Thus, we need to unbiased the PG by using an importance sampling term between the exploratory policy  $\pi_{\theta}^{-}$  and the behavior policy  $\pi_{\theta}$  (Degris et al.,
Method	Score	R@5	BLEU	Meteor	CIDEr	ppl-t (↓)	ppl-e (↓)	sBLEU (↓)	peak.(↓)
Pretrain	0.38	0.59	0.30	0.40	0.93	12	24	0.80	0.99
Pretrain + RL fine-tune	0.41	0.63	0.31	0.41	0.98	21	50	0.78	0.99
Scratch	0.01	0.04	0.00	0.00	0.00	107	106	0.75	1.00
Scratch + KL-task	0.11	0.29	0.24	0.27	0.24	102	102	0.27	0.74
Scratch + KL-ext	0.01	0.05	0.06	0.04	0.01	106	103	0.10	0.20
TruFLL (Task-LM)	0.35	0.56	0.21	0.15	0.11	24	102	0.78	0.99
TruFLL (Ext-LM)	0.34	0.52	0.18	0.15	0.04	102	24	0.83	0.99
Language Quality
Language Grounding
Pairwise comparisons:  $\%$  of questions chosen for the model in bold (rows) when compared to the concurrent model (columns).
Diversity/Originality
Figure 3: VQAv2 results for Human Evaluation study detailed in Section 4.4. The three matrices on top are pairwise comparisons: each cell displays the proportion of questions chosen for the models in the row (bold) when compared to the concurrent model in the column. The table at the bottom displays the proportion of incorrect questions coming from each model among all incorrect samples. In all figures, bracket numbers indicate the model rank per criteria, from  $1 = \text{"best"}$  to  $5 = \text{"worst"}$ .
Table 3: VQAv2 metrics on 20k test episodes with 100k train episodes. Scores are averaged over the three decoding procedures. scratch+KL has  $\lambda_{KL} = 0.05$ , the truncation for TrufLL with (Task-LM) and TrufLL (Ext-LM) are respectively  $\mathrm{p_{th}}(\alpha = 0.005)$  and  $\mathrm{p_{th}}(\alpha = 0.0075)$ . Best values are underlined, best values without task-data are in bold.
pretrain (2)	pretrain+RL (3)	scratch+KL-task (5)	TrufLL (Task-LM) (4)	TrufLL (Ext-LM) (1)
Syntax errors	16%	17%	27%	24%	15%
Method	Text-gen	Score	BLEU	CIDEr	ppl-e
pretrain	greedy	0.40	0.32	1.01	51
sampling	0.37	0.30	0.88	62
lm-ranking	0.37	0.14	0.87	54
pretrain + RL	greedy	0.42	0.32	1.05	55
sampling	0.40	0.30	0.92	71
lm-ranking	0.40	0.31	0.99	26
TrufLL (Task-LM)	greedy	0.36	0.20	0.11	366
sampling	0.35	0.20	0.11	337
lm-ranking	0.34	0.21	0.11	95
TrufLL (Ext-LM)	greedy	0.36	0.18	0.04	25
sampling	0.34	0.18	0.04	28
lm-ranking	0.33	0.19	0.15	20
2012). Formally, the off-policy PPO loss is defined by:
$$
L _ {\mathrm {p p o}} ^ {o f f} (\theta) = \mathbb {E} _ {\pi_ {\theta} ^ {-}} \left[ \min (\bar {\rho} _ {t} ^ {\theta} A _ {t}, \mathrm {c l i p} (1 - \epsilon , \bar {\rho} _ {t} ^ {\theta}, 1 + \epsilon) A _ {t}) \right],
$$
where  $\bar{\rho}_{t}^{\theta} = \frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{old}}(a_{t}|s_{t})}\frac{\pi_{\theta_{old}}(a_{t}|s_{t})}{\pi_{\theta_{old}}^{-}(a_{t}|s_{t})}$  is the new ratio.
Table 5 displays the on-policy and off-policy results on both VQG tasks for TrufLL (task-LM), and is further detailed in Appendix B.3. We also
monitor the probability mass of the policy attributed to the truncated action space (sumVA). The policy only samples words within the truncated action space when  $\mathrm{sumVA} = 1$ , without needing the truncation. On CLEVR, the TrufLLoff has lower - yet close performance on language and task scores than TrufLL. As its sumVA ratios are very close to 1, the agent has learned to generalize over the full vocabulary. However, the approach does not manage to sufficiently scale to VQAv2. It could be improved with regularisation techniques and the use of TruFLL within state-of-the-art off-policy RL algorithms. We leave such possibilities to future works.
Table 4: VQAv2: Ablation on the sampling methods. Overall best values are underlined, TrufLL best values are in bold.
Algo	Score	BLEU	CIDEr	ppl-e	sBLEU	sumVA
CLEVR
TrufLL	0.56	0.17	0.06	\( 10^3 \)	0.78	N.A
TrufLLoff	0.50	0.14	0.43	\( 10^4 \)	0.88	0.96
VQAv2
TrufLL	0.35	0.21	0.11	\( 10^4 \)	0.36	N.A
TrufLLoff	0.07	0.03	0.01	\( 10^4 \)	0.05	0.08
Table 5: On-policy vs. off-policy scores: when training with an off-policy loss, we remove the truncation at test time.
4Note that we did not simplify the expression to highlight the importance sampling ratio.
Additional experiments. We sweep over truncation hyper-parameters in Table 6 of Appendix B. In Table 8, we observe that rewarding an agent with a BLEU score is sub-optimal in both language and task scores on CLEVR. In VQA, we apply temperature scheduling on the LM to perform fine-grained truncations in Table 9 of B.2. Finally, we explore TrufLL with a pre-training phase in Table 10.
Generalization of the approach. TrufLL learns conditional language models able to solve specific Natural Language Generation tasks given a context  $c$ . For solving such tasks, it only requires the context, a reward function that scores the language generated by the RL agent with respect to the task, and eventually a few natural language demonstrations fed as input prompt to the generic language model used in the truncation algorithm. Hence, the method is transferable to a wide variety of NLG tasks, without requiring upfront large-scale labelled datasets. Additionally, the RL framework allows to optimize non-differentiable objectives, making TrufLL a natural choice to learn end-to-end task-oriented dialogs, such as (De Vries et al., 2017; Das et al., 2017). Other interesting tasks for TrufLL include the ones typically found in Vision and Language Representation Learning (Lu et al., 2020a), such as Image Captioning, Grounding Referring Expressions (generation of a referring expression over a specific bounding box of an image), Caption-based Image Retrieval (generation of a caption that discriminates an image between a set of images). Reward functions for such tasks can be based on similarity scores between the generated language and the associated image or image region, which can be computed using pretrained language representations such as BERT (Devlin et al., 2019) or multi-modal pretrained systems such as ViLBERT (Lu et al., 2019). The context can be any kind of data structure (natural language, database, video, etc): if it is a linguistic input, TrufLL can be applied for instance to text summarization, paraphrase generation (with reward functions based on similarity scores between the context and the generated language) or text-based games (Ammanabrolu and Riedl, 2018).
RL and NLP Tasks. Following (Singh et al., 2002; Lemon and Pietquin, 2007), recent RL-based task-oriented dialogues (De Vries et al., 2017; Das et al., 2017; Lewis et al., 2017; Narasimhan et al., 2015) have been developed, where the policy language model is generally pretrained with SL followed RL
fine-tuning. Yang et al. (2018); Fan et al. (2018) focused on tackling VQG tasks with RL, respectively on CLEVR and on the VQG dataset. Yet, the former uses slot filling with template questions, while the later computes a mixed objective with a MLE loss using ground-truth sentences. Bahdanau et al. (2016); Rennie et al. (2017) use RL to train language models as an alternative to SL to prevent typical text degeneration issues, but within training algorithms relying on ground-truth examples from labelled datasets.
RL methods for Language Action Spaces. Several RL algorithms have been developed to tackle large discrete action spaces. Hence, Dulac-Arnold et al. (2015); Tennenholtz and Mannor (2019); Chandak et al. (2019) embed the actions into a continuous action space, and then use classic RL algorithms to learn a policy over this continuous space. Zahavy et al. (2018); Seurin et al. (2020) proposes Q-learning algorithms with an elimination signal to eliminate forbidden actions. Closer to our work, a few algorithms (Ammanabrolu and Riedl, 2018) use the structure of language to prune the action space of text-based games, but within value-based algorithms, which are less scalable to large vocabularies. Similarly to TruffLL, CALM (Yao et al., 2020) combines a pretrained language model to prune the action space with a DeepQ network, aka DRNN (He et al., 2016). Yet, its truncation language model remains fine-tuned on the RL dataset. Besides, CALM is only evaluated on a vocabulary of 697 tokens, and on 4-words action sequences.
Learning Language Models from scratch. (Ziegler et al., 2019; Garg et al., 2021) finetune pretrained GPT-2 models with RL for language generation tasks without task-related data, only using reward signals. Yet, they still face optimization and computational challenges (Parisotto et al., 2020).
We proposed TrufLL, an original approach to learn a natural language generation (NLG) task using RL, without the usual pre-training phase requiring supervised datasets. To our knowledge, this is the first RL-based algorithm dedicated to learning a word-based text-generation task, which does not rely on a pre-training phase while scaling to large vocabularies. Although it comes with its limitations, the truncated RL algorithm provided by TrufLL gets free from labelled data in task-oriented language models, presents interesting language generation properties, and provides a generic and transferable method to learn any NLG problem.
This action benefited from the support of the Chair New Gen RetAll led by l'X - École Polytechnique and the Fondation de l'Ecole Polytechnique, sponsored by CARREFOUR.
Prithviraj Ammanabrolu and Mark O Riedl. 2018. Playing text-adventure games with graph-based deep reinforcement learning. In Proc. of the North American Chapter of the Association for Computational Linguistics (NAACL).
Dzmitry Bahdanau, Philemon Brakel, Kelvin Xu, Anirudh Goyal, Ryan Lowe, Joelle Pineau, Aaron Courville, and Yoshua Bengio. 2016. An actor-critic algorithm for sequence prediction. In Proc. of the International Conference on Learning Representations (ICLR).
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015. Neural machine translation by jointly learning to align and translate. In Proc. of the International Conference on Learning Representations (ICLR).
Dzmitry Bahdanau, Harm de Vries, Timothy J O'Donnell, Shikhar Murty, Philippe Beaudoin, Yoshua Bengio, and Aaron Courville. 2019. Closure: Assessing systematic generalization of clevr models. Visually Grounded Interaction and Language (ViGIL).
Satanjeev Banerjee and Alon Lavie. 2005. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proc. of ACL Workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization.
Samy Bengio, Oriol Vinyals, Navdeep Jaitly, and Noam Shazeer. 2015. Scheduled sampling for sequence prediction with recurrent neural networks. In Proc. of Neural Information Processing Systems (NIPS).
Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. 2020. Experience grounds language.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. In Proc. of Neural Information Processing Systems (NeurIPS).
Yash Chandak, Georgios Theocharous, James Kostas, Scott Jordan, and Philip S. Thomas. 2019. Learning action representations for reinforcement learning. In Proc. of International Conference on Machine Learning (ICML).
Sanyuan Chen, Yutai Hou, Yiming Cui, Wanxiang Che, Ting Liu, and Xiangzhan Yu. 2020. Recall and learn: Fine-tuning deep pretrained language models with less forgetting. In Proc. of Empirical Methods in Natural Language Processing (EMNLP).
Leshem Choshen, Lior Fox, Zohar Aizenbud, and Omri Abend. 2020. On the weaknesses of reinforcement learning for neural machine translation. In Proc. of the International Conference on Learning Representations (ICLR).
Abhishek Das, Satwik Kottur, Khushi Gupta, Avi Singh, Deshraj Yadav, José MF Moura, Devi Parikh, and Dhruv Batra. 2017. Visual dialog. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR).
Harm De Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, and Aaron Courville. 2017. Guesswhat?! visual object discovery through multi-modal dialogue. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR).
Thomas Degris, Martha White, and Richard S Sutton. 2012. Off-policy actor-critic. arXiv preprint arXiv:1205.4839.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Gabriel Dulac-Arnold, Richard Evans, Hado van Hasselt, Peter Sunehag, Timothy Lillicrap, Jonathan Hunt, Timothy Mann, Theophane Weber, Thomas Degris, and Ben Coppin. 2015. Deep reinforcement learning in large discrete action spaces. arXiv preprint arXiv:1512.07679.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, Iain Dunning, et al. 2018. Impala: Scalable distributed deep-rl with importance weighted actor-learner architectures. In Proc. of International Conference on Machine Learning (ICML).
Zhihao Fan, Zhongyu Wei, Siyuan Wang, Yang Liu, and Xuan-Jing Huang. 2018. A reinforcement learning framework for natural question generation using bi-discriminators. In Proc. of the International Conference on Computational Linguistics (ICCL).
David A Ferrucci. 2012. Introduction to "this is watson". IBM Journal of Research and Development, 56(3.4):1-1.
Sonal Garg, Sumanth Prabhu, Hemant Misra, and G Srinivasaraghavan. 2021. Unsupervised contextual paraphrase generation using lexical control and reinforcement learning. arXiv preprint arXiv:2103.12777.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. 2017. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR).
Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. 2016. Deep reinforcement learning with a natural language action space. In Proc. of Association for Computational Linguistics (ACL).
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. *Neural computation*, 9(8):1735-1780.