SlowGuess/NAACL_2022_OCR huggingface dataset:
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration. In Proceedings of the International Conference on Learning Representations (ICLR).
Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. In Proc. of the International Conference on Learning Representations (ICLR).
Natasha Jaques, Asma Ghandeharioun, Judy Hanwen Shen, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Gu, and Rosalind Picard. 2019. Way off-policy batch deep reinforcement learning of implicit human preferences in dialog. arXiv preprint arXiv:1907.00456
Natasha Jaques, Shixiang Gu, Dzmitry Bahdanau, José Miguel Hernández-Lobato, Richard E Turner, and Douglas Eck. 2017. Sequence tutor: Conservative fine-tuning of sequence generation models with kl-control. In Proc. of International Conference on Machine Learning (ICML).
Natasha Jaques, Judy Hanwen Shen, Asma Ghandeharoui, Craig Ferguson, Agata Lapedriza, Noah Jones, Shixiang Shane Gu, and Rosalind Picard. 2020 Human-centric dialog training via offline reinforcement learning. In Proc. of Empirical Methods in Natural Language Processing (EMNLP).
Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. 2017. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR).
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. In Proceedings of the International Conference on Learning Representations (ICLR).
Satwik Kottur, José M. F. Moura, Devi Parikh, Dhruv Batra, and Marcus Rohrbach. 2019. CLEVR-dialog: A diagnostic dataset for multi-round reasoning in visual dialog. In Proc. of the North American Chapter of the Association for Computational Linguistics (NAACL).
Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. 2016. Multi-agent cooperation and the emergence of (natural) language. CoRR, abs/1612.07182.
Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. 2020. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Proc. of Association for Computational Linguistics (ACL).
Oliver Lemon and Olivier Pietquin. 2007. Machine learning for spoken dialogue systems. In Proc. of Conference of the International Speech Communication Association (ISCA).
Mike Lewis, Denis Yarats, Yann N Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? end-to-end learning for negotiation dialogues. In Proc. of Empirical Methods in Natural Language Processing (EMNLP).
Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2015. A diversity-promoting objective function for neural conversation models. In Proc. of the North American Chapter of the Association for Computational Linguistics (NAACL).
Jiwei Li, Will Monroe, Alan Ritter, Michel Galley, Jianfeng Gao, and Dan Jurafsky. 2016. Deep reinforcement learning for dialogue generation. In Proc. of Empirical Methods in Natural Language Processing (EMNLP).
Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólar, and C Lawrence Zitnick. 2014. Microsoft coco: Common objects in context. In Proc. of European Conference on Computer Vision (ECCV).
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. 2019. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In Proceedings of the Advances in Neural Information Processing Systems (NeurIPS).
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 2020a. 12-in-1: Multi-task vision and language representation learning. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR).
Yuchen Lu, Soumye Singhal, Florian Strub, Olivier Pietquin, and Aaron Courville. 2020b. Countering language drift with seeded iterated learning. In Proc. of International Conference on Machine Learning (ICML).
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefenstette, Shimon Whiteson, and Tim Roktaschel. 2019. A survey of reinforcement learning informed by natural language. In Proc. of International Joint Conferences on Artificial Intelligence (IJCAI).
Nasrin Mostafazadeh, Ishan Misra, Jacob Devlin, Margaret Mitchell, Xiaodong He, and Lucy Vanderwende. 2016. Generating natural questions about an image. In Proc. of Association for Computational Linguistics (ACL).
Karthik Narasimhan, Tejas Kulkarni, and Regina Barzilay. 2015. Language understanding for text-based games using deep reinforcement learning. In Proc. of Empirical Methods in Natural Language Processing (EMNLP).
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proc. of the Association for Computational Linguistics (ACL).
Emilio Parisotto, Francis Song, Jack Rae, Razvan Pascanu, Caglar Gulcehre, Siddhant Jayakumar, Max Jaderberg, Raphael Lopez Kaufman, Aidan Clark, Seb Noury, et al. 2020. Stabilizing transformers for reinforcement learning. In Proc. of International Conference on Machine Learning (ICML).
Matthew E Peters, Sebastian Ruder, and Noah A Smith. 2019. To tune or not to tune? adapting pretrained representations to diverse tasks. In Proc. of Association for Computational Linguistics (ACL).
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. 2021. Zero-shot text-to-image generation. arXiv preprint arXiv:2102.12092.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2016. Sequence level training with recurrent neural networks. In Proceedings of the International Conference on Learning Representations (ICLR).
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015. Faster r-cnn: Towards real-time object detection with region proposal networks. In Proc. of Neural Information Processing Systems (NIPS).
Steven J Rennie, Etienne Marcheret, Youssef Mroueh, Jerret Ross, and Vaibhava Goel. 2017. Self-critical sequence training for image captioning. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR).
Sebastian Ruder. 2019. Neural transfer learning for natural language processing. Ph.D. thesis, NUI Galway.
Abdelrhman Saleh, Natasha Jaques, Asma Ghandeharioun, Judy Shen, and Rosalind Picard. 2020. Hierarchical reinforcement learning for open-domain dialog. In Proc. of Conference on Artificial Intelligence (AAAI).
John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. 2016. High-dimensional continuous control using generalized advantage estimation. In Proceedings of the International Conference on Learning Representations (ICLR).
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
Mathieu Seurin, Philippe Preux, and Olivier Pietquin. 2020. "i'm sorry dave, i'm afraid i can't do that" deep q-learning from forbidden actions. In Proc. of the International Joint Conference on Neural Networks (IJCNN).
David Silver, Satinder Singh, Doina Precup, and Richard S Sutton. 2021. Reward is enough. Artificial Intelligence, page 103535.
Satinder Singh, Diane Litman, Michael Kearns, and Marilyn Walker. 2002. Optimizing dialogue management with reinforcement learning: Experiments with the njfun system. Journal of Artificial Intelligence Research, 16:105-133.
Nisan Stiannon, Long Ouyang, Jeff Wu, Daniel M Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul Christiano. 2020. Learning to summarize from human feedback. arXiv preprint arXiv:2009.01325.
Florian Strub, Harm De Vries, Jérémie Mary, Bilal Piot, Aaron Courville, and Olivier Pietquin. 2017. End-to-end optimization of goal-driven and visually grounded dialogue systems. In Proc. of International Joint Conferences on Artificial Intelligence (IJCAI).
Richard S Sutton, Andrew G Barto, et al. 1998. Introduction to reinforcement learning, volume 135. MIT press Cambridge.
Richard S Sutton, David A McAllester, Satinder P Singh, Yishay Mansour, et al. 1999. Policy gradient methods for reinforcement learning with function approximation. In Proc. of Neural Information Processing Systems (NIPS).
Guy Tennenholtz and Shie Mannor. 2019. The natural language of actions. In Proc. of International Conference on Machine Learning (ICML).
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. 2015. Cider: Consensus-based image description evaluation. In Proc. of Conference on Computer Vision and Pattern Recognition (CVPR).
Pei-Hsin Wang, Sheng-Iou Hsieh, Shih-Chieh Chang, Yu-Ting Chen, Jia-Yu Pan, Wei Wei, and Da-Chang Juan. 2020. Contextual temperature for language modeling. arXiv preprint arXiv:2012.13575.
Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256.
Qingyang Wu, Yichi Zhang, Yu Li, and Zhou Yu. 2019. Alternating roles dialog model with large-scale pre-trained language models. In Proc. of European Association for Computational Linguistics (EACL).
Jianwei Yang, Jiasen Lu, Stefan Lee, Dhruv Batra, and Devi Parikh. 2018. Visual curiosity: Learning to ask questions to learn visual recognition. In Proc. of International Conference on Machine Learning (ICML).
Shunyu Yao, Rohan Rao, Matthew Hausknecht, and Karthik Narasimhan. 2020. Keep calm and explore: Language models for action generation in text-based games. In Proc. of Empirical Methods in Natural Language Processing (EMNLP).
Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel J Mankowitz, and Shie Mannor. 2018. Learn what not to learn: Action elimination with deep reinforcement learning. In Proc. of Neural Information Processing Systems (NeurIPS).
Hugh Zhang, Daniel Duckworth, Daphne Ippolito, and Arvind Neelakantan. 2020. Trading off diversity and quality in natural language generation. In Proc. of European Association for Computational Linguistics (EACL).
Xu Zhang, Felix Xinnan Yu, Svebor Karaman, Wei Zhang, and Shih-Fu Chang. 2018. Heated-up softmax embedding. arXiv preprint arXiv:1809.04157.
Yizhe Zhang, Zhe Gan, Kai Fan, Zhi Chen, Ricardo Henao, Dinghan Shen, and Lawrence Carin. 2017. Adversarial feature matching for text generation. In Proc. of International Conference on Machine Learning (ICML).
Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. 2019. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593.
A Dataset and training details
For the BLEU and METEOR scores, we used the NLTK<sup>5</sup> implementations with the smoothing function number 2 for the BLEU score. For the CIDEr score, we used the nlg-eval implementation<sup>6</sup>.
A.2 Answer filtering
For each dataset, we remove yes and no question-answer pairs which frequency largely exceeds other answers, to avoid any bias in the question generation process, as usually done in the VQG literature (Mostafazadeh et al., 2016).
A.3 Dataset split
For CLEVR (resp. VQAv2), the RL language agent is trained for 50k (resp. 100k) episodes over the first 20k images (resp. all the images) of the training dataset, and is then evaluated on the first 5k (resp. 20k) images of the validation set. Besides, we uniformly sample the answer in the set of reference answers for each image to reduce the bias in the distribution of answers. Finally, questions are limited to 20 (resp. 10) words.
A.4 Language Agent Networks and Training
For CLEVR (resp. VQAv2), we used a single-layer LSTM with 64 (resp. 256) units for the policy network. At every time step, the LSTM input is then the concatenation of the word embedding of dimension 32 (resp. 128), the answer embedding of dimension 32 (resp. 128), and the image representation. For CLEVR, the image representation is extracted from a pretrained ResNet50 and projected into a tensor of size (32,7,7) before being flattened. For VQAv2, the image representation is the average of 200 bounding box features of dimension 1048, extracted from a faster R-CNN (Ren et al., 2015).
We optimize the full loss  $L = L_{PPO} + \alpha L_{VF} + \beta L_{E}$  with  $\alpha = 0.5$ ,  $\beta = 0.01$  and a PPO clipping ratio  $\epsilon = 0.02$  (resp. 0.01) for CLEVR (resp. VQAv2). We use Adam optimizer (Kingma and Ba, 2014) with a learning rate (lr) of  $10^{-3}$  for TrufLL and the scratch baseline,  $10^{-5}$  (resp.  $10^{-6}$ ) for RL algorithms with a pre-training phase on CLEVR (resp. VQAv2), and  $5 * 10^{-4}$  for models including a KL regularization term. We use a batch size (bs) of 128 for all models except the ones with KL regularization, for which we use a batch size of 64. Finally, for the RL from scratch baselines, we perform gradient clipping (gladclip) of 1 (resp. 5) for CLEVR and VQAv2.
Such hyper-parameters were selected, after conducting an extensive hyper-parameter search. The following values were tested:  $\beta \in \{0.01, 0.02, 0.05, 0.1\}$ ,  $\epsilon \in \{0.01, 0.02, 0.05, 0.1, 0.5, 0.9\}$ ,  $\operatorname{lr} \in \{10^{-6}, 10^{-5}, 10^{-4}, 5*10^{-4}, 10^{-3}, 5*10^{-3}, 10^{-2}, 5*10^{-2}\}$ , gradclip  $\in \{\text{None}, 1, 5, 10, 100\}$ ,  $\text{bs} \in \{32, 64, 128\}$ .
Additionally, we also tested for VQAv2 policy networks with 64, 256 and 1024 units, with respectively 32, 128 and 512 word embedding dimensions. We kept the network size giving the best performances, i.e. policy network of 256 units and 128 word embedding dimension.
A.5 Reward formula for VQAv2
In this section, we detail the reward function used for the VQAv2 task.  $r(w_{t},w_{< t},c) = \mathbb{1}_{\mathrm{rk}(\mathcal{A})\leq 10,t = T - 1}e^{-\mathrm{rk}(\mathcal{A}) / 2}$ , with  $\mathrm{rk}(\mathcal{A})$  the rank of the ground-truth answer given by the VQA model, when predicting the actual answer from the terminal state  $(c,w_{< T})$ . Formally, it is defined as:
$$
\operatorname {r k} (\mathcal {A}) = \operatorname {r a n k} \left(\operatorname {V Q A} \left(c, w _ {<   T}\right) [ \mathcal {A} ]\right),
$$
with  $\mathrm{VQA}(c, w_{<T})$  the probability distribution given by the VQA model over the set of answers, and rank the function which ranks the probability of answer  $\mathcal{A}$  within  $\mathrm{VQA}(c, w_{<T})$  probability distribution.
B.1 CLEVR
Table 6 displays the complete ablation on the truncation functions with parameters sweep. The 'sizeVA' variable indicates the average size of the truncated action space for each truncation function. Table 7 displays the
<sup>5</sup>https://www.nltk.org/
$^{6}$ https://github.com/Maluuba/nlg-eval
ablation over the three decoding procedures defined in Section 4.5. Such an ablation presents a similar pattern than VQAv2 results described in section 5.2.
Finally, Table 8 reports CLEVR metrics when using the BLEU score as the reward. While on such a task TrufLL still exhibits promising language scores, the n-grams metrics remain lower than the pretrained baselines. This illustrates that using a language similarity score as a reward signal is much less interesting than a reward based on a task completion score.
Table 6: CLEVR task: Ablation on the truncation functions with parameters sweep. Best values are in bold.
trunc.	Score	BLEU	CIDEr	ppl-e(↓)	sBLEU(↓)	Size VA
TruFLL (Task-LM)
top(k=10))	0.50	0.12	0.32	102	0.93	10
top(k=20)	0.45	0.10	0.24	103	0.87	20
pth(α=0.05)	0.55	0.18	0.63	25	0.96	4.4
pth(α=0.1)	0.47	0.18	0.87	6.7	0.98	2.4
pth(α=1/V)	0.50	0.16	0.49	41	0.97	6.6
top(p=0.85)	0.52	0.17	0.69	10.4	0.96	4.6
top(p=0.9)	0.51	0.17	0.69	11.5	0.96	5.1
sample(k=20)	0.50	0.18	0.73	18.9	0.86	5.4
sample(k=30)	0.50	0.18	0.73	16.1	0.89	6.1
TruFLL (Ext-LM)
top(k=10))	0.52	0.06	0.15	102	0.94	10
top(k=20)	0.48	0.05	0.12	102	0.89	20
pth(α=0.05)	0.48	0.08	0.34	3.03	0.95	3.3
pth(α=0.1)	0.45	0.17	0.74	2.2	0.99	2.1
pth(α=1/V)	0.44	0.11	0.37	3.7	0.96	5.7
top(p=0.85)	0.45	0.10	0.39	3.2	0.92	4.1
top(p=0.9)	0.48	0.15	0.57	2.8	0.97	4.3
sample(k=20)	0.45	0.14	0.50	2.4	0.92	4.1
sample(k=30)	0.43	0.13	0.46	2.7	0.92	4.6
Table 7: CLEVR task: Ablation on sampling methods. Best overall values are underlined, while best values for TruFLL are in bold.
method	text-gen	score	BLEU	CIDEr	ppl-e
pretrain	greedy	0.32	0.22	1.01	14
sampling	0.29	0.17	0.76	58
lm-ranking	0.28	0.18	0.73	20
pretrain + RL	greedy	0.53	0.18	0.73	24
sampling	0.40	0.16	0.68	39
lm-ranking	0.40	0.17	0.68	5
Task-LM	greedy	0.57	0.17	0.65	39
sampling	0.55	0.17	0.66	24
lm-ranking	0.51	0.16	0.65	9
Ext-LM	greedy	0.48	0.09	0.34(±0.11)	3.0
sampling	0.48	0.10	0.35(±0.11)	3.1
lm-ranking	0.48	0.06	0.34(±0.11)	2.9
B.2 VQAv2
Temperature scheduling: On the CLEVR task, we observed that dynamic truncations outperform static ones such as  $\mathrm{top}(\mathrm{k})$ : indeed, they better take into account the inherent variability of the language structure at the sentence-level. When scaling up to the 15k words of the VQAv2 task, we also dynamically decrease the truncation size through training, by applying a decreasing temperature schedule on the language model. While temperature scaling (Bahdanau et al., 2015) is usually used at test time to control the smoothness of the language model distribution, temperature schedules during training of language models have been used in several settings (Jang et al., 2016; Zhang et al., 2018; Wang et al., 2020). Formally,  $f_{LM}(w_i|w_{<t})$  distribution is computed as  $\mathrm{softmax}(x_i) = e^{-x_i / \tau} / \sum_j e^{-x_j / \tau}$ , with  $x_j$  the LM logits and  $\tau$  the temperature, which decreases from  $\tau_{max}$  to  $\tau_{min}$  by a factor  $T_F$  every  $T_u$  training step. In Table 9, both TrufLL (Task-LM) and TrufLL
Table 8: CLEVR, BLEU reward. Scores are averaged over the three decoding procedures detailed in Section 4.5 and over 5 seeds, standard deviation are displayed whenever greater than 0.01 for accuracy metrics. We here report the models with the highest task-success, i.e. the scratch with KL regularization baseline with  $\lambda_{KL} = 0.1$ , and the truncation model with a probability threshold,  $\mathrm{p_{th}}(\alpha = 0.05)$ . Baseline and Metrics are respectively detailed in Section 4.4 and 4.3. Best overall values are underlined, while best values for models without task-data (i.e RL from scratch algorithms) are in bold.
Method	Score	R@5	BLEU	Meteor	CIDEr	ppl-t (↓)	ppl-e (↓)	sBLEU (↓)	peak.(↓)
pretrain	0.30	0.71	0.19	0.38	0.83	3.1	31	0.44	0.96
pretrain + RL fine-tune	0.34	0.80	0.20	0.38	0.83	3.8	12	0.56	0.96
scratch	0.03	0.19	0.06	0.09	0.09	108	106	0.13	0.14
scratch + KL-task	0.09	0.33(±0.15)	0.15	0.31	0.58(±0.23)	3.8	63	0.34	0.95
scratch + KL-ext	0.06	0.30(±0.23)	0.13	0.25	0.42	103	3.6	0.37	0.96
scratch + Truncation-task	0.17	0.51	0.18	0.37	0.80	2.6	17	0.63	1.0
scratch + Truncation-ext	0.07	0.36	0.16	0.29	0.49	102	2.3	0.60	1.0
(Ext-LM) benefit slightly from truncation with a temperature schedule compared to a vanilla truncation. The former displays the best performance/language scores trade-off for the schedule "  $\tau$ :  $3 > 1$ . &  $T_{u} = 5,000$ ", while the latter has the best metrics trade-off for "  $\tau$ :  $1.5 > 1$ . &  $T_{u} = 5,000$ ".
Finally, Figure 4 displays the evolution of the training return for TrufLL and the baselines. As expected, the pretrain+RL fine-tune baseline return does not evolve much, confirming that the policy distribution almost does not shift through the fine-tuning phase. The training curves of TrufLL present a steady increase in the return until reaching convergence, confirming that our approach, by guiding the exploration of the action space, provides a sufficient learning signal. On the other hand, the scratch+KL baselines stay stuck to a low training return. This suggests that the KL regularization term, while encouraging the policy distribution to resemble the language model distribution, fails to capture the task pragmatics, which requires generating a language that is visually grounded.
Table 9: VQA task: Ablation on the temperature schedules. "no temp. sch" is a classic truncation without temperature scheduling. We then report different schedules  $\tau : \tau_{max} > \tau_{min}$ ,  $T_u$ , with  $\tau_{max}, \tau_{min}, T_u$ , and  $T_f = 0.75$  as defined in section B.2. Best values are in bold.
Scheduling	Score	BLEU	CIDEr	ppl-e(↓)	sBLEU(↓)
TruFLL (Task-LM)
no temp. sch	0.35	0.20	0.11	102	0.78
τ: 1.5 > 1.	0.34	0.18	0.11	102	0.79
τ: 3 > 1.	0.35	0.22	0.13	102	0.76
τ: .1.5 > 1.	0.31	0.23	0.23	102	0.73
TruFLL (Ext-LM)
no temp. sch	0.34	0.18	0.04	25	0.83
τ: 1.5 > 1.	0.33	0.19	0.05	20	0.83
τ: 3 > 1.	0.32	0.15	0.05	35	0.82
τ: 1.5 > 1.	0.29	0.16	0.08	38	0.68
B.3 Additional discussion
TrufLL with a pre-training phase. Although TrufLL aims at providing a robust method to learn a language model (almost) from scratch, we investigate whether such algorithm can be complementary to RL algorithms with a pre-training phase. Therefore, when using the task-related dataset, we evaluate TrufLL from a pretrained policy, and we refer to it as  $\mathrm{TrufLL}_{\mathrm{pretrain}}$ .
In table 10, while on CLEVR, TrufLLpretrain marginally improves the results of the pretrain+RL fine-tune baseline, the combination of TrufLL with a pre-training phase leads to performance degradation on VQAv2. This suggests that on a large vocabulary task, the language distribution learned by the SL pretrained policy is significantly different from the one learned with TrufLL.
On-policy TrufLL versus off-policy TrufLL. To ease off-policy learning, we propose to add a KL-regularization term in the RL loss (Jaques et al., 2017, 2019; Wu et al., 2019), and refer to it as  $\mathrm{TrufLL}_{\mathrm{off},\mathrm{KL}}$ . Intuitively, it encourages the policy to stay close to the language model's distribution, with a distribution support attributing negligible probabilities to words outside the truncated action space.
Figure 4: VQAv2: Training curves. Reward is a rolling average over 5000 timesteps.
Table 10: TrufLLpretrain results on the 2 tasks. Additionally, we report the results for the pretrain+RL fine-tune baseline as a comparison. Best values are in bold.
Algo	Score	BLEU	CIDEr	ppl-e	sBLEU
CLEVR
pretrain+RL	0.44	0.17	0.70	35	0.46
TrufLLpretrain	0.61	0.18	0.77	22	0.84
VQAv2
pretrain+RL	0.41	0.31	0.98	50	0.78
TrufLLpretrain	0.33	0.27	0.42	35	1.0
Table 11 displays the full results of on-policy versus off-policy scores for TrufLL (Task-LM) and TrufLL (Ext-LM) on the two tasks. The full results emphasize the challenges of the approach for the large vocabulary of VQAv2. Indeed, on the off-policy setting for such a task, the exploding values for e-ppl suggest that the optimized language agent samples incoherent words taken outside the truncated action space, as corroborated by the low values of the sumVA ratio.
Interestingly, while on CLEVR,  $\mathrm{TrufLL_{off,KL}}$  trades off task performance for language quality when compared to  $\mathrm{TrufLL_{off}}$ , on VQAv2, it mainly provides a better learning signal for the complete (large) vocabulary. In such a setting, it hence improves the global scores of the off-policy version of TrufLL, and enables a much better generalization at test time of the global policy over the full vocabulary. Yet, keeping truncation at test time remains crucial with large vocabulary. Note that for VQAv2, the poor performances of  $\mathrm{TrufLL_{off,KL}}$  on the external LM is mainly due to numerical instability challenges when using GPT-2 as the target policy of the KL regularization term.
Additionally, on-policy versus off-policy scores split per sampling procedure are displayed in table 12: unsurprisingly, greedy decoding for  $\mathrm{TrufLL_{off}}$  outperforms the two sampling-based methods, that are more penalized by the imperfect generalization of the optimized policy over the full vocabulary.