SlowGuess/NAACL_2021_OCR huggingface dataset:
During training, one frame is taken from every  $K$  frames to form the supervised learning dataset. To preserve unit level actions for the executor training, all actions that happen in  $[tK, (t + 1)K)$  frames are stacked onto the  $tK$ th frame.
Reinforcement learning To train agents via reinforcement learning, we initialize them with the behavior cloning objective in Eq. (6) to provide an initial, human-meaningful grounding of message semantics (analogous to the initialization of the executor parameter  $\theta_{E}^{(0)}$  in Section 3). We then fine-tune them on game success, providing a sparse reward of 1 when agents win the game, -1 when they lose or draw the game.
As in Section 3, learned agents are trained on the game reward, using a proximal policy optimization (PPO) (Schulman et al., 2017) objective to optimize the expected reward:
$$
\mathbb {E} _ {(s, a) \sim (I, E)} R (s, a). \tag {7}
$$
Multi-task RL The original game of Hu et al. (2019) is defined by a set of attack multipliers: the aforementioned rock-paper-scissors dynamic arises because spearmen are especially effective against cavalry, archers against dragons, etc. To create alternative "tasks" in the MiniRTS environment, we create alternative versions of the game featuring different multipliers: e.g. making dragons invulnerable to archers or cavalry extra-effective against swordsmen. Table 1 shows these multipliers for the original rule, and a set of game variants with different multipliers are described in Appendix D. These variants are labeled B-J in the experiments that follow. Multiplier changes have significant effects on the optimal high-level strategy, affecting both which units are most effective overall, and how players should respond to opponents' choices.
As in Section 3, we perform multitask LLP training in the MINIRTS environment by jointly optimizing expected reward across multiple game variants at once, assigning each variant its own set of instructor parameters  $\theta_{I}$  (initialized to the same value) but sharing a single set of executor parameters  $\theta_{E}$  across all contexts. The training pseudo-code can be found in Appendix C.
Unlike in the signaling game considered in Section 3, MINIRTS is complex, and we cannot take for granted that reinforcement learning of LLPs (with either ordinary or multitask objectives) will converge to an improved good solution at all. We thus begin with an analysis of policy performance and sample efficiency, then conclude this section with an analysis of semantic drift. (Model training details can be found in Appendix B.)
4.4.1 Evaluating performance and sample efficiency
To evaluate policy quality and sample complexity, we compare the final win rate (against the fixed pool of rule-based agents) for the policy-cloned (BC), RL-tuned  $(\mathrm{RL}^{\mathrm{joint}})$ , and multitask-RL-tuned  $(\mathrm{RL}^{\mathrm{multi}})$  agents described above. We perform this evaluation for multiple game configurations: original, with the same rules used by Hu et al. (2019) for human data collection and evaluation, and 3 alternative variants (variant G, variant H, variant J), in which the relative strengths of various units has been modified (see Appendix D). We train 4 separate  $(\mathrm{RL}^{\mathrm{joint}})$  agents corresponding to each of the environments and 2  $(\mathrm{RL}^{\mathrm{multi}})$  agents. Following D'Eramo et al. (2019), we provide both training strategies with a fixed budget of training experience across environments: both  $\mathrm{RL}^{\mathrm{joint}}$  and  $\mathrm{RL}^{\mathrm{multi}}$  have been trained on the same number of game episodes per training environment. We also present the win rates for  $\mathrm{RL}^{\mathrm{joint}}$  and  $\mathrm{RL}^{\mathrm{multi}}$ , when trained on  $3\times$
Attack multiplier
Unit name	Swordman	Spearman	Cavalry	Archer	Dragon
SWORDMAN	1.0	1.5	0.5	1.0	0.0
SPEARMAN	0.5	1.0	1.5	1.0	0.0
CAVALRY	1.5	0.5	1.0	1.0	0.0
ARCHER	0.5	0.5	0.5	0.5	2.0
DRAGON	1.0	1.0	1.0	0.5	1.0
more episodes per environment.
Results are shown in Table 2. Both RL finetuning strategies allow the policy to significantly improve over the behavior-cloned initializer, showing that effective reinforcement learning of LLPs is possible in MINIRTS. In most environments, performance of the model fine-tuned with the multitask training is higher than ordinary joint RL training. When RL $^{\text{joint}}$  is provided extra training budget, it sometimes surpasses the performance of RL $^{\text{multi}}$  model with standard number of episodes. However, when RL $^{\text{multi}}$  is also given the extra training budget, it performs better in all but one environment. At a high level, these results indicate that multitask training of LLPs can be applied at small (and in some cases no) cost in accuracy and significantly less per-environment training cost.
4.4.2 Evaluating semantic drift
Next, we consider the second question from the introduction: outside of performance effects, does multitask training with populations of instructors reduce semantic drift in executors? We present two
Table 1: Attack multipliers for the original game rules. For example, cavalry are extra-effective against swordsmen (1.5 in Swordsman col.); only archers and dragons can attack dragons (nonzero entries in Dragon col.). See Appendix D for other game variants' multipliers.
Training strategy	Evaluation environment	Win rate
(standard)	(3× training)
BC	30.3	-
RL joint[orig.]	original	65.7	86.9
RL multi[orig., B, C]	76.5	90.6
BC	11.6	-
RL joint[G]	variant G	73.0	74.1
RL multi[G, H, J]	75.7	77.6
BC	26.2	-
RL joint[H]	variant H	82.2	91.4
RL multi[G, H, J]	79.4	83.5
BC	14.6	-
RL joint[J]	variant J	87.2	93.0
RL multi[G, H, J]	91.2	93.7
Table 2: Evaluation of policy quality in MINIRTS. Policies are evaluated against a rule-based opponent pool in four environments: MINIRTS with original rules, and three rule variants described in Appendix D. We compare the original behavior-cloned LLP of Hu et al. (2019) (BC) with one fine-tuned directly on the evaluation environment  $(\mathrm{RL}^{\mathrm{joint}}[\mathrm{env}])$  and one with multitask tuning on the evaluation environment and two others  $(\mathrm{RL}^{\mathrm{multi}}[\mathrm{env}1,\mathrm{env}2,\mathrm{env}3])$ . Both RL fine-tuning strategies significantly outperform their behavior-cloned initializer. When using the same number of game episodes per training environment,  $\mathrm{RL}^{\mathrm{multi}}$  is generally best but when  $\mathrm{RL}^{\mathrm{joint}}$  is provided additional budget, it sometimes beats  $\mathrm{RL}^{\mathrm{multi}}$ . Differences between models in environments original, G and J are significant  $(p < 0.05$  under a permutation test).
different quantitative evaluations that provide different perspectives on the answer to this question.
Semantic drift In MINIRTS, executor semantic drift occurs when the executor performs actions that are not consistent with the instruction produced by the instructor. (i.e., create spearman instruction produced by the instructor leads to the executor producing swordman instead). In particular, this occurs in RL $^{\text{joint}}$  because the instructor and executor can co-adapt to new semantics during exploration as they are only trained to win the game.
Agent interoperability First, we evaluate the robustness of executor policies to alternative choices of instructors. Specifically, we pair each RL-trained executor with an instructor trained either via behavior cloning (and thus guaranteed to implement the human annotators' semantics) or fine-tuning  $(\mathrm{RL}^{\mathrm{instr}})$  on a different game variant from the executor (and thus not co-adapted with it). Intuitively, to succeed at this task, executors must follow messages produced by the instructors trained in that domain. Executors that have undergone less semantic drift should perform better when paired with these different instructors. Results are shown in Table 3; here, it can be seen that multitask learning matches or exceeds the performance of single-task training on this evaluation of semantic drift in all rule variants studied, even when  $\mathrm{RL}^{\mathrm{joint}}$  is provided additional training budget. As evidence that performance comes from instructor-executor pairs,
Instructor	Executor	Eval. env.	Win rate
(standard)	(3× training)
BC	RLjoint[orig.]	original	48.4	59.2
RLmulti[orig., B, C]	60.7	67.2
RLinstr[D]	RLjoint[orig.]	variant D	74.8	85.9
RLmulti[orig., B, C]	88.3	89.1
RLinstr[E]	RLjoint[orig.]	variant E	57.5	68.7
RLmulti[orig., B, C]	72.9	76.6
RLinstr[F]	RLjoint[orig.]	variant F	73.2	83.8
RLmulti[orig., B, C]	87.3	92.3
Table 3: Evaluation of semantic drift in MINIRTS. Here, reinforcement-learned executor models are paired with instructors different from those they are trained with: either the original behavior-cloned instructor, or a instructor fine-tuned in an entirely different environment. The multitask executor RLmulti performs better than RLjoint when paired with new instructors, even when RLjoint is given additional training budget. Differences in all environments are significant  $(p < 0.05$  under a permutation test).
rather than executors alone, using a random coach paired with RL $^{\text{multi}}$  [orig., B, C] on variant D gives  $33.2\%$  accuracy. Additionally, when RL $^{\text{multi}}$  [orig., B, C] is paired with a coach from a different variant, we get an accuracy of just  $41\%$  on variant D.
Low-level action semantics As an alternative means of gaining insight into learned behaviors, we can directly inspect the correspondence between instructor messages and executor actions. We do this by uniformly sampling messages from a random instructor, then feeding them to the  $\mathrm{RL}^{\mathrm{multi}}$  and  $\mathrm{RL}^{\mathrm{joint}}$  executors and observing their choice of low-level actions  $a$ . We then restrict these these  $(m,a)$  pairs to those in which (1) the text of  $m$  includes one of the words create, build, train or make and the name of a unit (peasant, spearman, etc.) and (2)  $a$  is a TRAIN UNIT action for any unit. We then compute the empirical probability  $P(\mathrm{unit}_1 \in a \mid \mathrm{unit}_2 \in m)$  as shown in Fig. 4. If there is semantic drift, we expect to observe nonzero probability on the off-diagonal entries (the executor is building units different from those it is instructed to build).  $\mathrm{RL}^{\mathrm{multi}}$  places less probability mass on the off-diagonal entries compared to  $\mathrm{RL}^{\mathrm{joint}}$ , consistent with less semantic drift. In Fig. 4, one can also note that some word meanings change more than others. We hypothesize that, this is because like in natural languages, environmental pressures cause the meanings of some words to change at a greater rate than others. In this case, the dynamics of the game makes the spearman unit slightly stronger than the swordman unit overall. This results in unexpectedly good performance for players who accidentally misinterpret swordman as spearman. Therefore, this creates pressure for the conventional meaning of swordman to shift more than other units.
Taken together, these two evaluation results show that, when fine-tuning a policy initialized via imitation learning on the same objective, ordinary RL can be quite effective: the resulting executor model performs well even when paired with other instructors. But as in Section 3, multitask training is even more helpful, especially by reducing semantic drift in both familiar and new environments.
We have presented a theoretical and empirical analysis of semantic drift and sample efficiency in multitask reinforcement learning of latent language policies (LLPs). In a Lewis signaling game, we
Figure 4: Message-action drift in MINIRTS. Unit X on the x-axis indicates that a message  $m$  of the form build unit  $X$  was sampled. Unit Y on the y-axis shows the low-level TRAIN UNIT Y action  $a$  sampled by the executor. Matrix entries show the empirical probability  $P(a \mid m)$  computed using the relative frequencies of the sampled instructor messages and the corresponding executor actions. The total sum of off-diagonal entries is 2.98 for RL $^{\text{joint}}$  [orig.] and 1.98 for RL $^{\text{multi}}$ , indicating less semantic drift for RL $^{\text{multi}}$  [orig., B, C].
proved that multitask training can completely eliminate semantic drift. In a two-player real-time strategy game, we showed that multitask training is effective at mitigating semantic drift, improves the quality of learned policies and is sample efficient. Future work might integrate these results with other forms of population-based training (like those proposed by Gupta et al. (2019) for reference games) and explore other environmental factors affecting dynamics of language change in populations of learned agents.
We thank Hengyuan Hu for assistance in reproducing the original work and MIT Supercloud for compute resources. We also thank Eric Chu, Alana Marzoev, Ekin Akyurek and Debby Clymer for feedback and proof-reading.
Akshat Agarwal, Swaminathan Gurumurthy, Vasu Sharma, Mike Lewis, and Katia Sycara. 2019. Community regularization of visually-grounded dialog. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems, pages 1042-1050.
Jacob Andreas, Dan Klein, and Sergey Levine. 2017. Modular multitask reinforcement learning with policy sketches. In Proceedings of the 34th International Conference on Machine Learning, ICML
2017, Sydney, NSW, Australia, 6-11 August 2017, pages 166-175.
Jacob Andreas, Dan Klein, and Sergey Levine. 2018. Learning with latent language.
Pierre-Luc Bacon, Jean Harb, and Doina Precup. 2017. The option-critic architecture. In Thirty-First AAAI Conference on Artificial Intelligence.
Andreas Blank. 1999. Why do new meanings occur? A cognitive typology of the motivations for lexical semantic change. *Historical semantics and cognition*, 13(6).
Lindell Bromham, Xia Hua, Thomas G Fitzpatrick, and Simon J Greenhill. 2015. Rate of language evolution is affected by population size. Proceedings of the National Academy of Sciences, 112(7):2097-2102.
Emma Brunskill and Lihong Li. 2013. Sample complexity of multi-task reinforcement learning. In Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence, pages 122-131.
Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D Manning, and Quoc Le. 2019. Bam! Born-again multi-task networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5931-5937.
Peter Dayan and Geoffrey E. Hinton. 1993. Feudal reinforcement learning. Advances in Neural Information Processing Systems.
Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. 2019. Sharing knowledge in multi-task deep reinforcement learning. In International Conference on Learning Representations.
Thomas G Dietterich. 2000. Hierarchical reinforcement learning with the maxq value function decomposition. Journal of artificial intelligence research, 13:227-303.
Michael C Frank and Noah D Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998-998.
Edward Gibson, Richard Futrell, Julian Jara-Ettinger, Kyle Mahowald, Leon Bergen, Sivalogeswaran Ratnasingam, Mitchell Gibson, Steven T Piantadosi, and Bevil R Conway. 2017. Color naming across languages reflects color use. Proceedings of the National Academy of Sciences, 114(40):10785-10790.
Shixiang Gu, Ethan Holly, Timothy P. Lillicrap, and Sergey Levine. 2016. Deep reinforcement learning for robotic manipulation. CoRR, abs/1610.00633.
Abhinav Gupta, Ryan Lowe, Jakob Foerster, Douwe Kiela, and Joelle Pineau. 2019. Seeded self-play for language learning. In Proceedings of the Beyond Vision and LANguage: inTEgrating Realworld kKnowledge (LANTERN), pages 62-66.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360.
Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuan-dong Tian, and Mike Lewis. 2019. Hierarchical decision making by generating and following natural language instructions. In Advances in neural information processing systems, pages 10025-10034.
Simon M Huttegger, Brian Skyrms, Rory Smead, and Kevin JS Zollman. 2010. Evolutionary dynamics of Lewis signaling games: signaling systems vs. partial pooling. Synthese, 172(1):177.
Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. 2016. Reinforcement learning with unsupervised auxiliary tasks. arXiv preprint arXiv:1611.05397.
Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. 2019. Language as an abstraction for hierarchical deep reinforcement learning. In Advances in Neural Information Processing Systems, pages 9419-9431.
Sham Machandranath Kakade et al. 2003. On the sample complexity of reinforcement learning. Ph.D. thesis, University of London London, England.
Andres Karjus, Richard A Blythe, Simon Kirby, and Kenny Smith. 2020. Communicative need modulates competition in language change. arXiv preprint arXiv:2006.09277.
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.
Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. 2017. Natural language does not emerge 'naturally' in multi-agent dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2962-2967.
Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. 2020. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7663-7674.
Jason Lee, Kyunghyun Cho, and Douwe Kiela. 2019. Countering language drift via visual grounding. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4376-4386.
David Lewis. 1969. Convention: A philosophical study. John Wiley & Sons.
Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? End-to-end learning of negotiation dialogues. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2443-2453.
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep reinforcement learning for dialogue generation. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1192-1202.
Yuchen Lu, Soumye Singhal, Florian Strub, Aaron Courville, and Olivier Pietquin. 2020. Countering language drift with seeded iterated learning. In International Conference on Machine Learning, pages 6437-6447. PMLR.
April MS McMahon and McMahon April. 1994. Understanding language change. Cambridge University Press.
V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. 2013a. Playing Atari with deep reinforcement learning. CoRR.
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013b. Playing atari with deep reinforcement learning. arXiv preprint arXiv:1312.5602.
Rémi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare. 2016. Safe and efficient off-policy reinforcement learning. In Advances in Neural Information Processing Systems, pages 1054-1062.
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. 2017. Zero-shot task generalization with multi-task deep reinforcement learning. ICML.
Mark Pagel, Quentin D Atkinson, and Andrew Meade. 2007. Frequency of word-use predicts rates of lexical evolution throughout Indo-European history. Nature, 449(7163):717-720.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In Advances in neural information processing systems, pages 8026-8037.
Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, John Schulman, Emanuel Todorov, and Sergey Levine. 2017. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. CoRR, abs/1709.10087.
Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. arXiv preprint arXiv:1511.06732.
Sutton Richard, Precup Doina, and Singh Satinder. 1999. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. Artif. Intell., 112(1-2).
John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015. Trust region policy optimization. In International conference on machine learning, pages 1889-1897. PMLR.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
Damien Scieur, Vincent Roulet, Francis Bach, and Alexandre d'Aspremont. 2017. Integration methods and optimization algorithms. In Advances in Neural Information Processing Systems, pages 1109-1118.
Evan Shelhamer, Parsa Mahmoudieh, Max Argus, and Trevor Darrell. 2016. Loss is its own reward: Self-supervision for reinforcement learning. arXiv preprint arXiv:1612.07307.
Tianmin Shu, Caiming Xiong, and Richard Socher. 2018. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. In International Conference on Learning Representations.
D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, and M. Lanctot. 2016. Mastering the game of go with deep neural networks and tree search. Nature 529.
Elizabeth S Spelke. 2017. Core knowledge, language, and number. *Language Learning and Development*, 13(2):147-170.
Nisan Stiannon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. Advances in Neural Information Processing Systems, 33.
Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256.
Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144.
A Lewis signaling games: details
This appendix provides details of the formal analysis of the signaling game discussed in Section 3.
Single-task learning: Proof of Proposition 1 In the single-task case, we wish to train the policy given in Eq. (2) (with parameters  $(\theta_I,\theta_E)$ ). For the reward function  $R$  in Fig. 2, this policy has expected reward:
$$
\begin{array}{l} J(\theta_{E},\theta_{I}) = \sum_{\substack{o\in O\\ m\in M\\ a\in A}}p(o)I(m\mid o;\theta_{I})E(a\mid m;\theta_{E})R(o,a)  . \\ = \frac {1}{2} \theta_ {I} \theta_ {E} + \frac {1}{2} (1 - \theta_ {I}) (1 - \theta_ {E}) \tag {8} \\ \end{array}
$$
As noted in Eq. (4), the gradient of this expected reward with respect to agent parameters is
$$
\frac {\partial J}{\partial \theta_ {I}} = \theta_ {E} - \frac {1}{2} \tag {9}
$$
$$
\frac {\partial J}{\partial \theta_ {E}} = \theta_ {I} - \frac {1}{2} \tag {10}
$$
Performing gradient ascent will thus give a series of parameters updates with
$$
\theta_ {I} ^ {(t)} = \theta_ {E} ^ {(t - 1)} + \alpha^ {(t)} \left(\theta_ {E} ^ {(t - 1)} - \frac {1}{2}\right) \tag {11}
$$
$$
\theta_ {E} ^ {(t)} = \theta_ {I} ^ {(t - 1)} + \alpha^ {(t)} \left(\theta_ {I} ^ {(t - 1)} - \frac {1}{2}\right) \tag {12}
$$