Venue: NAACL 2021

# Multitasking Inhibits Semantic Drift

 Athul Paul Jacob apjacob@mit.edu MIT CSAIL&Mike Lewis miklewis@fb.com Facebook AI Research &Jacob Andreas jda@mit.edu MIT CSAIL

###### Abstract

When intelligent agents communicate to accomplish shared goals, how do these goals shape the agents' language? We study the dynamics of learning in latent language policies (LLPs), in which instructor agents generate natural-language subgoal descriptions and executor agents map these descriptions to low-level actions. LLPs can solve challenging long-horizon reinforcement learning problems and provide a rich model for studying task-oriented language use. But previous work has found that LLP training is prone to semantic drift (use of messages in ways inconsistent with their original natural language meanings). Here, we demonstrate theoretically and empirically that _multitask_ training is an effective counter to this problem: we prove that multitask training eliminates semantic drift in a well-studied family of signaling games, and show that multitask training of neural LLPs in a complex strategy game reduces drift and while improving sample efficiency.

## 1 Introduction

A major goal in the study of artificial and natural intelligence is to understand how language can scaffold more general problem-solving skills [e.g. Spelke2017], and how these skills in turn shape language itself (e.g. Gibson et al., 2017). In NLP and machine learning, **latent language policies** (LLPs; Andreas et al., 2018) provide a standard framework for studying these questions. An LLP consists of _instructor_ and _executor_ subpolicies: the instructor generates natural language messages (e.g. high-level commands or subgoals), and the executor maps these messages to sequences of low-level actions (Fig. 1). LLPs have been used to construct interactive agents capable of complex reasoning (e.g. programming by demonstration) and planning over long horizons (e.g. in strategy games; Hu et al., 2019). They promise an effective and interpretable interface between planning and control.

However, they present a number of challenges for training. As LLPs employ a human-specified space of high-level commands, they must be initialized with human supervision, typically obtained by pretraining the executor. On its own, this training paradigm restricts the quality of the learned executor policy to that exhibited in (possibly suboptimal) human supervision. For tasks like the real-time strategy game depicted in Fig. 1, we would like to study LLPs trained via **reinforcement learning****RL**, _jointly_ learning from a downstream reward signal, and optimizing both instructors and executors for task success rather than fidelity to human teachers.

Training LLPs via RL has proven difficult. Past work has identified two main challenges: primarily, the LLP-specific problem of **semantic drift**, in which agents come to deploy messages in ways

Figure 1: In latent language policies, instructor agents (a) send natural-language commands (b) to executor agents (c), which execute them in an interactive environment (d). Jointly trained instructor–executor pairs learn to use messages in ways inconsistent with their natural language meanings (top shows a real message–action pair from a model described in Section 4). We show that _multitask_ training with a population of task-specific instructors stabilizes message semantics and in some cases improves model performance.



inconsistent with their original (natural language) meanings Lewis et al. (2017); Lee et al. (2019); secondarily, the general problem of **sample inefficiency** in RL algorithms Kakade et al. (2003); Brunskill and Li (2013). Model-free deep RL is particularly notorious for requiring enormous amounts of interaction with the environment Munos et al. (2016); Mnih et al. (2013b). For LLPs to meet their promise as flexible, controllable, and understandable tools for deep learning, better approaches are needed to limit semantic drift and perhaps improve sample efficiency.

While semantic change is a constant and well-documented feature of human languages McMahon and April (1994), (human) word meanings are on the whole remarkably stable relative to the rate of change in the tasks for which words are deployed Karjus et al. (2020). In particular, disappearance of lexical items is mitigated by increased population size Bromham et al. (2015) and increased frequency of use Pagel et al. (2007). Drawing on these facts about stabilizing factors in human language, we hypothesize that training of machine learning models with latent language variables can be made more robust by incorporating a population of instructors with diverse communicative needs that exercise different parts of the lexicon.

We describe a multitask LLP training scheme in which task-specific instructors communicate with a shared executor. We show that complex long-horizon LLPs can be effectively tuned via _joint reinforcement learning_ of instructors and executors using _multitask training_:

* Section 3 presents a formal analysis of LLP training as an iterated Lewis signalling game Lewis (1969). By modeling learning in this game as a dynamical system, we completely characterize a class of simple policies that are subject to semantic drift. We show that a particular multitask training scheme eliminates the set of initializations that undergo semantic drift.
* Section 4 evaluates the empirical effectiveness of multitask learning in a real-time strategy game featuring rich language, complex complex dynamics, and LLPs implemented with deep neural networks. Again, we show that multitask training reduces semantic drift (and improves sample efficiency) of LLPs in multiple game variants.

Together, these results show that diverse shared goals and communicative needs can facilitate (and specifically stabilize) learning of communication strategies.

## 2 Background and Related Work

Deep reinforcement learning (DRL) has recently made impressive progress on many challenging domains such as games Mnih et al. (2013a); Silver et al. (2016), locomotion Schulman et al. (2015) and dexterous manipulation tasks Gu et al. (2016); Rajeswaran et al. (2017). However, even state-of-the-art approaches to reinforcement struggle with tasks involving complex goals, sparse rewards, and long time horizons. A variety of models and algorithms for _hierarchical_ reinforcement learning have been proposed to address this challenge Dayan and Hinton (1993); Dietterich (2000); Richard et al. (1999); Bacon et al. (2017) via supervised or unsupervised training of a fixed, discrete set of sub-policies.

Language can express arbitrary goals, and has compositional structure that allows generalization across commands. Building on this intuition, several recent papers have explored hierarchical RL in which natural language is used to parameterize the space of high-level actions Oh et al. (2017); Andreas et al. (2017); Shu et al. (2018); Jiang et al. (2019); Hu et al. (2019). While there are minor implementation differences between all these approaches, we will refer to them collectively as **latent language policies (LLPs)**. Like other hierarchical agents, an LLP consists of a pair of subpolicies: an instructor \(I(m~{}|~{}o)\) and an executor \(E(a~{}|~{}m,o)\). An LLP takes actions by first sampling a string-valued message \(m\sim I\) from the instructor, and then an action \(a\sim E\) from the executor. For these messages to correspond to _natural language_, rather than arbitrary strings, policies need some source of information about what human language users think they mean. This is typically accomplished by pretraining executors via human demonstrations or reinforcement learning; here we focus on the ingredients of effective joint RL of instructors and executors.

Reinforcement learning has been widely used to improve supervised language generation policies, particularly for dialogue Li et al. (2016); Lewis et al. (2017), translation Ranzato et al. (2015); Wu et al. (2016) and summarization Stiennon et al. (2020). Here, we instead focus on models where language is a latent variable as part of a hierarchical policy for a non-linguistic task 

As noted in Section 1, an observed shortcoming of reinforcement learning in all these settings is its susceptibility to **semantic drift**. In the literature on human language change Blank (1999), semantic drift refers to a variety of phenomena, including specific terms becoming more general, general terms becoming specific, and parts coming to refer to wholes. In machine learning, it refers broadly to the use of messages inconsistent with their natural language meanings in language-generation policies Lazaridou et al. (2020).

Lee et al. (2019) mitigate semantic drift in pivot-based machine translation by using visual grounding, whereas Lu et al. (2020) periodically update a student model on data generated by an RL teacher. Work in emergent communication has found that reinforcement learning tends not to learn policies with natural language-like properties Kottur et al. (2017), although population-based training has been found to be helpful Gupta et al. (2019). Most relatedly to our work, Lazaridou et al. (2020) train speaker-listener agents jointly in a visual referential communication task and introduce auxiliary loss functions for stabilizing training. Our work focuses on a more general setting where the interactions are temporally extended, have large action spaces and is partially observable. Agarwal et al. (2019) use populations of agents to reduce semantic drift in visual dialogue. We view the current paper's analysis of multitask learning as complementary to these approaches from the emergent communication literature; future work might consider ways of combining the two.

A great deal of recent work in both RL (e.g. Jaderberg et al., 2016; Shelhamer et al., 2016) and language processing (e.g. Clark et al., 2019; Gururangan et al., 2020) has observed that carefully designed training objectives can serve as a source of model-agnostic inductive bias. Our results bring these two lines of work together: multitask training improves the faithfulness and adaptability of learned language understanding models, even when optimizing for a downstream reward.

## 3 Multitask Communication in Theory: Lewis Signaling Games

We begin our analysis with the simple **signaling game** depicted in Fig. 2. In this game, one agent receives an observation, then sends a message to another agent, which then performs an action. Signaling games like this one are widely studied in NLP as models of reference resolution and language generation Frank and Goodman (2012). The instructor-executor pair may together be viewed as the simplest LLP of the kind described in Section 2.

Formally, a (2-observation, 2-message) **Lewis signalling game** is defined by:

* a set of **observations**\(O=\{o_{1},o_{2}\}\)
* a set of **messages**\(M=\{m_{1},m_{2}\}\)
* a set of **actions**\(A=\{a_{1},a_{2}\}\)
* a **reward function**\(R:O\times A\rightarrow\mathbb{R}\)

The game is played between two agents: a **instructor** (with parameters \(\theta_{I}\)), which receives an observation and samples an observation-specific message from a distribution \(I(m\mid o;\theta_{I})\); and a **executor** (with parameters \(\theta_{E}\)), which receives the instructor's message and uses it to sample an action from a distribution \(E(a\mid m;\theta_{E})\). The agents then receive a reward \(R(o,a)\) that depends on the observation and action but not on the message sent. This policy's expected reward is given by:

\[\sum_{\begin{subarray}{c}o\in O\\ m\in M\\ a\in A\end{subarray}}p(o)I(m\mid o;\theta_{I})E(a\mid m;\theta_{E})R(o,a)~{}. \tag{1}\]

Gradient ascent on Eq. (1) with respect to \(\theta_{I}\) and \(\theta_{E}\) (e.g. using a policy gradient algorithm; Williams (1992) can be used to improve the expected reward obtained by an LLP.

Figure 2: A signaling game with two possible observations, two possible messages, and two possible actions. The instructor observes either a triangle or a square, then sends a message to a executor, who pushes either the red or blue buttons. The players’ reward depends on the observation and the action but not the message. Two possible reward functions, \(R\) and \(R^{\prime}\), are shown at right.

 

As an example, the right portion of Fig. 2 shows two reward functions \(R\) and \(R^{\prime}\). In both, each observation is paired with a single action, and the executor must take the action corresponding to the observation to receive a positive reward. For \(R\), two strategies obtain the optimal expected reward of 1: one in which \(I(m_{1}\mid\blacktriangle)=1\) and \(E(\texttt{red}\mid m_{1})=1\), and one in which \(I(m_{1}\mid\blacksquare)=1\) and \(E(\texttt{blue}\mid m_{1})=1\). Almost every initialization of \(\theta_{E}\) and \(\theta_{I}\) (excluding a set of _pooling equilibria_; see e.g. Huttegger et al., 2010) converges to one of these two strategies when agents are jointly trained to optimize Eq. (1).

Semantic driftSuppose, as shown in Fig. 2, the messages \(m_{1}\) and \(m_{2}\) are not arbitrary symbols, but correspond to the natural language expressions \(m_{1}=\textit{push red}\) and \(m_{2}=\textit{push blue}\). In this case, only of the policies described above corresponds to the semantics of natural language--namely, the one in which \(E(a_{1}\mid m_{1})=E(\texttt{red}\mid\textit{push red})=1\). What is needed to ensure that a pair of agents playing converge to the natural language strategy?

In the analysis that follows, we will consider instructor and executor agents (each with a single scalar parameter \(\in[0,1]\)):

\[I(m_{j}\mid o_{i};\theta_{I}) =\begin{cases}\theta_{I}&i=j\\ 1-\theta_{I}&\text{otherwise}\end{cases} \tag{2}\] \[E(a_{j}\mid m_{i}) =\begin{cases}\theta_{E}&i=j\\ 1-\theta_{E}&\text{otherwise}\end{cases} \tag{3}\]

For the game depicted in Fig. 2, we would like to avoid any outcome in which, after training, \(\theta_{E}=E(\texttt{red}\mid\textit{push red})<\frac{1}{2}\). More generally, let us assume that we have an initial set of executor parameters that are possibly suboptimal but correspond to natural language semantics in the sense that \(E(a\mid m_{i};\theta_{E}^{(0)})>\frac{1}{2}\) if and only if the meaning of \(m\) is _do_\(a\). In this case, we will say that a parameter initialization \((\theta_{I}^{(0)},\theta_{E}^{(0)})\) undergoes **executor semantic drift** if, after training, any such \(E(a\mid m_{i};\theta_{E})=\theta_{E}<\frac{1}{2}\).

To analyze semantic drift in this game, we consider the final values of the parameters \((\theta_{I},\theta_{E})\) when optimized from an initialization \((\theta_{I}^{(0)},\theta_{E}^{(0)})\). For the reward function \(R\) depicted in Fig. 2, we can perform gradient ascent on Eq. (1) with respect to \(\theta_{I}\) and \(\theta_{E}\) in this model by observing that:

\[\frac{\partial J}{\partial\theta_{I}}=\theta_{E}-\frac{1}{2}\hskip 28.452756pt \frac{\partial J}{\partial\theta_{E}}=\theta_{I}-\frac{1}{2} \tag{4}\]

By considering the limiting behavior of gradient ascent as step size goes to zero (a **gradient flow**; see Appendix A), it is possible to give a closed-form expression for the value of these parameters as a function of time:

**Proposition 1**.: _Suppose \(\theta_{E}^{(0)}+\theta_{I}^{(0)}<1\). Then two agents optimizing Eq. (1) via Eq. (4) undergo semantic drift (converging to \(\theta_{E}=0\))._

Proof is given in Appendix A. Note in particular that semantic drift will occur whenever \(\theta_{I}^{(0)}<1-\theta_{E}^{(0)}\), which can occur even assuming a well-initialized executor with \(\theta_{E}>\frac{1}{2}\). Fig. 5 in the appendix provides a visualization of learning dynamics and these drift-susceptible initializations. However, we will next show that this drift can be eliminated via multitask training.

Multitask signaling gamesConsider a **multitask** version of this game with the **two** reward functions \(R\) and \(R^{\prime}\) depicted in Fig. 2. As discussed in the introduction and depicted in Fig. 1, our approach to multitask training focuses on sharing a single executor \(E(a\mid m;\theta_{E})\) between multiple task-specific instructors, here \(I(m\mid o;\theta_{I1})\) and \(I(m\mid o;\theta_{I2})\), both parameterized as in Eq. (2). As above, we train \((\theta_{II},\theta_{I2},\theta_{E})\) jointly to optimize:

\[\sum_{o,m,a}p(o)I(m|o;\theta_{I1})E(a|m;\theta_{E})R(o,a)\] \[+\sum_{o,m,a}p(o)I(m|o;\theta_{I2})E(a|m;\theta_{E})R^{\prime}(o,a) \tag{5}\]

We assume that the two instructors share the same initialization, with \(\theta_{I1}^{(0)}=\theta_{I2}^{(0)}=\theta_{I}^{(0)}\). In this case, the following is true:

**Proposition 2**.: _Suppose \(\theta_{E}^{(0)}>\frac{1}{2}\) and \(\theta_{I1}^{(0)}=\theta_{I2}^{(0)}\). Then three agents optimizing Eq. (5) via its gradient flow do not undergo semantic drift. In fact, the eventual executor parameter \(\theta_{E}^{(t)}\) is independent of the initial speaker parameters \(\theta_{I1}^{(0)}\) and \(\theta_{I2}^{(0)}\)._

Proof is again given in Appendix A. It is important to emphasize that these results concern the simplest possible policies for the signaling games considered here: agents with a single parameter which already "bake in" the assumption that different signals should trigger different behaviors. We leave generalization of this formal analysis to general signaling games with more complex agents and message spaces for future work, noting that--at least in this simple case--we have succeeded in constructing a concrete multitask objective that 

reduces (indeed eliminates) the set of initial model parameters subject to semantic drift.

## 4 Multitask Communication in Practice: The MiniRTS Environment

We next verify whether this result extends to the complex LLP-learning tasks discussed in Section 2. Our focus in this section is the MiniRTS environment of Hu et al. (2019) (depicted in Fig. 1), in which agents must build and control an army of units like archers, spearmen, swordman, cavalry, and dragons, each with specialized abilities, with the goal of destroying the opponent's town center. Using this game, Hu et al. (2019) crowdsourced a dataset of high-level instructions (like _attack with dragon_ and _send idle peasant to mine_) paired with low-level action sequences (Fig. 1). They showed that an LLP trained on this supervised data via behavior cloning significantly outperformed a flat policy trained with imitation learning directly on low-level action sequences.

Here we investigate (1) whether these policy-cloned LLP can be further improved via reinforcement learning directly on a sparse win-loss signal from the game, (2) whether we can improve sample efficiency during reinforcement learning by jointly training executor models on multiple game variants simultaneously through multitask learning, and (3) whether semantic drift can be avoided during multi-task training. Below, Section 4.1, Section 4.2 and Section 4.3 provide more detail about the task, model, and training procedure. Section 4.4 reports experimental results.

### Task and Training Data

MiniRTS is a partially-observable real-time strategy game environment, in which the actions of a large number of units must be coordinated on long time scales to defeat an opposing player. In a typical episode, a player must use its initial units to gather resources, use resources to build specialized structures for producing other units, and finally deploy these units to attack the opposing player's base. This involves challenging problems in both low-level tactics (controlling the placement of individual units for resource-gathering and combat) and high-level strategy (deciding which unit types to build, and when to deploy them).

MiniRTS additionally features a dataset collected from pairs of humans playing collaboratively against rule-based opponents. One human, the _instructor_, designs high-level strategies and describes them in natural language. The other human, the _executor_ observes the environment state as well as the natural language strategy descriptions from the instructor and selects appropriate low-level actions. The dataset consists of 5,392 games, with a total of 76,045 (instruction, execution) pairs.

### Model

Hu et al. (2019) use the labeled data to train an LLP for the MiniRTS environment. Our experiments use the same model architecture (Fig. 3), which we briefly review here; see the original for details.

Observation encoderThe _instructor_ and _executor_ models condition on a fixed-sized representation of the current game state which are constructed using different encoders for various aspects of the game state (Fig. 3):

* [noitemsep]
* _Spatial input encoder_: The spatial information of the map is encoded using a convolutional neural network.
* _Non-spatial input encoder_: The non-spatial attributes and internal state of game objects are encoded using a simple MLP. These include attributes like the number of enemy units, the agent's units, and resource locations.
* _Instruction encoder_: The current instruction is encoded with a recurrent neural network.
* _Auxiliary encoder_: Global variables, such as the total number of resources collected, are additionally encoded with an MLP.

Instructor modelThe _instructor_ takes in the game state from the observation encoder and produces instructions. The 500 instructions appearing most frequently in the training set are encoded with an RNN into a fixed-sized vector. The score for each instruction is proportional to its dot product with the game state encoding. This instructor model achieved the best performance on several metrics in the original work (Hu et al., 2019). By restricting the instructor to the most frequent 500 well-formed natural language strings, we are able to focus our attention on **semantic** drift. A generative model free to generate arbitrary strings might also be subject to **syntactic** drift.

Executor modelThe _executor_ predicts an action for every unit controlled by the agent based on of the current observation as encoded by the various encoders. The executor then predicts an action 

based on these features. In particular, for each unit, it predicts one of the 7 action types Idle, Continue, Gather, Attack, Train Unit, Build Building, Move), an action target location (for the Move, Attack, Gather and Build Building actions) and a unit type (for the Train Unit and Build Building actions). Taking the product of the factorized action and location arguments across all units, the action space for the executor can be enormous, with as many as \(10^{24}\) distinct actions available on a single turn.

### Training

As mentioned above, the original work of Hu et al. (2019) (and other work on learning LLPs) focused on behavior cloning or independent supervision of the _instructor_ and _executor_. In the current paper, we are interested in the the dynamics of joint reinforcement learning of LLPs in both single- and multitask settings. Experiments in Section 4.4 make use of models trained with all three strategies.

Rule-based opponent poolAgents are trained against a similar pool of rule-based bots (see Hu et al., 2019) used to collect the human data. These bots follows a randomly selected, unit-specific strategy, building a fixed number of swordmen, spearmen, cavalry, archers or dragons and attacking as soon as they are constructed.

Behavior cloningBehavior-cloned models are trained using the supervised MiniRTS dataset. Given a collection of game observations \(o\), each annotated with a high-level action \(m\) and a low-level action \(a\), we maximize:

\[\max_{\theta_{I},\theta_{E}}~{}\sum_{o,m,a}\big{[} \log I(m~{}|~{}o;\theta_{I})\] \[+\log E(a~{}|~{}m,o;\theta_{E})\big{]}\;. \tag{6}\]

During training, one frame is taken from every \(K\) frames to form the supervised learning dataset. To preserve unit level actions for the _executor_ training, all actions that happen in \([tK,(t+1)K)\) frames are stacked onto the \(tK\)th frame.

Reinforcement learningTo train agents via reinforcement learning, we initialize them with the behavior cloning objective in Eq. (6) to provide an initial, human-meaningful grounding of message semantics (analogous to the initialization of the executor parameter \(\theta_{E}^{(0)}\) in Section 3). We then fine-tune them on game success, providing a sparse reward of 1 when agents win the game, -1 when they lose or draw the game.

As in Section 3, learned agents are trained on the game reward, using a **proximal policy optimization (PPO)**Schulman et al. (2017) objective to optimize the expected reward:

\[\mathbb{E}_{(s,a)\sim(I,E)}R(s,a)\;. \tag{7}\]

Multi-task RLThe original game of Hu et al. (2019) is defined by a set of **attack multipliers**: the aforementioned rock-paper-scissors dynamic arises because spamren are especially effective against cavalty, archers against dragons, etc. To create alternative "tasks" in the MiniRTS environment, we create alternative versions of the game featuring different multipliers: e.g. making dragons invulnerable to archers or cavalry extra-effective against swordsmen. Table 1 shows these multipliers for the original rule, and a set of game **variants** with different multipliers are described in Appendix D. These variants are labeled B-J in the experiments that follow. Multiplier changes have significant effects on the optimal high-level strategy, affecting both which units are most effective overall, and how players should respond to opponents' choices.

Figure 3: State representations for the MiniRTS environment. The model encodes the spatial observations, non-spatial observations, instructions, and auxiliary information about environment state at each timestep. These features are used by both the _instructor_ and _executor_ models. Reproduced with permission from Hu et al. (2019).

 

As in Section 3, we perform multitask LLP training in the MiniRTS environment by jointly optimizing expected reward across multiple game variants at once, assigning each variant its own set of _instructor_ parameters \(\theta_{I}\) (initialized to the same value) but sharing a single set of executor parameters \(\theta_{E}\) across all contexts. The training pseudo-code can be found in Appendix C.

### Experiments

Unlike in the signaling game considered in Section 3, MiniRTS is complex, and we cannot take for granted that reinforcement learning of LLPs (with either ordinary or multitask objectives) will converge to an improved good solution at all. We thus begin with an analysis of policy performance and sample efficiency, then conclude this section with an analysis of semantic drift. (Model training details can be found in Appendix B.)

#### 4.4.1 Evaluating performance and sample efficiency

To evaluate policy quality and sample complexity, we compare the final win rate (against the fixed pool of rule-based agents) for the policy-cloned (BC), RL-tuned (RLjoint), and multitask-RL-tuned (RLmulti) agents described above. We perform this evaluation for multiple game configurations: **original**, with the same rules used by Hu et al. (2019) for human data collection and evaluation, and 3 alternative variants (**variant G**, **variant H**, **variant J**) , in which the relative strengths of various units has been modified (see Appendix D). We train 4 separate (RLjoint) agents corresponding to each of the environments and 2 (RLmulti) agents. Following D'Eramo et al. (2019), we provide both training strategies with a fixed budget of training experience _across environments_: both RLjointand RLmultihave been trained on the same number of game episodes per training environment. We also present the win rates for RLjointand RLmulti, when trained on \(3\times\) more episodes per environment.

Results are shown in Table 2. Both RL fine-tuning strategies allow the policy to significantly improve over the behavior-cloned initializer, showing that effective reinforcement learning of LLPs is possible in MiniRTS. In most environments, performance of the model fine-tuned with the multitask training is higher than ordinary joint RL training. When RLjoint is provided extra training budget, it sometimes surpasses the performance of RLmulti model with standard number of episodes. However, when RLmulti is also given the extra training budget, it performs better in all but one environment. At a high level, these results indicate that multitask training of LLPs can be applied at small (and in some cases no) cost in accuracy and significantly less per-environment training cost.

#### 4.4.2 Evaluating semantic drift

Next, we consider the second question from the introduction: outside of performance effects, does multitask training with populations of instructors reduce semantic drift in executors? We present two

\begin{table}
\begin{tabular}{l c|c|c|c|c} \hline  & \multicolumn{5}{c}{Attack multiplier} \\ \hline
**Unit name** & **Swortman** & **Spearman** & **Cavality** & **Archer** & **Dragon** \\ \hline swordman & 1.0 & 1.5 & 0.5 & 1.0 & 0.0 \\ \hline spradman & 0.5 & 1.0 & 1.5 & 1.0 & 0.0 \\ \hline cavalry & 1.5 & 0.5 & 1.0 & 1.0 & 0.0 \\ \hline acher & 0.5 & 0.5 & 0.5 & 0.5 & 2.0 \\ \hline dragon & 1.0 & 1.0 & 1.0 & 0.5 & 1.0 \\ \hline \end{tabular}
\end{table}
Table 1: Attack multipliers for the **original** game rules. For example, cavalry are extra-effective against swordsmen (1.5 in Swordsman col.); only archers and dragons can attack dragons (nonzero entries in Dragon col.). See Appendix D for other game variants’ multipliers.

\begin{table}
\begin{tabular}{l l r r} \hline \hline
* \\ RLjoint[orig..] & original & 65.7 & 86.9 \\ RLjoint[orig.., B, C] & & **76.5** & 90.6 \\ \hline BC & & 11.6 &
- \\ RLjoint[G] & variant G & 73.0 & 74.1 \\ RLjoint[G, H, J] & & **75.7** & **77.6** \\ \hline BC & & 26.2 &
- \\ RLjoint[H] & variant H & 82.2 & **91.4** \\ RLjoint[G, H, J] & & 79.4 & 83.5 \\ \hline BC & & 14.6 &
- \\ RLjoint[J] & variant J & 87.2 & 93.0 \\ RLjoint[G, H, J] & & **91.2** & 93.7 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Evaluation of policy quality in MiniRTS. Policies are evaluated against a rule-based opponent pool in four environments: MiniRTS with original rules, and three rule variants described in Appendix D. We compare the original behavior-cloned LLP of Hu et al. (2019) (BC) with one fine-tuned directly on the evaluation environment (RLjoint[env]) and one with multitask tuning on the evalution environment and two others (RLmulti[env1, env2, env3]). Both RL fine-tuning strategies significantly outperform their behavior-cloned initializer. When using the same number of game episodes per training environment, RLmultiis generally best but when RLjointIs provided additional budget, it sometimes beats RLmulti. Differences between models in environments original, G and J are significant (\(p<0.05\) under a permutation test).
 

different quantitative evaluations that provide different perspectives on the answer to this question.

Semantic driftIn MiniRTS, executor semantic drift occurs when the executor performs actions that are not consistent with the instruction produced by the instructor. ( i.e., _create spearman_ instruction produced by the instructor leads to the executor producing _swordman_ instead). In particular, this occurs in RLjointbecause the instructor and executor can co-adapt to new semantics during exploration as they are only trained to win the game.

Agent interoperabilityFirst, we evaluate the robustness of executor policies to alternative choices of instructors. Specifically, we pair each RL-trained executor with an instructor trained either via behavior cloning (and thus guaranteed to implement the human annotators' semantics) or fine-tuning (RLinstr) on a different game variant from the executor (and thus not co-adapted with it). Intuitively, to succeed at this task, executors must follow messages produced by the instructors trained in that domain. Executors that have undergone less semantic drift should perform better when paired with these different instructors. Results are shown in Table 3; here, it can be seen that multitask learning matches or exceeds the performance of single-task training on this evaluation of semantic drift in all rule variants studied, even when RLjoint is provided additional training budget. As evidence that performance comes from instructor-executor pairs, rather than executors alone, using a random coach paired with RLmulti[orig., B, C] on variant D gives 33.2% accuracy. Additionally, when RLmulti[orig., B, C] is paired with a coach from a different variant, we get an accuracy of just 41% on variant D.

Low-level action semanticsAs an alternative means of gaining insight into learned behaviors, we can directly inspect the correspondence between instructor messages and executor actions. We do this by uniformly sampling messages from a random instructor, then feeding them to the RLmulti and RLjoint executors and observing their choice of low-level actions \(a\). We then restrict these these \((m,a)\) pairs to those in which (1) the text of \(m\) includes one of the words _create, build, train_ or _make_ and the name of a unit (_peasant, spearman_, etc.) and (2) \(a\) is a Train Unit action for _any_ unit. We then compute the empirical probability \(P(\text{unit}_{1}\in a~{}|~{}\text{unit}_{2}\in m)\) as shown in Fig. 4. If there is semantic drift, we expect to observe non-zero probability on the off-diagonal entries (the executor is building units different from those it is instructed to build). RLmulti places less probability mass on the off-diagonal entries compared to RLjoint, consistent with less semantic drift. In Fig. 4, one can also note that some word meanings change more than others. We hypothesize that, this is because like in natural languages, environmental pressures cause the meanings of some words to change at a greater rate than others. In this case, the dynamics of the game makes the _spearman_ unit slightly stronger than the _swordman_ unit overall. This results in unexpectedly good performance for players who accidentally misinterpret _swordman_ as _spearman_. Therefore, this creates pressure for the conventional meaning of _swordman_ to shift more than other units.

Taken together, these two evaluation results show that, when fine-tuning a policy initialized via imitation learning on the same objective, ordinary RL can be quite effective: the resulting executor model performs well even when paired with other instructors. But as in Section 3, multitask training is even more helpful, especially by reducing semantic drift in both familiar and new environments.

## 5 Conclusions

We have presented a theoretical and empirical analysis of semantic drift and sample efficiency in multitask reinforcement learning of latent language policies (LLPs). In a Lewis signaling game, we

\begin{table}
\begin{tabular}{l l c r r} \hline \hline \multicolumn{1}{c}{**Instruction**} & \multicolumn{1}{c}{**Executor**} & \multicolumn{1}{c}{**Eval. env.**} & \multicolumn{1}{c}{**Win rate**} \\  & & **env.** & (standard) & (\(\varnothing\) = tuning) \\ \hline BC & RLjoint[orig.] & original & 48.4 & 59.2 \\  & RLmulti[orig., B, C] & original & **60.7** & **67.2** \\ \hline RLIint[D] & RLjoint[orig.] & \multirow{2}{*}{variant D} & 74.8 & 85.9 \\  & RLmulti[orig., B, C] & & **88.3** & 89.1 \\ \hline RLinit[E] & RLjoint[orig.] & \multirow{2}{*}{variant E} & 57.5 & 68.7 \\  & RLmulti[orig., B, C] & & **72.9** & **76.6** \\ \hline RLinit[F] & RLjoint[orig.] & \multirow{2}{*}{variant F} & 73.2 & 83.8 \\  & RLmulti[orig., B, C] & & **87.3** & **92.3** \\ \hline \hline \end{tabular}
\end{table}
Table 3: Evaluation of semantic drift in MiniRTS. Here, reinforcement-learned executor models are paired with instructors different from those they are trained with: either the original behavior-cloned instructor, or a instructor fine-tuned in an entirely different environment. The multitask executor RLmulti performs better than RLjoint when paired with new instructors, even when RLjoint is given additional training budget. Differences in all environments are significant (\(p<0.05\) under a permutation test).

 

proved that multitask training can completely eliminate semantic drift. In a two-player real-time strategy game, we showed that multitask training is effective at mitigating semantic drift, improves the quality of learned policies and is sample efficient. Future work might integrate these results with other forms of population-based training (like those proposed by Gupta et al. (2019) for reference games) and explore other environmental factors affecting dynamics of language change in populations of learned agents.

## Acknowledgments

We thank Hengyuan Hu for assistance in reproducing the original work and MIT Supercloud for compute resources. We also thank Eric Chu, Alana Marzoev, Ekin Akyurek and Debby Clymer for feedback and proof-reading.

## References

* Agarwal et al. (2019) Akshat Agarwal, Swaminathan Gurumurthy, Vasu Sharma, Mike Lewis, and Katia Sycara. 2019. Community regularization of visually-grounded dialog. In _Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems_, pages 1042-1050.
* Andreas et al. (2017) Jacob Andreas, Dan Klein, and Sergey Levine. 2017. Modular multitask reinforcement learning with policy sketches. In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_, pages 166-175.
* Andreas et al. (2018) Jacob Andreas, Dan Klein, and Sergey Levine. 2018. Learning with latent language.
* Bacon et al. (2017) Pierre-Luc Bacon, Jean Harb, and Doina Precup. 2017. The option-critic architecture. In _Thirty-First AAAI Conference on Artificial Intelligence_.
* Blank (1999) Andreas Blank. 1999. Why do new meanings occur? A cognitive typology of the motivations for lexical semantic change. _Historical semantics and cognition_, 13(6).
* Bromham et al. (2015) Lindell Bromham, Xia Hua, Thomas G Fitzpatrick, and Simon J Greenhill. 2015. Rate of language evolution is affected by population size. _Proceedings of the National Academy of Sciences_, 112(7):2097-2102.
* Brunskill and Li (2013) Emma Brunskill and Lihong Li. 2013. Sample complexity of multi-task reinforcement learning. In _Proceedings of the Twenty-Ninth Conference on Uncertainty in Artificial Intelligence_, pages 122-131.
* Clark et al. (2019) Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D Manning, and Quoc Le. 2019. Bam! Born-again multi-task networks for natural language understanding. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5931-5937.
* Dayan and Hinton (1993) Peter Dayan and Geoffrey E. Hinton. 1993. Feudal reinforcement learning. _Advances in Neural Information Processing Systems_.
* D'Eramo et al. (2019) Carlo D'Eramo, Davide Tateo, Andrea Bonarini, Marcello Restelli, and Jan Peters. 2019. Sharing knowledge in multi-task deep reinforcement learning. In _International Conference on Learning Representations_.
* Dietterich (2000) Thomas G Dietterich. 2000. Hierarchical reinforcement learning with the maxq value function decomposition. _Journal of artificial intelligence research_, 13:227-303.
* Frank and Goodman (2012) Michael C Frank and Noah D Goodman. 2012. Predicting pragmatic reasoning in language games. _Science_, 336(6084):998-998.
* Gibson et al. (2017) Edward Gibson, Richard Futrell, Julian Jara-Ettinger, Kyle Mahowald, Leon Bergen, Sivalogeswaran Ratnasingam, Mitchell Gibson, Steven T Piantdosi, and Bevil R Conway. 2017. Color naming across languages reflects color use. _Proceedings of the National Academy of Sciences_, 114(40):10785-10790.
* Gu et al. (2016) Shixiang Gu, Ethan Holly, Timothy P. Lillicrap, and Sergey Levine. 2016. Deep reinforcement learning for robotic manipulation. _CoRR_, abs/1610.00633.
* Gupta et al. (2019) Abhinav Gupta, Ryan Lowe, Jakob Foerster, Douwe Kiela, and Joelle Pineau. 2019. Seed self-play for language learning. In _Proceedings of the Beyond Vision and LANguage: iTEgrating Realworld kNowledge (LANTERN)_, pages 62-66.

Figure 4: Message–action drift in MiniRTS. Unit X on the x-axis indicates that a message \(m\) of the form _build unit X_ was sampled. Unit Y on the y-axis shows the low-level Train Unit Y action \(a\) sampled by the executor. Matrix entries show the empirical probability \(P(a~{}|~{}m)\) computed using the relative frequencies of the sampled instructor messages and the corresponding executor actions. The total sum of off-diagonal entries is 2.98 for RL\({}^{\text{joint}}\)[orig.] and 1.98 for RL\({}^{\text{multi}}\), indicating less semantic drift for RL\({}^{\text{multi}}\)[orig., B, C].

 

* Gururangan et al. (2020) Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020. Don't stop pretraining: Adapl language models to domains and tasks. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 8342-8360.
* Hu et al. (2019) Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis. 2019. Hierarchical decision making by generating and following natural language instructions. In _Advances in neural information processing systems_, pages 10025-10034.
* Huttegger et al. (2010) Simon M Huttegger, Brian Skyrms, Rory Smead, and Kevin JS Zollman. 2010. Evolutionary dynamics of Lewis signaling games: signaling systems vs. partial pooling. _Synthese_, 172(1):177.
* Jaderberg et al. (2016) Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu. 2016. Reinforcement learning with unsupervised auxiliary tasks. _arXiv preprint arXiv:1611.05397_.
* Jiang et al. (2019) Yiding Jiang, Shixiang Shane Gu, Kevin P Murphy, and Chelsea Finn. 2019. Language as an abstraction for hierarchical deep reinforcement learning. In _Advances in Neural Information Processing Systems_, pages 9419-9431.
* Kakade et al. (2003) Sham Machandranath Kakade et al. 2003. _On the sample complexity of reinforcement learning_. Ph.D. thesis, University of London London, England.
* Karjus et al. (2020) Andres Karjus, Richard A Blythe, Simon Kirby, and Kennyn Smith. 2020. Communicative need modulates competition in language change. _arXiv preprint arXiv:2006.09277_.
* Kingma and Ba (2014) Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. _arXiv preprint arXiv:1412.6980_.
* Kottur et al. (2017) Satwik Kottur, Jose Moura, Stefan Lee, and Dhruv Batra. 2017. Natural language does not emerge naturally'in multi-agent dialog. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2962-2967.
* Lazaridou et al. (2020) Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. 2020. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 7663-7674.
* Lee et al. (2019) Jason Lee, Kyunghyun Cho, and Douwe Kiela. 2019. Countering language drift via visual grounding. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 4376-4386.
* Lewis (1969) David Lewis. 1969. _Convention: A philosophical study_. John Wiley & Sons.
* Lewis et al. (2017) Mike Lewis, Denis Yarats, Yann Dauphin, Devi Parikh, and Dhruv Batra. 2017. Deal or no deal? End-to-end learning of negotiation dialogues. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_, pages 2443-2453.
* Li et al. (2016) Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. 2016. Deep reinforcement learning for dialogue generation. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 1192-1202.
* Lu et al. (2020) Yuchen Lu, Soumye Singhal, Florian Strub, Aaron Courville, and Olivier Pietquin. 2020. Countering language drift with seeded iterated learning. In _International Conference on Machine Learning_, pages 6437-6447. PMLR.
* McMahon and April (1994) April MS McMahon and McMahon April. 1994. _Understanding language change_. Cambridge University Press.
* Mnih et al. (2013a) V. Mnih, K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. 2013a. Playing Atari with deep reinforcement learning. _CoRR_.
* Mnih et al. (2013b) Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, and Martin Riedmiller. 2013b. Playing atari with deep reinforcement learning. _arXiv preprint arXiv:1312.5602_.
* Munos et al. (2016) Remi Munos, Tom Stepleton, Anna Harutyunyan, and Marc Bellemare. 2016. Safe and efficient off-policy reinforcement learning. In _Advances in Neural Information Processing Systems_, pages 1054-1062.
* Oh et al. (2017) Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. 2017. Zero-shot task generalization with multi-task deep reinforcement learning. _ICML_.
* Pagel et al. (2007) Mark Pagel, Quentin D Atkinson, and Andrew Meade. 2007. Frequency of word-use predicts rates of lexical evolution throughout Indo-European history. _Nature_, 449(7163):717-720.
* Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In _Advances in neural information processing systems_, pages 8026-8037.
* Rajeswaran et al. (2017) Aravind Rajeswaran, Vikash Kumar, Abhishek Gupta, John Schulman, Emanuel Todorov, and Sergey Levine. 2017. Learning complex dexterous manipulation with deep reinforcement learning and demonstrations. _CoRR_, abs/1709.10087 

Marc'Aurelio Ranzato, Sumit Chopra, Michael Auli, and Wojciech Zaremba. 2015. Sequence level training with recurrent neural networks. _arXiv preprint arXiv:1511.06732_.
* Richard et al. (1999) Sutton Richard, Precup Doina, and Singh Satinder. 1999. Between mdps and semi-mdps: A framework for temporal abstraction in reinforcement learning. _Artif. Intell._, 112(1-2).
* Schulman et al. (2015) John Schulman, Sergey Levine, Pieter Abbeel, Michael Jordan, and Philipp Moritz. 2015. Trust region policy optimization. In _International conference on machine learning_, pages 1889-1897. PMLR.
* Schulman et al. (2017) John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. _arXiv preprint arXiv:1707.06347_.
* Scieur et al. (2017) Damien Scieur, Vincent Roulet, Francis Bach, and Alexandre d'Aspremont. 2017. Integration methods and optimization algorithms. In _Advances in Neural Information Processing Systems_, pages 1109-1118.
* Shelhamer et al. (2016) Evan Shelhamer, Parsa Mahmoudieh, Max Argus, and Trevor Darrell. 2016. Loss is its own reward: Self-supervision for reinforcement learning. _arXiv preprint arXiv:1612.07307_.
* Shu et al. (2018) Tianmin Shu, Caiming Xiong, and Richard Socher. 2018. Hierarchical and interpretable skill acquisition in multi-task reinforcement learning. In _International Conference on Learning Representations_.
* Silver et al. (2016) D. Silver, A. Huang, C. Maddison, A. Guez, L. Sifre, G. Van Den Driessche, J. Schrittwieser, I. Antonoglou, V. Panneershelvam, and M. Lanctot. 2016. Mastering the game of go with deep neural networks and tree search. _Nature 529_.
* Spelke (2017) Elizabeth S Spelke. 2017. Core knowledge, language, and number. _Language Learning and Development_, 13(2):147-170.
* Stiennon et al. (2020) Nisan Stiennon, Long Ouyang, Jeffrey Wu, Daniel Ziegler, Ryan Lowe, Chelsea Voss, Alec Radford, Dario Amodei, and Paul F Christiano. 2020. Learning to summarize with human feedback. _Advances in Neural Information Processing Systems_, 33.
* Williams (1992) Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. _Machine learning_, 8(3-4):229-256.
* Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016. Google's neural machine translation system: Bridging the gap between human and machine translation. _arXiv preprint arXiv:1609.08144_.

 

Lewis signaling games: details

This appendix provides details of the formal analysis of the signaling game discussed in Section 3.

Single-task learning: Proof of Proposition 1In the single-task case, we wish to train the policy given in Eq. (2) (with parameters \((\theta_{I},\theta_{E})\)). For the reward function \(R\) in Fig. 2, this policy has expected reward:

\[J(\theta_{E},\theta_{I})=\sum_{\begin{subarray}{c}o\in O\\ m\in M\\ a\in A\end{subarray}}p(o)I(m~{}|~{}o;\theta_{I})E(a~{}|~{}m;\theta_{E})R(o,a)~{}.\] \[=\frac{1}{2}\theta_{I}\theta_{E}+\frac{1}{2}(1-\theta_{I})(1- \theta_{E}) \tag{8}\]

As noted in Eq. (4), the gradient of this expected reward with respect to agent parameters is

\[\frac{\partial J}{\partial\theta_{I}}=\theta_{E}-\frac{1}{2} \tag{9}\] \[\frac{\partial J}{\partial\theta_{E}}=\theta_{I}-\frac{1}{2} \tag{10}\]

Performing gradient ascent will thus give a series of parameters updates with

\[\theta^{(t)}_{I}=\theta^{(t-1)}_{E}+\alpha^{(t)}\big{(}\theta^{( t-1)}_{E}-\frac{1}{2}\big{)} \tag{11}\] \[\theta^{(t)}_{E}=\theta^{(t-1)}_{I}+\alpha^{(t)}\big{(}\theta^{( t-1)}_{I}-\frac{1}{2}\big{)} \tag{12}\]

The exact sequence of iterates will depend on the choice of optimization algorithm, step size, and other hyperparameters. In order to provide the most general characterization of learning in this signaling game, we consider optimization of \(\theta_{I}\) and \(\theta_{E}\) in _continuous time_. (This can be viewed as the limiting case of ordinary SGD as the step size goes to zero; for more discussion of relationships between gradient descent and continuous gradient flows see Scieur et al., 2017.) Taking \(\theta^{(t)}_{I}\) and \(\theta^{(t)}_{E}\) to now be functions of a real-valued variable \(t\), optimization corresponds to the system of ordinary differential equations:

\[\frac{\mathrm{d}\theta^{(t)}_{I}}{\mathrm{d}t}=\theta_{E}-\frac{1 }{2} \tag{13}\] \[\frac{\mathrm{d}\theta^{(t)}_{E}}{\mathrm{d}t}=\theta_{I}-\frac{1 }{2} \tag{14}\]

It can be verified that solutions to this system of equations have the following general form:

\[\theta^{(t)}_{I}=(c_{1}+c_{2})e^{t}+(c_{1}-c_{2})e^{-t}+\frac{1}{2} \tag{15}\] \[\theta^{(t)}_{E}=(c_{1}+c_{2})e^{t}+(c_{2}-c_{1})e^{-t}+\frac{1}{2} \tag{16}\]

They are visualized in Fig. 5 (left). By setting \(t=0\), we can solve for \(c_{1}=\frac{1}{2}\theta^{(0)}_{I}-\frac{1}{4}\) and \(c_{2}=\frac{1}{2}\theta^{(0)}_{E}-\frac{1}{4}\). Thus, if (and only if) \(\theta^{(0)}_{E}+\theta^{(0)}_{I}>1\), \(\theta_{I}\) and \(\theta_{E}\) will both tend towards 1, and any \(\theta^{(0)}_{I}<\frac{1}{2}\) is susceptible to semantic drift (Fig. 5, right).

One minor complication is that probabilities must be between 0 and 1; these equations only govern \(\theta_{E},\theta_{I}\in[0,1]\). If we clip these values by defining:

\[\frac{\mathrm{d}\theta^{(t)}_{E}}{\mathrm{d}t}=\begin{cases}0&\text{if $ \theta_{E}=0$}\\ 0&\text{if $\theta_{E}=1$}\\ \theta_{I}-\frac{1}{2}&\text{otherwise}\end{cases}\] (17) 

defining \(\theta_{I}\) analogously, and assuming that \(\theta^{(0)}_{E},\theta^{(0)}_{I}\in(0,1)\), a small amount of additional work suffices to prove that convergence behavior is the same as the unconstrained case presented above. First observe that once one parameter has reached a value of 1 or 0, the other parameter will converge to the same value: e.g. with \(0<\theta_{I}<1\),

\[\frac{\mathrm{d}\theta_{I}}{\mathrm{d}t}=\begin{cases}-\frac{1}{2}&\text{if $ \theta_{E}=0$}\\ \frac{1}{2}&\text{if $\theta_{E}=1$}\end{cases} \tag{18}\]

Parameters will evolve as in the unconstrained case until either \(\theta_{E}\) or \(\theta_{I}\) reaches the boundary of the feasible set. This can happen in one of four ways:

**Case 1:**: \(\theta^{(t)}_{E}=0\), \(0<\theta^{(1)}_{I}\leq 1\).
**Case 2:**: \(\theta^{(t)}_{E}=1\), \(0<\theta^{(1)}_{I}\leq 1\).
**Case 3:**: \(\theta^{(t)}_{I}=0\), \(0\leq\theta^{(1)}_{E}<1\).
**Case 4:**: \(\theta^{(t)}_{I}=1\), \(0\leq\theta^{(1)}_{E}<1\).

Semantic drift occurs in **Case 1** and **Case 3** and is avoided in **Case 2** and **Case 4**. By setting Eq. (15) and Eq. (16) to 1 and solving for \(t\), it can be verified that solutions to \(\theta^{(t)}=1\) exist for positive \(t\) only when \(c_{1}+c_{2}=\frac{1}{2}\theta^{(0)}_{E}+\frac{1}{2}\theta^{(0)}_{I}-\frac{1}{ 2}>0\). Thus **Case 2** and **Case 4** occur (and semantic drift is avoided) only when \(\theta^{(0)}_{E}+\theta^{(0)}_{I}>1\).

Multitask learning: Proof of Proposition 2Now suppose we train models for both \(R\) and \(R^{\prime}\) simultaneously, with a shared executor and reward-specific instructors with parameters \(\theta_{I1}\) and \(\theta_{I2}\). The expected reward is now:

\[J(\theta_{I1},\theta_{I2},\theta_{E})=\frac{1}{4}\Big{(}\theta_{ I1}\theta_{E}+(1-\theta_{I1})(1-\theta_{E})+\theta_{I2}(1-\theta_{E})+(1- \theta_{I2})\theta_{E}\Big{)} \tag{19}\]

Then,

\[\frac{\mathrm{d}\theta^{(t)}_{I1}}{\mathrm{d}t} =\frac{1}{2}\theta^{(t)}_{E}-\frac{1}{4} \tag{20}\] \[\frac{\mathrm{d}\theta^{(t)}_{I2}}{\mathrm{d}t} =-\frac{1}{2}\theta^{(t)}_{E}+\frac{1}{4}\] (21) \[\frac{\mathrm{d}\theta^{(t)}_{E}}{\mathrm{d}t} =\frac{1}{2}\theta^{(t)}_{I1}-\frac{1}{2}\theta^{(t)}_{I2} \tag{22}\]

Figure 5: Dynamics of learning in the signaling game. Left: flow field for a single-task pair. Right: parameter initializations that are susceptible to semantic drift (shaded).

 

Restricting solutions to those satisfying the initial conditions \(\theta_{I1}^{(0)}=\theta_{I2}^{(0)}\), it can again be verified that

\[\theta_{I1}^{(t)} =\frac{1}{4}e^{-t/\sqrt{2}}(4c_{1}e^{t/\sqrt{2}}+\sqrt{2}c_{2}e^{ \sqrt{2}t}-\sqrt{2}c_{2}) \tag{23}\] \[\theta_{I2}^{(t)} =-\frac{1}{4}e^{-t/\sqrt{2}}(-4c_{1}e^{t/\sqrt{2}}+\sqrt{2}c_{2} e^{\sqrt{2}t}-\sqrt{2}c_{2})\] (24) \[\theta_{E}^{(t)} =\frac{1}{2}\Big{(}\theta_{E}^{(0)}-\frac{1}{2}\Big{)}\Big{(}e^{ (\sqrt{2}-\frac{1}{\sqrt{2}})t}+e^{-\frac{1}{\sqrt{2}}t}\Big{)}+\frac{1}{2} \tag{25}\]

As noted in the body of the paper, the question of whether \(\theta_{E}\to 1\) (no semantic drift) is _independent_ of \(S^{(0)}\) and \(S^{(0)}\), and happens whenever \(\theta_{E}^{(0)}>\frac{1}{2}\).

Now consider the clipped version of this objective described in Eq. (17). With the initial conditions \(\theta_{I1}^{(0)}=\theta_{I2}^{(0)}\) and \(\theta_{E}^{(0)}>\frac{1}{2}\), \(\theta_{I1}\) must increase monotonically, \(\theta_{I2}\) must decrease monotonically, and \(\theta_{E}\) must increase monotonically within the interior of the unit cube until one of the following conditions holds:

**Case 1:**: \(\theta_{I1}=1\). Thereafter, \(\theta_{E}>\frac{1}{2}\), \(0\geq\theta_{I2}<1\), so \(\dfrac{\mathrm{d}\theta_{E}}{\mathrm{d}t}>0\).
**Case 2:**: \(\theta_{I2}=0\). Thereafter, \(\theta_{E}>\frac{1}{2}\), \(0<\theta_{I1}\leq 1\), so \(\dfrac{\mathrm{d}\theta_{E}}{\mathrm{d}t}>0\).
**Case 3:**: \(\theta_{E}=1\) and will remain fixed by definition.

Thus, semantic drift is avoided globally.

## Appendix B Implementation and hyperparameter details

We use the same executor and instructor architecture and model hyperparameters as used in Hu et al. (2019). As described in section 4.2, we use a PPO objective to train \(RL^{\text{joint}}\), \(RL^{\text{multi}}\) and \(RL^{\text{instr}}\) agents. For all the models, we set the PPO batch size to 32 and the PPO update epochs to 4. We do not use an entropy term as it led to instability during training. We used the Adam optimizer Kingma and Ba (2014) and used the learning rate of 6e-6 for \(RL^{\text{joint}}\), \(RL^{\text{multi}}\) and 3e-6 for \(RL^{\text{instr}}\). We sweeped through the following learning rates: \([1e-7,1e-6,2e-6,3e-6,4e-6,5e-6,6e-6,7e-6,8e-6,9e-6,1e-5,1e-4]\) to pick the best learning rates (during evaluations) for each of the models.

Each of the agents in section 4.4 were trained with several random seeds and played 15000 game episodes per training environment. And the model checkpoints used in the experiments were picked by evaluating those agents on 100 games. 2000 game episodes were used to compute the win rates in tables 2, 3.

\(RL^{\text{joint}}\) and \(RL^{\text{multi}}\) models takes approximately 70 hours to train on Intel Xeon Gold 6248 and Nvidia Volta V100, while, \(RL^{\text{inst}}\) takes 35 hours to train on 45000 game episodes. Each instructor in the models presented in the paper have 2.8M parameter, while the executors have 2.4M parameters. Models were implemented in PytorchPaszke et al. (2019) 

Multitask RL LLP training pseudo-code

The training pseudo-code for Multitask RL LLP described in Section 4.3 is presented in Algorithm 1.

```
N Environments \(M_{1}\), \(M_{2}\), ... , \(M_{N}\); N Instructors \(\pi_{I1}\), \(\pi_{I2}\), ... , \(\pi_{IN}\) ;
5 Opponents \(O_{1},O_{2}\), ..., \(O_{5}\); Shared Executor \(\pi_{E}\); foreepoch \(i=1,2,\ldots\)do foreiteration \(j=1,\ldots,N\)do Sample environment \(M_{k}\); Select policy \(\pi^{(ij)}=f(\pi_{I_{k}},\pi_{E})\) Begin \(T\) games \(g_{1},\ldots,g_{T}\) where game \(g_{\ell}\) uses opponent \(O_{\ell\%5}\) Reset Buffer \(B\) while\(g_{1},\ldots,g_{T}\) are not terminated do Simulate gameplay between \(\pi^{(ij)}\) and \(O_{\ell\%5}\) Add each game state, value and actions: (\(s_{g_{j}(t)}\), \(v_{g_{j}(t)}\), \(a_{g_{j}(t)}\)) to buffer \(B^{ij}\). end Compute rewards \(R_{1},\ldots,R_{T}\) for \(g_{1},\ldots,g_{T}\). Optimize the PPO objective w.r.t. \(\theta_{Ik}\) and \(\theta_{E}\) using buffer \(B\) for K epochs.  end if  end while
```

**Algorithm 1**Multitask LLP RL Training

## Appendix D Game variants

Tables 4-11 below enumerate the attack multipliers of the units under the various multi-task rule sets.

## Appendix E

\begin{table}
\begin{tabular}{|l||c|c|c|c|c|} \hline  & \multicolumn{6}{c|}{Attack multiplier} \\ \hline \hline
**Unit name** & **Swordman** & **Spearman** & **Cavlarry** & **Archer** & **Dragon** \\ \hline swordan & 1.0 & 1.0 & 1.0 & 0.5 & 1.0 \\ \hline spearman & 1.5 & 0.5 & 1.0 & 1.0 & 0.0 \\ \hline cavalry & 0.5 & 1.0 & 1.5 & 1.0 & 0.0 \\ \hline archer & 0.5 & 0.5 & 0.5 & 0.5 & 2.0 \\ \hline dragon & 1.0 & 1.5 & 0.5 & 1.0 & 0.0 \\ \hline \end{tabular}
\end{table}
Table 4: Rule B attack modifier

\begin{table}
\begin{tabular}{|l||c|c|c|c|c|} \hline  & \multicolumn{6}{c|}{Attack multiplier} \\ \hline \hline
**Unit name** & **Swordman** & **Spearman** & **Cavlarry** & **Archer** & **Dragon** \\ \hline swordman & 0.5 & 1.0 & 1.5 & 1.0 & 0.0 \\ \hline spearman & 1.0 & 1.5 & 0.5 & 1.0 & 0.0 \\ \hline cavalry & 1.5 & 0.5 & 1.0 & 1.0 & 0.0 \\ \hline archer & 1.0 & 1.0 & 1.0 & 0.5 & 1.0 \\ \hline dragon & 0.5 & 0.5 & 0.5 & 0.5 & 2.0 \\ \hline \end{tabular}
\end{table}
Table 5: Rule C attack multipliers 



\begin{table}
\begin{tabular}{|l||c|c|c|c|c|} \hline  & \multicolumn{5}{c|}{Attack multiplier} \\ \hline
**Unit name** & **Swordman** & **Spearman** & **Cavalry** & **Archer** & **Dragon** \\ \hline swordman & 0.5 & 0.5 & 0.5 & 0.5 & 2.0 \\ \hline spearman & 1.0 & 1.0 & 1.0 & 0.5 & 1.0 \\ \hline cavlarv & 0.5 & 1.0 & 1.5 & 1.0 & 0.0 \\ \hline archer & 1.5 & 0.5 & 1.0 & 1.0 & 0.0 \\ \hline dragon & 1.0 & 1.5 & 0.5 & 1.0 & 0.0 \\ \hline \end{tabular}
\end{table}
Table 6: Rule D attack multipliers

\begin{table}
\begin{tabular}{|l||c|c|c|c|c|} \hline  & \multicolumn{5}{c|}{Attack multiplier} \\ \hline
**Unit name** & **Swordman** & **Spearman** & **Cavalry** & **Archer** & **Dragon** \\ \hline swordman & 1.5 & 0.5 & 1.0 & 1.0 & 0.0 \\ \hline spearman & 1.0 & 1.5 & 0.5 & 1.0 & 0.0 \\ \hline separman & 1.0 & 1.0 & 1.0 & 0.5 & 1.0 \\ \hline cavlarv & 1.5 & 0.5 & 1.0 & 1.0 & 0.0 \\ \hline archer & 0.5 & 0.5 & 0.5 & 0.5 & 2.0 \\ \hline dragon & 0.5 & 1.0 & 1.5 & 1.0 & 0.0 \\ \hline \end{tabular}
\end{table}
Table 7: Rule E attack multipliers

\begin{table}
\begin{tabular}{|l||c|c|c|c|c|} \hline  & \multicolumn{5}{c|}{Attack multiplier} \\ \hline
**Unit name** & **Swordman** & **Spearman** & **Cavalry** & **Archer** & **Dragon** \\ \hline swordman & 0.5 & 1.0 & 1.5 & 1.0 & 0.0 \\ \hline spearman & 1.5 & 0.5 & 1.0 & 1.0 & 0.0 \\ \hline cavlarv & 0.5 & 0.5 & 0.5 & 0.5 & 2.0 \\ \hline archer & 1.0 & 1.0 & 1.0 & 0.5 & 1.0 \\ \hline dragon & 1.0 & 1.5 & 0.5 & 1.0 & 0.0 \\ \hline \end{tabular}
\end{table}
Table 8: Rule F attack multipliers

\begin{table}
\begin{tabular}{|l||c|c|c|c|c|} \hline  & \multicolumn{5}{c|}{Attack multiplier} \\ \hline
**Unit name** & **Swordman** & **Spearman** & **Cavalry** & **Archer** & **Dragon** \\ \hline swordman & 0.5 & 1.0 & 1.5 & 1.0 & 0.0 \\ \hline spearman & 1.5 & 0.5 & 1.0 & 1.0 & 0.0 \\ \hline caralxy & 1.0 & 1.5 & 0.5 & 1.0 & 0.0 \\ \hline archer & 0.5 & 0.5 & 0.5 & 0.5 & 2.0 \\ \hline dragon & 1.0 & 1.0 & 1.0 & 0.5 & 1.0 \\ \hline \end{tabular}
\end{table}
Table 9: Rule G attack multipliers