SlowGuess/NAACL_2021_OCR huggingface dataset:
The exact sequence of iterates will depend on the choice of optimization algorithm, step size, and other hyperparameters. In order to provide the most general characterization of learning in this signaling game, we consider optimization of  $\theta_{I}$  and  $\theta_{E}$  in continuous time. (This can be viewed as the limiting case of ordinary SGD as the step size goes to zero; for more discussion of relationships between gradient descent and continuous gradient flows see Scieur et al., 2017.) Taking  $\theta_{I}^{(t)}$  and  $\theta_{E}^{(t)}$  to now be functions of a real-valued variable  $t$ , optimization corresponds to the system of ordinary differential equations:
$$
\frac {\mathrm {d} \theta_ {I} ^ {(t)}}{\mathrm {d} t} = \theta_ {E} - \frac {1}{2} \tag {13}
$$
$$
\frac {\mathrm {d} \theta_ {E} ^ {(t)}}{\mathrm {d} t} = \theta_ {I} - \frac {1}{2} \tag {14}
$$
It can be verified that solutions to this system of equations have the following general form:
$$
\theta_ {I} ^ {(t)} = \left(c _ {1} + c _ {2}\right) e ^ {t} + \left(c _ {1} - c _ {2}\right) e ^ {- t} + \frac {1}{2} \tag {15}
$$
$$
\theta_ {E} ^ {(t)} = \left(c _ {1} + c _ {2}\right) e ^ {t} + \left(c _ {2} - c _ {1}\right) e ^ {- t} + \frac {1}{2} \tag {16}
$$
They are visualized in Fig. 5 (left). By setting  $t = 0$ , we can solve for  $c_{1} = \frac{1}{2}\theta_{I}^{(0)} - \frac{1}{4}$  and  $c_{2} = \frac{1}{2}\theta_{E}^{(0)} - \frac{1}{4}$ . Thus, if (and only if)  $\theta_E^{(0)} + \theta_I^{(0)} > 1$ ,  $\theta_I$  and  $\theta_E$  will both tend towards 1, and any  $\theta_I^{(0)} < \frac{1}{2}$  is susceptible to semantic drift (Fig. 5, right).
One minor complication is that probabilities must be between 0 and 1; these equations only govern  $\theta_E, \theta_I \in [0,1]$ . If we clip these values by defining:
$$
\frac {\mathrm {d} \theta_ {E} ^ {(t)}}{\mathrm {d} t} = \left\{ \begin{array}{l l} 0 & \text {i f} \theta_ {E} = 0 \\ 0 & \text {i f} \theta_ {E} = 1 \\ \theta_ {I} - \frac {1}{2} & \text {o t h e r w i s e} \end{array} \right. \tag {17}
$$
Figure 5: Dynamics of learning in the signaling game. Left: flow field for a single-task pair. Right: parameter initializations that are susceptible to semantic drift (shaded).
defining  $\theta_{I}$  analogously, and assuming that  $\theta_{E}^{(0)},\theta_{I}^{(0)}\in (0,1)$ , a small amount of additional work suffices to prove that convergence behavior is the same as the unconstrained case presented above. First observe that once one parameter has reached a value of 1 or 0, the other parameter will converge to the same value: e.g. with  $0 < \theta_{I} < 1$
$$
\frac {\mathrm {d} \theta_ {I}}{\mathrm {d} t} = \left\{ \begin{array}{c l} - \frac {1}{2} & \text {i f} \theta_ {E} = 0 \\ \frac {1}{2} & \text {i f} \theta_ {E} = 1 \end{array} \right. \tag {18}
$$
Parameters will evolve as in the unconstrained case until either  $\theta_{E}$  or  $\theta_{I}$  reaches the boundary of the feasible set. This can happen in one of four ways:
Case 1:  $\theta_E^{(t)} = 0, 0 < \theta_I^{(1)} \leq 1$ . Case 2:  $\theta_E^{(t)} = 1, 0 < \theta_I^{(1)} \leq 1$ .
Case 3:  $\theta_I^{(t)} = 0, 0 \leq \theta_E^{(1)} < 1$ . Case 4:  $\theta_I^{(t)} = 1, 0 \leq \theta_E^{(1)} < 1$ .
Semantic drift occurs in Case 1 and Case 3 and is avoided in Case 2 and Case 4. By setting Eq. (15) and Eq. (16) to 1 and solving for  $t$ , it can be verified that solutions to  $\theta^{(t)} = 1$  exist for positive  $t$  only when  $c_{1} + c_{2} = \frac{1}{2}\theta_{E}^{(0)} + \frac{1}{2}\theta_{I}^{(0)} - \frac{1}{2} > 0$ . Thus Case 2 and Case 4 occur (and semantic drift is avoided) only when  $\theta_{E}^{(0)} + \theta_{I}^{(0)} > 1$ .
Multitask learning: Proof of Proposition 2 Now suppose we train models for both  $R$  and  $R'$  simultaneously, with a shared executor and reward-specific instructors with parameters  $\theta_{I1}$  and  $\theta_{I2}$ . The expected reward is now:
$$
J \left(\theta_ {I 1}, \theta_ {I 2}, \theta_ {E}\right) = \frac {1}{4} \left(\theta_ {I 1} \theta_ {E} + \left(1 - \theta_ {I 1}\right) \left(1 - \theta_ {E}\right) + \theta_ {I 2} \left(1 - \theta_ {E}\right) + \left(1 - \theta_ {I 2}\right) \theta_ {E}\right) \tag {19}
$$
$$
\frac {\mathrm {d} \theta_ {I 1} ^ {(t)}}{\mathrm {d} t} = \frac {1}{2} \theta_ {E} ^ {(t)} - \frac {1}{4} \tag {20}
$$
$$
\frac {\mathrm {d} \theta_ {I 2} ^ {(t)}}{\mathrm {d} t} = - \frac {1}{2} \theta_ {E} ^ {(t)} + \frac {1}{4} \tag {21}
$$
$$
\frac {\mathrm {d} \theta_ {E} ^ {(t)}}{\mathrm {d} t} = \frac {1}{2} \theta_ {I 1} ^ {(t)} - \frac {1}{2} \theta_ {I 2} ^ {(t)} \tag {22}
$$
Restricting solutions to those satisfying the initial conditions  $\theta_{I1}^{(0)} = \theta_{I2}^{(0)}$ , it can again be verified that
$$
\theta_ {I 1} ^ {(t)} = \frac {1}{4} e ^ {- t / \sqrt {2}} \left(4 c _ {1} e ^ {t / \sqrt {2}} + \sqrt {2} c _ {2} e ^ {\sqrt {2} t} - \sqrt {2} c _ {2}\right) \tag {23}
$$
$$
\theta_ {I 2} ^ {(t)} = - \frac {1}{4} e ^ {- t / \sqrt {2}} \left(- 4 c _ {1} e ^ {t / \sqrt {2}} + \sqrt {2} c _ {2} e ^ {\sqrt {2} t} - \sqrt {2} c _ {2}\right) \tag {24}
$$
$$
\theta_ {E} ^ {(t)} = \frac {1}{2} \left(\theta_ {E} ^ {(0)} - \frac {1}{2}\right) \left(e ^ {(\sqrt {2} - \frac {1}{\sqrt {2}}) t} + e ^ {- \frac {1}{\sqrt {2}} t}\right) + \frac {1}{2} \tag {25}
$$
As noted in the body of the paper, the question of whether  $\theta_E \to 1$  (no semantic drift) is independent of  $S^{(0)}$  and  $S^{\prime (0)}$ , and happens whenever  $\theta_{E}^{(0)} > \frac{1}{2}$ .
Now consider the clipped version of this objective described in Eq. (17). With the initial conditions  $\theta_{I1}^{(0)} = \theta_{I2}^{(0)}$  and  $\theta_E^{(0)} > \frac{1}{2}$ ,  $\theta_{I1}$  must increase monotonically,  $\theta_{I2}$  must decrease monotonically, and  $\theta_E$  must increase monotonically within the interior of the unit cube until one of the following conditions holds:
Case 1:  $\theta_{I1} = 1$ . Thereafter,  $\theta_E > \frac{1}{2}, 0 \geq \theta_{I2} < 1$ , so  $\frac{\mathrm{d}\theta_E}{\mathrm{d}t} > 0$ .
Case 2:  $\theta_{I2} = 0$ . Thereafter,  $\theta_E > \frac{1}{2}, 0 < \theta_{I1} \leq 1$ , so  $\frac{\mathrm{d}\theta_E}{\mathrm{d}t} > 0$ .
Case 3:  $\theta_E = 1$  and will remain fixed by definition.
Thus, semantic drift is avoided globally.
B Implementation and hyperparameter details
We use the same executor and instructor architecture and model hyperparameters as used in (Hu et al., 2019). As described in section 4.2, we use a PPO objective to train  $RL^{\mathrm{joint}}$ ,  $RL^{\mathrm{multi}}$  and  $RL^{\mathrm{instr}}$  agents. For all the models, we set the PPO batch size to 32 and the PPO update epochs to 4. We do not use an entropy term as it led to instability during training. We used the Adam optimizer (Kingma and Ba, 2014) and used the learning rate of 6e-6 for  $RL^{\mathrm{joint}}$ ,  $RL^{\mathrm{multi}}$  and 3e-6 for  $RL^{\mathrm{instr}}$ . We swepted through the following learning rates: [ 1e - 7, 1e - 6, 2e - 6, 3e - 6, 4e - 6, 5e - 6, 6e - 6, 7e - 6, 8e - 6, 9e - 6, 1e - 5, 1e - 4 ] to pick the best learning rates (during evaluations) for each of the models.
Each of the agents in section 4.4 were trained with several random seeds and played 15000 game episodes per training environment. And the model checkpoints used in the experiments were picked by evaluating those agents on 100 games. 2000 game episodes were used to compute the win rates in tables 2, 3.
$RL^{\mathrm{joint}}$  and  $RL^{\mathrm{multi}}$  models takes approximately 70 hours to train on Intel Xeon Gold 6248 and Nvidia Volta V100, while,  $RL^{\mathrm{instr}}$  takes 35 hours to train on 45000 game episodes. Each instructor in the models presented in the paper have 2.8M parameter, while the executors have 2.4M parameters. Models were implemented in Pytorch(Paszke et al., 2019).
C Multitask RL LLP training pseudo-code
The training pseudo-code for Multitask RL LLP described in Section 4.3 is presented in Algorithm 1.
D Game variants
Tables 4-11 below enumerate the attack multipliers of the units under the various multi-task rule sets.
Attack multiplier
Unit name	Swordman	Spearman	Cavalry	Archer	Dragon
SWORDMAN	1.0	1.0	1.0	0.5	1.0
SPEARMAN	1.5	0.5	1.0	1.0	0.0
CAVALRY	0.5	1.0	1.5	1.0	0.0
ARCHER	0.5	0.5	0.5	0.5	2.0
DRAGON	1.0	1.5	0.5	1.0	0.0
Table 4: Rule B attack modifier
Attack multiplier
Unit name	Swordman	Spearman	Cavalry	Archer	Dragon
SWORDMAN	0.5	1.0	1.5	1.0	0.0
SPEARMAN	1.0	1.5	0.5	1.0	0.0
CAVALRY	1.5	0.5	1.0	1.0	0.0
ARCHER	1.0	1.0	1.0	0.5	1.0
DRAGON	0.5	0.5	0.5	0.5	2.0
Table 5: Rule C attack multipliers
Attack multiplier
Unit name	Swordman	Spearman	Cavalry	Archer	Dragon
SWORDMAN	0.5	0.5	0.5	0.5	2.0
SPEARMAN	1.0	1.0	1.0	0.5	1.0
CAVALRY	0.5	1.0	1.5	1.0	0.0
ARCHER	1.5	0.5	1.0	1.0	0.0
DRAGON	1.0	1.5	0.5	1.0	0.0
Table 6: Rule D attack multipliers
Attack multiplier
Unit name	Swordman	Spearman	Cavalry	Archer	Dragon
SWORDMAN	1.5	0.5	1.0	1.0	0.0
SPEARMAN	0.5	1.0	1.5	1.0	0.0
CAVALRY	1.0	1.5	0.5	1.0	0.0
ARCHER	1.0	1.0	1.0	0.5	1.0
DRAGON	0.5	0.5	0.5	0.5	2.0
Table 7: Rule E attack multipliers
Attack multiplier
Unit name	Swordman	Spearman	Cavalry	Archer	Dragon
SWORDMAN	1.0	1.5	0.5	1.0	0.0
SPEARMAN	1.0	1.0	1.0	0.5	1.0
CAVALRY	1.5	0.5	1.0	1.0	0.0
ARCHER	0.5	0.5	0.5	0.5	2.0
DRAGON	0.5	1.0	1.5	1.0	0.0
Table 8: Rule F attack multipliers
Attack multiplier
Unit name	Swordman	Spearman	Cavalry	Archer	Dragon
SWORDMAN	0.5	1.0	1.5	1.0	0.0
SPEARMAN	1.5	0.5	1.0	1.0	0.0
CAVALRY	0.5	0.5	0.5	0.5	2.0
ARCHER	1.0	1.0	1.0	0.5	1.0
DRAGON	1.0	1.5	0.5	1.0	0.0
Table 9: Rule G attack multipliers
Attack multiplier
Unit name	Swordman	Spearman	Cavalry	Archer	Dragon
SWORDMAN	0.5	1.0	1.5	1.0	0.0
SPEARMAN	1.5	0.5	1.0	1.0	0.0
CAVALRY	1.0	1.5	0.5	1.0	0.0
ARCHER	0.5	0.5	0.5	0.5	2.0
DRAGON	1.0	1.0	1.0	0.5	1.0
Table 10: Rule H attack multipliers
Attack multiplier
Unit name	Swordman	Spearman	Cavalry	Archer	Dragon
SWORDMAN	0.5	1.0	1.5	1.0	0.0
SPEARMAN	1.0	1.0	1.0	0.5	1.0
CAVALRY	1.0	1.5	0.5	1.0	0.0
ARCHER	0.5	0.5	0.5	0.5	2.0
DRAGON	1.5	0.5	1.0	1.0	0.0
Table 11: Rule J attack multipliers