SlowGuess/NeurIPS_2023_OCR huggingface dataset:
To mitigate the issues of handshakes, we propose mixed-play, which improves cross-play regularization by expanding the valid distribution of "self-play states" and reducing underspecification. Mixed-play rolls out episodes using varying combinations of self-play and cross-play to enlarge state coverage. All the states in the coverage will be associated with the positive rewards of self-play, so agents cannot infer reward inversion and sabotage the game. For example, if the cross-play situation from Fig. 2 becomes a valid state that self-play can continue
from, the blue agent can no longer confidently sabotage the game when it encounters this state, since this harms the self-play score as well.
Mixed-play consists of two phases in each episode: mixed-state generation and self-play. We choose a random timestep within the episode that represents the length of the first phase. Until this timestep occurs, we randomly sample the action from self-play or cross-play for both players, but do not store these transitions in the buffer. For the rest of the episode, we perform the second phase: taking self-play actions and storing them in the buffer. When optimizing, we treat this new buffer the same as we would treat self-play, but modified with a positive weight hyperparameter,  $\beta$ , representing the importance of mixed-play. Unlike ADVERSITY, our technique does not assume that the environment is partially observable and does not require privileged access to the simulator, making it applicable to our fully observable settings.
Figure 2: Example of a handshake in Overcooked. The blue player initiates a handshake by placing a plate in a strange location. Under self-play, the green player will pick up the plate, but otherwise it will continue picking up onions. If the blue player sees that the plate has not moved, it will sabotage the game.
Figure 3: Visualization of how mixed-play resolves handshakes. Agents must act in good faith since they might switch to self-play.
footage the game when it encounters this state, since
Figure 4: The Blind Bandits environment with  $k = 2$  steps. The solid lines indicate the first player's actions while the dashed lines indicate the second player's actions. The blue line represents the conventions that converge to the  $S$  reward while the orange line represents the convention that converges to the  $G$  reward.
Figure 5: Frequency of self-play scores during the optimization of the two conventions using CoMeDi. The left plot is the first convention while the right plot is the second convention. We choose  $k = 3$  as the steps in the environment, and the shaded region indicates the frequency bounds from 10 independent initializations.
After training with mixed-play, the agent cannot determine the current identity of the partner. In particular, even if the partner fails the handshake in an earlier timestep, the setting might be mixed-play instead of cross-play, as visualized in Fig. 3. If  $\beta$  is large enough, the agent will learn that handshakes significantly harm the scores in the second phase of mixed-play and will learn to act in good faith at all timesteps. In the following section, we present our full algorithm that incorporates mixed-play into our technique for generating diverse conventions.
3.4 CoMeDi: Self-play, Cross-play, and Mixed-play
So far, we have described how cross-play is useful for constructing a reward-aligned diversity metric, while mixed-play ensures that trained agents always act in good faith. In the full training regime, which we name CoMeDi, we (1) maximize self-play rewards to ensure that agents perform well in the environment, (2) minimize cross-play rewards with previously discovered conventions to ensure diversity, and (3) maximize rewards from the collected mixed-play buffers to ensure that conventions act in good faith. Note that we collect separate buffers for self-play, cross-play, and mixed-play to estimate their respective expected returns.
To train convention  $\pi_{n}$ , the full loss function we evaluate is
$$
\mathcal {L} \left(\pi_ {n}\right) = - \mathcal {J} \left(\pi_ {n}, \pi_ {n}\right) + \alpha \mathcal {J} \left(\pi_ {n}, \pi^ {*}\right) - \beta \mathcal {J} _ {\mathrm {M}} \left(\pi_ {n}, \pi^ {*}\right), \tag {8}
$$
where  $\mathcal{J}_{\mathrm{M}}$  represents the expected mixed-play reward, and  $\pi^{*}$  is the most compatible convention to  $\pi^n$  found in  $D_{n - 1}$ . We also have tunable parameters  $\alpha$  and  $\beta$  representing the weight of the cross-play and mixed-play terms. Pseudocode and specific implementation details for the algorithm incorporating this loss, including the mixed-play buffer generation, is presented in Appendix A.
We now describe our experiments and results using three different environments: a Blind Bandits environment, Balance Beam, and Overcooked [5, 34, 26]. These three environments will help illustrate the differences between statistical diversity, cross-play diversity, and mixed-play diversity. In particular, the Blind Bandits highlight the importance of reward-alignment that is present in cross-play but not statistical diversity. The Balance Beam environment demonstrates the pitfall of handshakes during cross-play minimization, and how this is mitigated with mixed-play. Finally, the Overcooked environments demonstrate the scalability of CoMeDi, and allow us to evaluate the generated set of diverse agents by training convention-aware policies to play with real human partners. More detailed descriptions of all experiments are in the appendix.
4.1 Blind Bandits
In the Blind Bandits environment, we extend the classic multi-armed bandit problem into a two-player game. The main twist is that each agent cannot observe the actions of their partner during the game, so they cannot change their strategy in reaction to their partner. It is a collaborative 2-player game
β	SP ↑	XP ↓	HS ↓	PX ↓	LS ↑	RS ↓
0.00	1.616	-0.392	0.16	0.00	1.128	-0.656
0.25	1.808	0.096	0.00	0.00	1.272	0.096
0.50	2.000	0.200	0.00	0.00	1.384	0.128
1.00	1.904	0.632	0.00	0.24	1.232	0.160
Table 1: Results of Mixed-Play in the Balance Beam Environment: SP is the self-play score. XP is the cross-play score with the first convention. HS (handshake) is the fraction of cross-play games where a player moves off the line, while PX (perfect XP) is the fraction of with a perfect 2.0 score. LS and RS are the scores under cross-play with the left-biased and right-biased hand-coded agents.
where each player takes  $k$  steps,  $k \geq 2$ , and at each step they have to choose to go left (L) or right (R). The possible trajectories when  $k = 2$  are represented in Fig. 4.
An important characteristic of the Blind Bandits environment is that all conventions that converge to  $S$  are compatible with one another while being incompatible with the only convention that converges to  $G$ . Therefore, if we want a diverse set of 2 conventions, we would like one convention to converge to  $S$  and another to converge to  $G$ .
Baseline Results for Blind Bandits. Existing statistical divergence techniques, like ADAP, typically converge to  $S$  conventions because it affords a larger space of policies. Even when trying to train more conventions, they could all converge to  $S$  conventions while still satisfying statistical diversity, since there are multiple possible distinct policies associated with  $S$ . Unfortunately, this gives a set of agents that are diverse with respect to trajectory metrics, but not with respect to the task, since the statistical diversity policies cannot lead to an agent compatible with the  $G$  convention.
CoMeDi Results for Blind Bandits. By minimizing cross-play, we can reliably train a set of agents that converge to both the  $S$  and  $G$  conventions. First, we train a single agent via self-play to converge to one of the  $S$  conventions. Then, under the cross-play regularization (with  $\alpha = 1.0$ ), the reward payoff for a second agent is a transformed environment where each self-play reward is subtracted by the cross-play reward with the first agent. All policies with a self-play score of  $S$  would have a score of 0 in the transformed environment, but all policies with a self-play score of  $G$  would still have a score of  $G$  in the transformed environment, so an agent trained on this transformed environment would converge to  $G$ . Indeed, as we see in right plot of Fig. 5, the second convention consistently converges to  $G$ .
4.2 Balance Beam
The Balance Beam environment is a collaborative environment with two players on a line that has 5 locations with unit spacing, illustrated in Fig. 6. Both agents move simultaneously and they get a high reward if they land on the same space after their actions, but they must move from their current location at each timestep. As a human-like test set of policies, we hand-design a left-biased agent and a right-biased agent. Fig. 6 shows a visualization of the environment.
As opposed to Blind Bandits, the Balance Beam environment involves multiple timesteps and is prone to handshake formation. As we will show in the next section, mixed-play allows us to combat the handshake formation of cross-play diversity in the Balance Beam environment.
Figure 6: Example of the Balance Beam environment. The agent with a white hat follows the left-biased policy. The blue agent has four different actions it can choose. If an agent steps off the line, the game ends with a large negative score.
Balance Beam Results. We train the first convention with base MAPPO, which converges to a self-play score of 2.0. The expected cross-play of this first convention with the hand-coded left-biased
agent is 0.384 while the score is 1.368 with the right-biased agent, so it acts very similar to the right-biased agent.
Ideally, we would want a second convention to have a high cross-play score with the left-biased agent and a low cross-play score with the right-biased agent so it would ensure that human players from either convention could find a compatible AI partner. Its self-play score should be 2.0, indicating an optimal equilibrium convention. Also, it should never take actions that are clearly incorrect, like stepping off of the line, which would result in a large negative score along with the early termination of the episode.
Although pure cross-play minimization  $(\beta = 0)$  results in the lowest cross-play scores (as shown in Table 1), we notice signs of handshakes:  $16\%$  of the cross-play games result in the action of stepping off of the line, as indicated in the HS column. Moreover, pure cross-play minimization does not give perfect self-play scores, indicating that it takes suboptimal actions in some states to establish a handshake. When the mixed-play weight is set to a low value  $(\beta = 0.25)$ , the agent no longer steps off the line when the convention is breached, but it still results in an imperfect convention since it takes suboptimal actions to minimize cross-play. If the mixed-play weight is set to a high amount  $(\beta = 1.0)$ , the agent follows an identical trajectory to the first convention in  $24\%$  of the games, indicating that it is overcorrecting the issue of handshakes. The best convention we found had  $\beta = 0.5$  as shown in Table 1, which was able to generate a perfect convention that does not exhibit signs of handshakes. This convention also has the highest cross-play with the left-biased agent, indicating that mixed-play enables us to generate an agent that is more aligned to human notions of "natural" conventions since this agent always acts in good faith instead of conducting handshakes.
From the results above, we empirically demonstrate that mixed-play addresses the issue of handshakes, generating a pair of meaningfully diverse agents.
4.3 Overcooked
Overcooked is a fully cooperative game where two chefs controlled by separate players try to cook and deliver as many dishes of food as possible in a given amount of time [5, 26]. The main challenge is to coordinate by subdividing tasks and not interfering with each other's progress.
For our experiments, we generate 2 baseline agents and 2 convention-aware agents using CoMeDi. The baselines are "SP" and "ADAP" which refer to the base MAPPO algorithm and training a convention-aware agent on ADAP. Our agents are "XP" and "CoMeDi" which refer to the convention-aware agents on CoMeDi where  $\beta = 0.0$  (pure cross-play minimization) and  $\beta = 1.0$  respectively.
Overcooked User Study. In both the Blind Bandits and Balance Beam environments, we could construct an artificial test set by designing human-like conventions. However, this is infeasible for Overcooked because humans can exhibit unpredictable dynamics, such as adapting to partners. Therefore, we test the quality of our four agents by experimenting on a population of 25 human users. The results of the
user study in the Coordination Ring environment are in Fig. 7. In this environment, players need to coordinate on a strategy so they do not bump into one another. A full description of the environment, simulation results, and additional user-study results in the Cramped Room layout are in the appendix. Videos of users interacting with each agent can be found on our website.
Figure 7: Results for the Coordination Ring: average scores (top left), environment visualization (top right), and user survey feedback using the 7-point Likert scale (bottom). Higher average scores are better. The dashed line is the average score single-player human score while the solid line is the average score when paired with an expert human. Higher is better for all but the last feedback score. Error bars represent the standard error.
but are in Fig. 7. In this environment, players need to be on one another. A full description of the environment, results in the Cramped Room layout are in the appendix. We have found on our website.
1	2	3	4	5	6	7	8
0.50	0.00	0.25	0.00	0.00	0.05	0.04	0.15
Table 2: Frequency of each convention being the most likely in the encountered "convention-dependent" observations in Coordination Ring.
In terms of scores, we see that CoMeDi (score of 3.68) performs best, followed by XP (2.48), ADAP (2.24), and SP (2.08). The average score when playing with an expert human partner is 3.25, so only CoMeDi surpasses human-level performance  $(p = 0.045)$ . In terms of statistical significance, CoMeDi outperforms all baselines with  $p < 0.001$ . CoMeDi also was strongest across all metrics in the survey. Users wrote that CoMeDi "knew exactly what to do next," with one user even claiming that it "is GODLIKE" and "adapted to my moves like he can see into the future." Sentiments were mixed for XP with some users reporting that it was "doing great decisions, but sometimes he seemed confused." With ADAP and SP, users complained that the bots would "block" their progress often and "just stood around the onions and did nothing."
Diversity of the Human Test Set. Although users tended to favor CoMeDi across the metrics we measured, we wanted to determine why CoMeDi performed well. Specifically, we wanted to determine whether humans actually followed a diverse set of conventions and whether CoMeDi's strength comes from finding a specific human-like convention or covering a larger observation space.
To answer these questions, we conducted an experiment where we explicitly predict the probability of being in a CoMeDi convention conditioned on the observation. Since our usage of QDagger can be approximated as a gated mixture-of-experts model, we can explicitly train a "gate" network as a classifier that determines which convention an observation belongs to. Out of the 5000 observations that CoMeDi encountered when working with users in Coordination Ring, 2349 of them were aligned with a specific convention with probability greater than  $50\%$ , which we refer to as "convention-specific" observations. The frequency of most likely conventions is reported in Table 2.
Although convention 1, which is equivalent to the pure self-play policy, behaves well in  $50\%$  of the convention-specific observations, knowledge of other conventions is still vital. Furthermore, not all players aligned with convention 1. For instance, one player had  $65\%$  of their convention-specific observations align with convention 8 while only  $27\%$  aligned with convention 1. When calculating which convention each player followed a plurality of the time, 20 players followed convention 1, three followed convention 8, and two followed convention 3. No user perfectly followed a single convention, indicating that understanding multiple conventions made CoMeDi more robust overall.
These results demonstrate that there is non-trivial diversity in the human distribution of conventions and that CoMeDi's strengths come from discovering multiple human-like conventions along with having a more robust policy to handle deviations from these conventions.
In this work, we introduce CoMeDi, an algorithm for generating diverse agents that combines the idea of minimizing cross-play with existing agents to generate novel conventions, and the technique of mixed-play to mitigate optimization challenges with cross-play. Compared to prior work that defines the difference between conventions as a difference in generated trajectory distributions, our work uses cross-play as a measure of incompatibility between conventions. As seen in the Blind Bandits environment, cross-play gives more information than statistical diversity techniques since it can identify whether conventions are redundant. In the Balance Beam environment, we show how mixed-play addresses the issue of handshakes by incentivizing agents to act in good faith at each state in the environment. Finally, we analyze Overcooked and see how a convention-aware agent trained with CoMeDi outperforms prior techniques with users, achieving expert human-level performance.
Although we use a simple BC-based algorithm for training a convention-aware agent, future work could utilize the diverse set of conventions generated by CoMeDi more effectively to create an agent that uses the history of past interactions to dynamically adapt to a user's convention.
This research was supported in part by AFOSR YIP, DARPA YIP, NSF Awards #2006388 and #2125511, and JP Morgan. We would also like to thank Jakob N. Foerster for his useful discussion with us regarding handshakes and Hengyuan Hu for clarifying the differences between CoMeDi and ADVERSITY.
[1] R. Agarwal, M. Schwarzer, P. S. Castro, A. C. Courville, and M. Bellemare. Reincarnating reinforcement learning: Reusing prior computation to accelerate progress. Advances in Neural Information Processing Systems, 35:28955-28971, 2022.
[2] N. Bard, J. N. Foerster, S. Chandar, N. Burch, M. Lanctot, H. F. Song, E. Parisotto, V. Dumoulin, S. Moitra, E. Hughes, et al. The Hanabi challenge: A new frontier for AI research. Artificial Intelligence, 280:103216, 2020.
[3] C. Boutilier. Planning, learning and coordination in multiagent decision processes. In Proceedings of the 6th conference on Theoretical aspects of rationality and knowledge, pages 195-210, 1996.
[4] K. Bullard, F. Meier, D. Kiela, J. Pineau, and J. N. Foerster. Exploring zero-shot emergent communication in embodied multi-agent populations. CoRR, abs/2010.15896, 2020. URL https://arxiv.org/abs/2010.15896.
[5] M. Carroll, R. Shah, M. K. Ho, T. Griffiths, S. Seshia, P. Abbeel, and A. Dragan. On the utility of learning about humans for human-ai coordination. In Advances in Neural Information Processing Systems, pages 5175–5186, 2019.
[6] R. Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
[7] R. Charakorn, P. Manoonpong, and N. Dilokthanakul. Generating diverse cooperative agents by learning incompatible policies. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=Uku05GOH7_6.
[8] B. Cui, A. Lupu, S. Sokota, H. Hu, D. J. Wu, and J. N. Foerster. Adversarial diversity in hanabi. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=uLE3WF3-H_5.
[9] K. Derek and P. Isola. Adaptable agent populations via a generative model of policies. In Advances in Neural Information Processing Systems, 2021.
[10] B. Eysenbach, A. Gupta, J. Ibarz, and S. Levine. Diversity is all you need: Learning skills without a reward function. CoRR, 2018. URL http://arxiv.org/abs/1802.06070.
[11] C. Florensa, Y. Duan, and P. Abbeel. Stochastic neural networks for hierarchical reinforcement learning. In International Conference on Learning Representations, 2017. URL https://openreview.net/forum?id=B1oK8aoxe.
[12] T. Greitemeyer and D. O. Mugge. Video games do affect social outcomes: A meta-analytic review of the effects of violent and prosocial video game play. *Personality and Social Psychology Bulletin*, 40(5):578–589, 2014. doi: 10.1177/0146167213520459. URL https://doi.org/10.1177/0146167213520459. PMID: 24458215.
[13] R. X. Hawkins, M. Frank, and N. D. Goodman. Convention-formation in iterated reference games. In Proceedings of the 39th Annual Meeting of the Cognitive Science Society, 2017.
[14] H. Hu, A. Lerer, A. Peysakhovich, and J. Foerster. "Other-play" for zero-shot coordination. In Proceedings of the 37th International Conference on Machine Learning, ICML, 2020.
[15] A. P. Jacob, D. J. Wu, G. Farina, A. Lerer, A. Bakhtin, J. Andreas, and N. Brown. Modeling strong and human-like gameplay with kl-regularized search. ArXiv, 2022.
[33] G. Tesauro. TD-Gammon, a Self-Teaching Backgammon Program, Achieves Master-Level Play. Neural Computation, 6(2):215–219, 03 1994. ISSN 0899-7667. doi: 10.1162/neco.1994.6.2.215. URL https://doi.org/10.1162/neco.1994.6.2.215.
[34] R. E. Wang, S. A. Wu, J. A. Evans, J. B. Tenenbaum, D. C. Parkes, and M. Kleiman-Weiner. Too many cooks: Coordinating multi-agent collaboration through inverse planning. In Proceedings of the 19th International Conference on Autonomous Agents and Multiagent Systems, AAMAS, 2020.
[35] W. Z. Wang, M. Beliaev, E. Biyik, D. A. Lazar, R. Pedarsani, and D. Sadigh. Emergent prosociality in multi-agent games through gifting. In 30th International Joint Conference on Artificial Intelligence (IJCAI), Aug. 2021. doi: 10.24963/ijcai.2021/61.
[36] C. Yu, A. Velu, E. Vinitsky, Y. Wang, A. Bayen, and Y. Wu. The surprising effectiveness ofppo in cooperative, multi-agent games. arXiv preprint arXiv:2103.01955, 2021.
A CoMeDi Algorithm Details
A.1 Mixed-play Buffer Collection
Mixed-play consists of two phases in each episode: mixed-state generation and self-play. The "input" policies are the policy for the convention we are currently training  $\pi_{n}$  and the partner policy used for cross-play optimization  $\pi^{*}$  (using the same notation from Eq 7). First, we choose a random timestep within the episode that represents the length of the first phase (Line 2). Until this timestep occurs, we randomly sample the action from self-play or cross-play for both players (Lines 5-9). We do not store any of these transitions in the training buffer. Instead, we use the state at the last timestep and pretend that this is the initial state of the environment. For the rest of the timesteps, we perform the second phase, by taking self-play actions and store that in the buffer (Lines 10-12). When optimizing, we treat this new buffer the same as we would treat self-play, but modified with a positive weight hyperparameter,  $\beta$ , representing the importance of mixed-play.
Algorithm 1: Generating Mixed Play Buffer
Input: policies  $\pi_{n},\pi^{*}$  , MDP  $M$  Output: Replay buffer from running mixed-play.   
1  $s_0,R\gets s,0$  // start state, reward   
2  $t\sim$  uniform(1,T) // length of mixed-state generation phase   
3 for  $i\gets [0,T)$  do   
4  $o^1,o^2\gets o(s_i)$    
5 if  $i <   t$  then   
6  $\pi_1^m\gets$  Randomly choose  $\pi_{n}$  or  $\pi^{*}$    
7  $\pi_2^m\gets$  Randomly choose  $\pi_{n}$  or  $\pi^{*}$    
8  $a_i^1,a_i^2\gets \pi_1^m (o^1),\pi_2^m (o^2)$    
9  $s_{i + 1}, - \leftarrow$  Step in  $M$  with  $(a_i^1,a_i^2)$    
10 else   
11  $a_i^1,a_i^2\gets \pi_n(o^1),\pi_n(o^2)$  // self-play   
12  $s_{i + 1},r_i\gets$  Step in  $M$  with  $(a_i^1,a_i^2)$  // keep reward in phase2   
Return: ReplayBuffer  $(s_{t:T},a_{t:T}^{1},a_{t:T}^{2},r_{t:T})$
A.2 Full CoMeDi Algorithm