SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bidipta Sarkar
bidiptas@stanford.edu
Andy Shih
andyshih@cs.stanford.edu
dorsa@cs.stanford.edu
Conventions are crucial for strong performance in cooperative multi-agent games, because they allow players to coordinate on a shared strategy without explicit communication. Unfortunately, standard multi-agent reinforcement learning techniques, such as self-play, converge to conventions that are arbitrary and non-diverse, leading to poor generalization when interacting with new partners. In this work, we present a technique for generating diverse conventions by (1) maximizing their rewards during self-play, while (2) minimizing their rewards when playing with previously discovered conventions (cross-play), stimulating conventions to be semantically different. To ensure that learned policies act in good faith despite the adversarial optimization of cross-play, we introduce mixed-play, where an initial state is randomly generated by sampling self-play and cross-play transitions and the player learns to maximize the self-play reward from this initial state. We analyze the benefits of our technique on various multi-agent collaborative games, including Overcooked, and find that our technique can adapt to the conventions of humans, surpassing human-level performance when paired with real users.<sup>1</sup>
In multi-agent cooperative games, understanding the intent of other players is crucial for seamless coordination. For example, imagine a game where teams of two players work together to cook meals for customers. Players have to manage the ingredients, use the stove, and deliver meals. As the team works together, they decide how tasks should be allocated among themselves so resources are used effectively. For example, player 1 could notice that player 2 tends to stay near the stove, so they instead spend more time preparing ingredients and delivering food, allowing player 2 to continue working at the stove. Through these interactions, the team creates a "convention" in the environment, which is an arbitrary solution to a recurring coordination problem [20, 3]. In the context of reinforcement learning, we can represent a specific convention as the joint policy function that each agent uses to choose their action given their own observation. If we wish to train an AI agent to work well with humans, it also needs to be aware of common conventions and be flexible to changing its strategy to adapt to human behavior [2, 29].
One approach to training an AI agent in a cooperative setting, called self-play [33], mimics the behavior of forming conventions by training a team of agents together to increase their reward in the environment, causing each player to adapt to the team's behavior. Eventually, self-play converges to an equilibrium, where each player's policy maximizes the score if their team does not change.
Unfortunately, self-play results in incredibly brittle policies that cannot work well with humans who may have different conventions [29], including conventions that are more intuitive for people to use. In the cooking task, self-play could converge to a convention that expects player 2 to bring onions to AI player 1 from the right side. If a human player instead decides to bring onions from the left side to the AI player 1, the AI's policy may not react appropriately since this is not an interaction
Supplemental videos can be found on our website along with source code and anonymized user study data.
that the agent has experienced under self-play. One way to address this issue is to train a diverse set of conventions that is representative of the space of all possible conventions. With a diverse set of conventions, we can train a single agent that will work well with arbitrary conventions or adapt to new conventions when working with humans.
To generate a diverse set of policies, prior works have used statistical divergence techniques that measure diversity as the difference in action distributions at each state between policies [9]. They assert that policies that result in different actions at each state will lead to players taking different paths in the environment, hence increasing diversity. However, it is often easy to trick this proxy objective for diversity by taking different actions that still lead to the same outcome. For example, in Fig. 1, we see two conventions that divide the task in the same way but differ in navigation. Even though this difference does not interfere with the partner and the conventions are perfectly compatible with one another, statistical divergence metrics treat these two policies as entirely different. Therefore, statistical diversity techniques do not fundamentally capture whether the conventions follow genuinely different strategies.
Figure 1: Example of conventions in Overcooked. Each team needs to (1) pick up onions, (2) drop them into pots, (3) fill up a dish, and (4) drop it off at the counter. There are 2 conventions shown above, a red convention and an orange convention, which only differ in terms of navigation, so they are compatible with one another.
We instead use the incompatibility of different conventions as a measure of diversity [7, 8]. Suppose we already have a set of conventions and we want to add a new convention that is different from all the conventions we have already found. We can measure the reward we would expect to receive if a player from this new convention played with another player from an existing convention, which we refer to as the cross-play reward. If this convention has a high cross-play, adding this convention would be redundant, because a policy from one of the existing conventions would already work with this new convention. On the other hand, a convention that has low cross-play with our set of conventions has a different strategy for the game and should be added to the set, because a human player might use that strategy.
Although directly minimizing cross-play rewards takes us closer to our goal of learning diverse conventions, it can result in poorly behaved conventions due to the adversarial nature of this optimization. In particular, a failure mode we refer to as *handshakes* occurs when policies intentionally sabotage the game upon realizing that the task rewards are inverted. For example, one convention could decide that players should always pick up and place down a plate at the start of the game, which is considered a handshake to establish one's identity without significantly decreasing its self-play reward. If a player 1 from this convention plays with a partner that does not execute the handshake, this player 1 can intentionally sabotage the game and achieve a low cross-play score even if the fundamental strategies are similar. To fix this issue, we introduce mixed-play, a setting where players start with a "mixed-state generation" phase in which they sample actions from self-play or cross-play, but transition into pure self-play at a random timestep with no notice. Since players receive a positive reward during self-play, handshakes cannot reliably indicate whether they should maximize their reward (self-play) or minimize it (cross-play), resulting in conventions that act in good faith.
We propose the CoMeDi algorithm, which stands for "Cross-play optimized, Mixed-play enforced Diversity." CoMeDi generates a sequence of conventions that minimize the cross-play with the most compatible prior conventions in the sequence while using mixed-play to regularize the conventions. We evaluate our method on three environments: Blind Bandits, Balance Beam, and Overcooked [5, 34, 26]. We see that CoMeDi is able to generate diverse conventions in settings where statistical diversity and naive cross-play fail. Finally, we evaluate our generated pool of conventions in a user study by training an agent that is aware of the conventions in the pool and testing it with human users in Overcooked. We find that CoMeDi significantly outperforms the pure self-play and statistical diversity baselines in terms of score and user opinions, surpassing human-level performance.
We model cooperative games as a multiagent Markov decision process [3]. The MDP,  $\mathcal{M}$ , is the tuple  $(S, \mathcal{A}, \mathcal{P}, r, \mathcal{O}, \gamma, T)$ , where  $S$  is the (joint) state space and  $\mathcal{A} = A^1 \times A^2$  is the joint action space. Although we only analyze 2-player games in the main text, we detail an extension to larger teams in Appendix A.5. The transition function,  $\mathcal{P}: S \times \mathcal{A} \times S \to [0,1]$ , is the probability of reaching a state given a current state and joint action. The reward function,  $r: S \times \mathcal{A} \to \mathbb{R}$ , gives a real value reward for each state transition. The observation function,  $\mathcal{O}: S \to O^1 \times O^2$ , generates the player-specific observations from the state. Finally,  $\gamma$  is the reward discount factor and  $T$  is the horizon.
Player  $i$  follows some policy  $\pi^i (a^i\mid o^i)$ . At time  $t$ , the MDP is at state  $s_t\in S$ , so the agents receive the observations  $(o_{t}^{1},o_{t}^{2}) = \mathcal{O}(s_{t})$  and sample an action from their policy,  $a_{t}^{i}\sim \pi^{i}(a_{t}^{i}\mid o_{t}^{i})$ . The environment generates the next time step as  $s_{t + 1}\sim \mathcal{P}(s_t,a_t,s_{t + 1})$  and the shared reward as  $r(s_{t},a_{t})$ . The trajectory is defined as the sequence of states and actions in the environment, which is  $\tau = (s_0,a_0,\dots s_{T - 1},a_{T - 1},s_T)$ . The discounted return for a trajectory is  $R(\tau) = \sum_{t = 0}^{T - 1}\gamma^t r(s_t,a_t)$ . The expected return for a pair of agent-specific policies is  $J(\pi^1,\pi^2) = \mathbb{E}_{\tau \sim (\pi^1,\pi^2)}[R(\tau)]$ . For simplicity of exposition, we assume that the agents in the environment are not ordered, so  $J(\pi^1,\pi^2) = J(\pi^2,\pi^1)$  for all  $\pi^1,\pi^2$ , although our method directly handles situations where order matters.
We refer to an instance of the joint policy of the agents  $\pi = \pi^1 \times \pi^2$  as a convention, following the definition of a convention as an arbitrary solution to a recurring coordination problem in literature [20, 3]. When the joint policy forms a Nash equilibrium, we refer to it as an equilibrium convention, based on some stricter definitions of conventions in literature [13, 19]. In equilibrium conventions, neither player can get a higher expected score with the current partner by following a different policy.
In this paper, we study the problem of generating a diverse set  $D$  of conventions that are representative of the set of potential conventions in the environment at test time,  $D_{\mathrm{test}}$ . To determine how well  $D$  generalizes to the test set, we can calculate a "nearest neighbor" score by finding which convention in  $D$  generates the highest reward for each convention in the test set, which corresponds to  $S$  in Eq. (1):
$$
S (D) = \underset {\pi \in D _ {\text {t e s t}}} {\mathbb {E}} \left[ \max  _ {\pi^ {*} \in D} J \left(\pi^ {*}, \pi\right) \right]. \tag {1}
$$
In practice, we can also treat our generated set  $D$  as the set of training policies used to learn a generalizable agent [21] for zero-shot coordination. We rely on off-the-shelf multitask training routines [6] for learning the generalizable agent, which have shown to be effective in learning an agent that can coordinate with new test partners [27, 21]. In particular, given the set  $D$  of diverse conventions, we can train a convention-aware agent using behavior-cloning [22, 17] in a similar fashion to QDagger [1]:
$$
\mathcal {L} (\hat {\pi}, D) = - \mathcal {J} (\hat {\pi}, \hat {\pi}) - \frac {\lambda}{| D |} \sum_ {\pi \in D} \underset {(o, a) \sim \pi} {\mathbb {E}} [ \log (\hat {\pi} (a | o)) ], \tag {2}
$$
where  $(o, a) \sim \pi$  represents the distribution of observation-action pairs sampled from self-play rollouts of  $\pi$ , and  $\lambda$  is a tunable hyperparameter for the weight of the BC component. At evaluation time, we pair the convention-aware agent  $\hat{\pi}$  with a new test partner and measure the task reward:
$$
\mathcal {J} (\hat {\pi}, D _ {\text {t e s t}}) = \underset {\pi \in D _ {\text {t e s t}}} {\mathbb {E}} [ \mathcal {J} (\hat {\pi}, \pi) ]. \tag {3}
$$
For both Eq. (1) and Eq. (3), the test agent distribution  $D_{\mathrm{test}}$  may be too expensive to sample from during the training loop, e.g., when this test dataset is based on samples from human users. Instead, we would like to generate a high quality set of training agents that is representative of the space of  $D_{\mathrm{test}}$ . Intuitively, a more diverse set  $D$  will improve the generalization of the convention-aware  $\hat{\pi}$  trained on the multi-task objective in Eq. (2).
A common recipe for generating a diverse set  $D$  is to train individual agents via standard self-play, and propose a diversity regularization loss that encourages the set  $D$  to be more "diverse" with respect to the regularization term:
$$
\mathcal {L} (D) = - \sum_ {\pi \in D} [ \mathcal {J} (\pi , \pi) ] + \alpha \mathcal {C} (D). \tag {4}
$$
In Eq. (4), the overall loss of the set  $D$  is a combination of the diversity regularization  $\mathcal{C}$  (scaled by  $\alpha$ ) and the self-play rewards of the individual agents in the set  $D$ . The choice of regularization term  $\mathcal{C}$  requires a difficult balance between tractable computation, alignment to the downstream objective (Eq. (1), Eq. (3)), and ease of optimization of Eq. (4). In Section 2.2 and Section 2.4, we highlight some existing diversity metrics and their limitations.
2.2 Statistical Diversity
The majority of prior works on diversity in deep reinforcement learning are based on the principle of maximizing statistical divergence [11, 10, 21, 9, 15]. In the realm of zero-shot coordination, Trajectory Diversity (TrajeDi) [21] uses the Jensen-Shannon divergence between the trajectories induced by policies as a diversity metric. The method of learning adaptable agent policies (ADAP) [9] maximizes the KL-divergence between the action distributions of two policies over all states. We compare against ADAP as a baseline by adding its diversity term in Eq. (5) to the MAPPO algorithm [36]:
$$
\mathcal {C} _ {\mathrm {A D A P}} (D) = \underset {s \in S} {\mathbb {E}} \left[ \underset {\pi_ {1}, \pi_ {2} \in D} {\mathbb {E}} \exp \left(- D _ {\mathrm {K L}} \left(\pi_ {1} (s) \right\| \pi_ {2} (s)\right) \right]. \tag {5}
$$
While statistical divergence techniques are computationally simple, they do not use the task reward and thus can be poorly aligned with the downstream objectives in Eq. (1) and Eq. (3). For example, consider the introductory Overcooked example in Fig. 1. There are many possible routes that the players can take to complete their individual task, leading to distinct but semantically similar trajectories. Ideally, we want a diversity regularizer to penalize trivial variations that are irrelevant to the task at hand. Unfortunately, statistical diversity would fail to recognize these semantic similarities.
2.3 Cross-Play Minimization Diversity
Recent work on generating strong zero-shot agents in cooperative games have also used the idea of generating a diverse set of conventions by minimizing cross-play. The LIPO algorithm [7] follows a similar approach to our baseline cross-play minimization algorithm described in Section 3.1, but it does not fundamentally tackle the issue of handshakes described in Section 3.2. The ADVERSITY algorithm [8] also follows an approach of minimizing cross-play scores, but addresses the issue of handshakes in Hanabi through a belief reinterpretation process using the fact that it is a partially observable game. Unfortunately, the ADVERSITY algorithm would not function in our fully observable settings since there is no hidden state for belief reinterpretation. We provide a more detailed comparison of our approach to these prior works in Appendix C.
2.4 Other Approaches to Diversity
Other prior works attempt to modify the training domain to induce a variety of conventions [32, 35, 23]. A common aspect to modify is the reward, which can be randomized [32] or shaped by having an external agent try to motivate a certain behavior [35]. This requires extra effort from an environment designer to specify the space of the reward function, which may be impractical in complex environments. Additional approaches using population-based training modify pre-specified aspects of the environment to generate a set of agents [16], or generate agents using different random seeds but require a human to manually select diverse agents [31]. Another set of work focuses on the easier setting of using imitation learning to learn to play with humans [18, 30, 28], but this assumes access to potentially expensive human behavioral data.
We do not explicitly compare CoMeDi to these other approaches in the main text because they assume access to the internal mechanisms of the environment or require external data, so they are not drop-in replacements for self-play, unlike CoMeDi or statistical diversity techniques.
Given the drawbacks of current statistical diversity methods at penalizing semantically similar trajectories, we seek alternative diversity regularization terms that can recognize semantically diverse behavior from trivial variations in trajectory space. A good mechanism for identifying trivial variations
is through task reward: if swapping the behavior of two agents does not affect the attained task reward, then the differences between their behavior are likely irrelevant to the task at hand. For example, consider the effect of the blue player following the red trajectory while the green player follows the orange trajectory in Fig. 1. The swapped teams will still attain the same high reward as the players following the same color trajectories, suggesting that these behaviors are not semantically different.
With this insight, we look to measure cross-play, the task reward attained when pairing together players that were trained separately. We propose a diversity regularization based on minimizing cross-play reward, and describe its benefits compared to statistical diversity. However, pure cross-play faces an adversarial optimization challenge that we refer to as handshakes, making it difficult to optimize. In Section 3.3, we describe mixed-play, our novel technique for mitigating handshakes in cross-play while still retaining the benefits of reward-aligned diversity metrics.
3.1 Cross-Play Minimization
The idea of cross-play is simple: pair two agents from different conventions together and evaluate their reward on the task. If those two conventions are semantically different, then the cross-play reward should be low. On the other hand, if the conventions of the two agents are similar, then their cross-play reward should be high, and it should be penalized by our diversity regularization term.
We can also derive the notion of cross play minimization by trying to maximize the nearest neighbors score  $S(D)$  from Eq. (1). Suppose we already have found  $n - 1$  conventions in the set  $D_{n - 1}$ , and we wish to add a new convention  $\pi_n$  to construct the new set  $D_{n}$ . We can evaluate a lower bound for the improvement of  $S$  as
$$
S \left(D _ {n}\right) \geq S \left(D _ {n - 1}\right) + p \left(\pi_ {n}\right) \left(J \left(\pi_ {n}, \pi_ {n}\right) - \max  _ {\pi^ {*} \in D _ {n - 1}} J \left(\pi_ {n}, \pi^ {*}\right)\right), \tag {6}
$$
where  $p(\pi_n)$  is the probability of choosing policy  $\pi_n$  at test-time. We explore the algorithmic implications of  $p(\pi_n)$  in Section 3.2.
Using the result above, we extend cross-play minimization to a set of  $n$  agents by using the cross-play reward between each convention and the most compatible existing convention as the regularization term  $\mathcal{C}_{\mathrm{X}}$  in Eq. (7). To determine which existing convention is the most compatible, we can simply collect cross-play buffers with all existing conventions and choose the convention with the highest expected return. Note that  $\mathcal{C}_X$  is minimized, so rewards are inverted under cross-play:
$$
\mathcal {C} _ {\mathrm {X}} (D) = \sum_ {i = 2} ^ {n} \max  _ {\pi^ {*} \in D _ {i - 1}} J \left(\pi_ {i}, \pi^ {*}\right). \tag {7}
$$
If we integrate Eq. (7) into the full loss function from Eq. (4), the cross-play loss is simply equal to the sum of the lower-bound expressions in Eq. (6) for all conventions in  $D$  when  $p(\pi_i)$  is ignored.
3.2 Handshakes: Challenges of Optimizing Cross-Play
Recall that the downstream objective is to maximize the nearest neighbors score from Eq. (1), meaning that we want policies that work well with the unseen test distribution. Unfortunately, directly minimizing cross-play can lead to undesirable policies for the downstream objective. In particular, policies optimized with pure cross-play often take illogical actions that would actively harm the task reward when they are in a cross-play setting while still performing well in self-play.
This sabotaging behavior is also observed in concurrent work in Hanabi [8]. They specifically analyze cases where players intentionally use unplayable cards to quickly end the game. Their "self-play worst response" policy, analogous to our pure cross-play minimization strategy for  $\pi_{2}$ , intentionally sabotages the game under cross-play, resulting in a brittle policy that identifies when it is under cross-play. To resolve this issue in Hanabi, ADVERSITY uses belief reinterpretation, similar to Off-Belief Learning [14], in order to construct a fictitious trajectory consistent with the observed trajectory. This prevents sabotaging behavior because observations are equally likely to come from self-play or cross-play. Although this technique is very effective in partially observable environments, like Hanabi, this unfortunately does not solve the issue in our fully observable settings because there is no hidden state that can be reinterpreted to construct plausible self-play trajectories given a cross-play trajectory.
We hypothesize that the challenge with the optimization of cross-play regularization arises when agents can easily discriminate between cross-play and self-play, as illustrated in Fig. 2. For example, consider the earlier Overcooked example in Fig. 1, where we now pair the red convention for the blue player and orange convention for the green player during cross-play. As we repeatedly update the policies based on minimizing cross-play, the agents will learn to recognize when they are being evaluated under cross-play and deliberately sabotage the game. The blue player can learn to develop a handshake, like shown in Fig. 2, and sabotage the game based on the green player's reaction. Note that the strategy after the handshake may be similar across both conventions, but they will still
have a low cross-play due to intentional sabotage. As a result, these agents have fooled our cross-play diversity metric into treating their behaviors as semantically different.
More generally, this problem occurs when there is a mismatch in observation distributions (or history distribution for recurrent policies) between self-play and cross-play, which allows the agents to infer the identity of their partner and the inversion of rewards. In fact, a mismatch can be identified as early as the second timestep: in the first timestep, both agents perform a handshake – an action intended to reveal the identity of the players. If the handshakes match, then the two agents cooperate to solve the task. Otherwise, they sabotage the game, e.g., by doing nothing for the rest of the episode.
Since a strong self-play policy is under-specified at many states, all of the mechanisms of handshake signalling and sabotaging can be encoded into a policy with almost no reduction in self-play performance. We would not expect this type of behavior from a convention in the test set, so this policy,  $\pi$ , would have a very low  $p(\pi)$  in Eq. (6) relative to other trained policies. The outcome of a cross-play minimization procedure (Eq. (7)) plagued by handshakes is a set of agents who have high individual self-play, low pairwise cross-play, yet may still be using semantically similar conventions.
3.3 Mixed-Play