Generating Diverse Cooperative Agents by Learning Incompatible Policies
Authors: Rujikorn Charakorn
Poramate Manoonpong
Nat Dilokthanakul
Venue: ICLR 2023 notable top 25% (2023)
Keywords: multi-agent systems
cooperation
collaboration
reinforcement learning
diversity
robustness

Abstract: Training a robust cooperative agent requires diverse partner agents. However, obtaining those agents is difficult. Previous works aim to learn diverse behaviors by changing the state-action distribution of agents. But, without information about the task's goal, the diversified agents are not guided to find other important, albeit sub-optimal, solutions: the agents might learn only variations of the same solution. In this work, we propose to learn diverse behaviors via policy compatibility. Conceptually, policy compatibility measures whether policies of interest can coordinate effectively. We theoretically show that incompatible policies are not similar. Thus, policy compatibility—which has been used exclusively as a measure of robustness—can be used as a proxy for learning diverse behaviors. Then, we incorporate the proposed objective into a population-based training scheme to allow concurrent training of multiple agents. Additionally, we use state-action information to induce local variations of each policy. Empirically, the proposed method consistently discovers more solutions than baseline methods across various multi-goal cooperative environments. Finally, in multi-recipe Overcooked, we show that our method produces populations of behaviorally diverse agents, which enables generalist agents trained with such a population to be more robust.

See our project page at https://bit.ly/marl-lipo


Published as a conference paper at ICLR 2023
GENERATING DIVERSE COOPERATIVE AGENTS BY LEARNING INCOMPATIBLE POLICIES
Rujikorn Charakorn's Poramate Manoonpong'+ : Nat Dilokthanakul"
VISTEC, Rayong, Thailand SDU, Odense, Denmark 'KMITL, Bangkok, Thailand
(rujikorn.c.sl) poramate wttptnteemeviiiette ac.th, nat.dikkmin ac. th
ABSTRACT
Training a robust cooperative agent requires diverse partner agents. However, obtaining those agents is difficult. Previous works aim to learn diverse behaviors by changing the state-action distribution of agents. But, without information about the task's goal, the diversified agents are not guided to find other important, beit sub-optimal, solutions: the agents might learn only variations of the same solution. In this work, we propose to learn diverse behaviors via policy compatibil- ity. Conceptually, policy compatibility measures whether policies of interest coordinate effectively. We theoretically show that incompatible policies are not similar. Thus, policy compatibility- whiih robustness-Cat can be used as a proxy for learning diverse behaviors. Then, we incorporate the proposed objective into population-based training scheme allow concurrent training of multiple agents. Additionally, we use state-action information induce local variations of each policy. Empirically, the proposed method consistently discovers more solutions than baseline methods across various multi-goal cooperative environments. Finally, multi-recipe Overcooked, we show that our method produces populations of behaviorally diverse agents, which enables generalist agents trained with such population to more robust.
which has been used exclusively as measure
INTRODUCTION
Cooperating with unseen agents (e.g.. humans) in multi-agent systems is challenging problem. Current state-of-the-art cooperative multi-agent reinforcement learning (MARL) techniques can produce highly competent agents in cooperative environments (Kuba et al., 2021; Yu et al., 2021). However, those agents are often overfitted to their training partners and cannot coordinate with unseen agents effectively (Carroll et al., 2019; Bard et al., 2020; Hu et al., 2020; Mahajan et al., 2022).
The problem working with unseen partners, i.e., ad-hoc teamwork problem (Stone et al., 2010), has been tackled in many different ways (Albrecht & Stone, 2018; Carroll et al., 2019; Shih et al., 2020; Gu et al., 2021; Rahman et al., 2021; Zintgraf et al., 2021; He et al., 2022; Mirsky et al., 2022; Parekh al., 2022). These methods allow an agent learn how coordinate with unseen agents and, sometimes, humans. However, the success of these methods depends on the quality of training partners; it has been shown that the diversity of training partners is crucial to the generalization the agent (Charakorn et al., 2021; Knott et al., 2021; Strouse et al., 2021; McKee et al., 2022; Muglich al., 2022). In spite of its importance, obtaining a diverse set of partners still an open problem.
The simplest way to generate training partners is to use hand-crafted policies (Ghosh et al., 2020; Xie et al., 2021: Wang al., 2022), -ppeiii-pecific reward shaping (Leibo al., 2021; Tang et al., 2021; Yu et al., 2023), or multiple runs of the self-play training process (Grover et al., 2018; Strouse et al., 2021). These methods, however, are not scalable nor guaranteed to produce diverse behaviors. Prior works propose techniques aiming to generate diverse agents by changing the state visitation and action distributions (Lucas & Allen, 2022), or joint trajectory distribution et al., 2019; Lupu et al., 2021). However, as discussed by Lupu et al. (2021), there is potential drawback of using such information from trajectories to diversify the behaviors. Specifically, agents that make locally different decisions do not necessarily exhibit different high-level behaviors.
the agents (Mahajan
To avoid this potential pitfall, we propose an alternative approach for learning diverse behaviors using information about the task's 's objective via the expected return. In contrast previous works that

---

Published as a conference paper at ICLR 2023
use joint trajectory distribution represent behavior, we use policy compatibility instead. Because cooperative environments commonly require all agents to coordinate on the same solution, if the agents have learned different solutions, they cannot coordinate effectively and, thus, are incompatible. Consequently, if an agent discovers a solution that is incompatible with all other agents in a population, then the solution must be unique relative to the population. Based this reasoning, simple but effective training objective that regularizes agents in a population to find solutions that are compatible with their partner agents but incompatible with others in the population. We call this method " "Learning Incompatible Policies" (LIPO).
introduce
We theoretically show that optimizing the proposed objective will yield a distinct policy. Then, we extend the objective to population-based training scheme that allows concurrent training of multiple policies. Additionally, we utilize a mutual information (MI) objective to diversify local behaviors of each policy. Empirically, without using any domain knowledge, LIPO can discover more solutions than previous methods under various multi-goal settings. To further study the effectiveness of LIPO in complex environment, we present multi-recipe variant of Overcooked and show that LIPO produces behaviorally diverse agents that prefer to complete different cooking recipes. Experimental results across three environments suggest that LIPO is robust to the state and action spaces, the reward structure, and the number of possible solutions. Finally, we find that training generalist agents with diverse population produced by LIPO yields more robust agents than training with less diverse baseline population. See our project page at https: /bitt . 1y/mart -1ipp
2 PRELIMINARIES
Our main focus lies in fully cooperative environments modeled decentralized partially observ- able Markov decision processes (Dec-POMDP, Bernstein et al. (2002)). In this work, we start our investigation in the two-player variant. A two-player Dec-POMDP is defined by tuple (S,A', "TT "., H), where is the state space, A = A'' x A² and Ω = L2' x Ω2 are the joint-action and joint-observation spaces of player 1 and player 2. The transition probability from state to 8' after taking a joint action (a² ,a²) is given by T(s' (s, a² a²). O(a², (os)) is the conditional probability of observing joint observation (o o²²) under state s. All players share a common reward function r(s, a a ), Y is the reward discount factor and H is the horizon length.
Players, with potentially different observation and action spaces, are controlled by policy n² and n² At each timestep t, the players observe 0g oZ) ~ O(o), s3))s) under state E S and produce a joint action a; (o) af) A sampled from the joint policy I (ac(T)) = ' (xe (a]lr?). where 7f and 7² contain a trajectory history until timestep 1 from the perspective of each agent. All players receive a shared reward V'z = r((e, at, az). The return of a joint trajectory
)x²
(oo, do, TH OH) E T = (Ω x A x R)" can be written as G(T) o r'r,. The expected return of a joint policy (T', (T² is J(x m²)- = Ex- G(T), where p( x²) is the distribution over trajectories of the joint policy (TT ²)) and P(T T²) is the probability of being
sampled from a joint policy (,²,m²)
We use subscripts to denote different joint policies and superscripts to refer to different player roles. For example, πA = (=), m)) is a different joint policy from Tg = (Th mi)), and F and T are policies of different roles. Finally, we denote the expected joint return of self-play (SP) trajectories where both policies are part of the same joint policy, π as ( == J[ = and the expected joint return of cross-play (XP) trajectories- -where policies are chosen from different joint policies, मI and - Jxp(T TB) == [[m]: m + J[T] 7))-
Since we are interested in creating distinct policies for any Dec-POMDP, we need an environment- agnostic measure that captures the similarity of policies. First, we consider a measure that can compute the similarity between policies of the same role i, e.g., TT and qंp. We can measure this with the probability of joint trajectory 7 produced by either T TB.. However, in the two-player setting, we need to pair these policies with reference policy T Specifically, nT and r'z are considered similar if they are likely to produce the same trajectories when paired with an arbitrary reference policy - We define similar policies as follows:
NNtt that LIPO can be applied to environments with more than two players with a slight modification. Specif- ically, a policy x² would represent the joint policy of all players except player i, x (af ) M x (af (f))
2

---

Published as a conference paper at ICLR 2023
Definition 2.1 (Similar policies). Considering two policies of the same role i, IT and T'y and reference policy &ie of different role T'A is similar to q up to if and only if max,eT [1 P(rim' m B
S E, where E
Next, we consider an alternate view on assessing the similarity between policies using policy compatibility (Section 3). Policy compatibility measures the performance difference of joint policy лв before and after one of its policies T' is substituted by another policy A'A We define compatibility between policy ='A and a joint policy ти as follows:
Definition 2.2 (Compatible policies). Given a policy T'A and joint policy ma, T'A is compatible with n if and only J T 2 (1 (TB))
3 LEARNING INCOMPATIBLE POLICIES (LIPO)
Our goal is to create distinct policies and, therefore, population of diverse agents. First, we theoretically show that policy compatibility can be used to identify whether two policies are different. Based on this observation, we propose a novel training objective that produces a distinct policy. Then, extend this objective for training population of diverse policies. Finally, incorporate an MI objective that encourages each policy to learn local variations.
3.1 LEARNING A DISTINCT POLICY VIA POLICY COMPATIBILITY
In this section, we motivate our objective by looking at two joint policies:
High return
( A: T and n = (m). = The goal is for л to learn different behavior from лn via the compati- bility criterion. Importantly, the compatibility criterion can be computed empirically without direct access to the trajectory distribution, which be difficult estimate. Under mild assump- tions, we can simplify the setting such that simple relationship between similarity measure and compatibility criterion emerges. By reason- ing about the expected return under different pairs of policies, we derive our main result.
T
A
A
T
A
High return
Similar
Low return
Dissimlor
(T
TE
2 T
(a)
(b)
(c)
Figure (a) The objective of A (Eq. I) relation to лT. (b, c) Conceptual illustration of Theorem 3.1 and Corollary 3.2. Solid lines represent given relationships, and dotted lines represent implied relationships.
Theorem 3.1. If A is similar to Th then T compatible with πg. (The proof is in App. A.) Corollary 3.2. If v is not compatible with TB then T'A not similar to T
The result from Corollary 3.2 shows that we can find a policy = that not similar to Tt by decreasing its compatibility with TB until they are incompatible, i.e., (( Th) < (1 JSp((( Additionally, can ensure that RA learns meaningful solution by maximizing (T). Assuming that Tg has learned a solution and is fixed, the optimization objective of TA can be written as
max (SP(TA) subject to A(TAA: TT)) < (1 - -)BBP(TT) Vi,j E (1,2),
(I)
A way to solve such constrained problem is to convert the constraints into regularization terms. For simplicity, we use common Axp > as a hyperparameter for the constraints. Then, we can write the soft objective of Eq. 1 as
max JSP(TTA (XP.)X(TTA.TB) NA
3.2 LEARNING A POPULATION OF DIVERSE POLICIES
To create a population of diverse policies. P = {TAIL < A A N). we need an objective that requires each member of the population to have different behavior relative to the rest of the population. We can write such an objective by expanding the XP term in Eq. (2) to include all other policies in the population. Additionally, we relax the assumption that other policies are fixed to
3

---

Published as a conference paper at ICLR 2023
allow concurrent training of all policies. For a policy Na E P. with an aggregation function fagg. its objective becomes
max LIIOO TA, A. P) = (s)(TT) - AxP-Ix((TA,
(3)
where JXP(TA, P) = Inee(
(4)
BA" = (T. TB E P-Ah
(5)
P-A PTTTA}
(6)
While using the average operation as the aggregation function is plausible, we find that using the max operation helps stabilize the training process and produces more diverse policies. We suspect that the average operation might produce many conflicting gradients and does not prioritize compatible XP
pairs. We refer to JIPPO as the compatibility gup between a policy म and population P.
We can see that the compatibility gap objective only uses the expected return (Jsp and JxP) and is insensitive to the state and action information. We argue that this distinction between LIPO and previous methods helps the agents discover more solutions in various situations (Sec. 4.1 and 4.5).
3.3 INDUCING VARIATIONS EACH POLICY
It is important to note that, regardless of the population size, there could be policies of role i that are compatible with A d but not similar to A We consider those policies to variations of A and propose capture such variations via an MI objective. Specifically, we condition o on latent variable zf such that n has the form of π (alT) (e ..,) ( (a 10),,,=)=z0 =))= (=)
where p(z!, 22) is pre-defined prior distribution. We can induce variations of T'A by maximizing I([o', af): ); where I(::-) is the MI between two random variables. Intuitively, this objective encourages each policy to observe different observations and perform different actions given different values of the latent variable. However, maximizing I(fo', a 2²) directly intractable, instead we optimize the variational lower bound of the MI (Jordan et al., 1999) (see App. B for the derivation)
T( T(fo', a =²);²) 2 H(²²) + Egg (o (o at) [log da (o'o'"aa))
(7)
where 40 (z'lo', "a) a²) is an approximation of the true posterior p(z'lo', a²) parameterized by d So. maximizing I(fo', a! 21) and I(fo?, 2?)
is an optimization problem that can be written as
max RA
H(²²) Esi (ot, log q (2 (e'lo',aa)
8
In the previous work (Mahajan al., 2019), shared 2 (i.e., 2?) used allowing both policies to collectively switch between different modes of behavior. However, LIPO uses independently sampled z as it utilizes for a different purpose. Specifically, LIPO maximizes JI to learn diverse solutions and optimizes the MI objective to learn variations of each solution. That is, the MI objective does not directly impact the diversity between different policies but increases variations of each individual policy. We note that the MI objective is optional; show that without the MI objective, LIPO still produces diverse policies (Sec. 4.4).
3.4 IMPLEMENTATION
In practice, we modify the MI objective (Eq. 8) be differentiable with respect to the policy T' Specifically, the variational posterior 4oa is modified such that, instead of a sampled action a', it takes the action distribution T (+lo",2²) MI-based approaches (Eysenbach et al., 2018; Sharma et al., 2019; Jiang Lu, 2021; Lucas & Allen, 2022), we can optimize I(fo', at 2f) directly without computing auxiliary reward (Mahajan et al., 2019; Osa et al., 2022). The loss function of the modified MI objective is
input, i.e., dd (=|o, T (-lo", z²)). In contrast to previous
2 Ext an log qo (2+10', T 2²))
L AA. 6A) =
Σ
(9)
2 1=1
The objective of a policy TA in a population P becomes
max TA.
(IA, (MAL
(10)
4

---

Published as a conference paper at ICLR 2023
We set discrete variable and use the uniform distribution for p(z') and p(=²). At the beginning of each episode, each policy is given an independently sampled & that will be used until the end of the episode. We use MAPPO (Yu et al., 2021) for maximizing Jsp and minimizing Jxp. More details, including the pseudocode and the extension to more than two players, can be found in App. D.
4 EXPERIMENTS
We study the effectivvness of LIPO under three multi-goal cooperative environments in which both players must collectively choose to accomplish one of the available goals. We evaluate the diversity of a population based on the number of distinct goals achieved. We compare LIPO to other cooperative MARL methods that do not require domain knowledge to generate diverse agents. Our baselines are as follows: (i) Multi SP (multiple runs of self-play). (ii) SP (A single run of SP with added MI objective), (iii) MAVEN (Mahajan et al., 2019), and (iv) TrajeDi (Lupu et al., 2021). We also use
Multi SP and Multi MAVEN as baselines by training SPMI and MAVEN multiple times. We also discuss on methods that utilize domain knowledge in Sec. 5.
4.1 DISCOVERING DIVERSE SOLUTIONS
We use two simple environments to study the effectiveness of various methods in discovering solutions: (i) One-Step Cooperative Matrix Game (CMG), in which there are many possible solutions, and (ii) Point Mass Rendezvous (PMR), a temporally extended cooperative navigation environment.
One-Step Cooperative Matrix Game (CMG): A game of CMG is defined by a tuple (M, (kim), J: frm F), where M is the number of solutions. For m (1,..., M), km is the number of compatible actions and r' is the reward of a solution m. By choosing the same solution, both players get a reward I' associated with the chosen solution. We consider two setups of CMG: sub-optimal (CMG-S) and hard-to-fing find (CMG-H), For CMG-S, we set (M = 32, ki = 8, Im 0.5 (1+ #=1 )) which causes each solution to have a different reward, ranging from 0.5 to 1. For CMG-H, we use (M = 32, km m, I'm = 1), which makes solutions with smaller number of compatible actions harder to be found by random exploration. An example payoff matrix is shown in Fig. 2a.
Point Mass Rendezvous (PMR): The environment is based on the Multi-Agent Particle Environment (Lowe et al., 2017; Terry et al., 2020). The goal of this envi- ronment is for the two agents to navigate to a landmark together. There are M
O
0
0
3
landmarks, and we consider each landmark as solution in this environment. This envi- ronment has two modes: PMR-C and PMR-L. In PMR-C landmarks are distributed evenly on the circumference of a circle. Thus, all landmarks are equally easy to find and optimal. In PMR-L, landmarks are placed on line. In this scenario, closer landmarks are easier to find.
0 0
(a)
(b)
(c)
Figure 2: (a) The payoff matrix of
a CMG game with (M = 3, km
m, I' = m). (b, c) The agents (orange) and landmark positions (blue) of PMR-C and PMR-L.
We define the population size |P| of each method fol- lows: For SP MI and MAVEN, [P] is equal to the number of dimensions of the latent variable, | For Multi MI and Multi MAVEN, [P] |=| nssed where Nesed is the number of random seeds and we use |2| = 8. For Multi SP, TrajeDi, and LIPO, IP] is the number of joint policies in the population.
Results: Fig. 3 shows the numbers of learned solutions, averaged over three runs. In all environments, LIPO consistently discovers more solutions than the baselines, given the same population size. The baselines find fewer solutions in CMG-H and PMR-L than they do in CMG-S and PMR-C, whereas LIPO performs similarly across settings. LIPO is also better than the baselines at finding sub-optimal solutions in CMG-S. We note that Multi SP and TrajeDi perform almost ideally in PMR-C, where all solutions are equivalent, but perform worse in other settings. Also, Multi SPMI finds all four solutions in PMR when the population size is bigger than However, it performs poorly in CMG. LIPO's consistency across environments and settings demonstrates that LIPO is still effective when (i) many solutions exist, (ii) solutions are not equally optimal, and (iii) solutions are not equally likely to be found by random exploration. We also have experimented with stronger regularization coefficients for the baselines, which help the baselines discover more solutions. However, if the regularization coefficient is too large, they fail to produce capable policies.
5

---

Published as a conference paper at ICLR 2023
MAVEN LIO
--0.5 ideal
SPM
TrajeDi
Multi MAVEN
32
Discovered solutions
Competent agents
0-
16
37
64
8
1.
32
Population size
Population size
Population size (a) PMR-C
(a) CMG-S
(b) CMG-H
Discovered solutions
Computent agents ntst
12
16 Population size
16 Populetion size
32
Population size
(c) PMR-C
(d) PMR-L
(b) PMR-L
Figure 3: Numbers of discovered solutions. Ideally, if the population size increases by one, one more solution should be discovered, as depicted by the dashed lines (assuming that joint policy does not produce multi-modal behavior).
Figure 4: Numbers of compe- tent joint policies using vari- ous combinations of N (X-axis) and Axp (colors) in PMR.
4.2 TRADE-OFF BETWEEN COMPETENCY AND DISSIMILARITY OF JOINT POLICIES
is possible that optimizing a regularized objective might incur training instability and create incapable policies. Here, investigate the effect of different combinations of Xxp and the population size (N) on the competency of the policies.
Fig. shows the number of competent joint policies when using different values of N and Axp in PMR. Particularly, in PMR, a joint policy is considered competent when both players stay close to landmark the end of an episode. We observe that when the population size is larger than the number of solutions (N > M), some surplus policies do not learn to reach goal. Importantly, the number of competent joint policies depends on the value of Axp: lower values of Axp yield more capable policies. However, using too low Axp will generate policies that share a common solution when S MM as shown in App. J.1.1. Additionally, when N < M, all trained agents are competent except when Axp is too high in PMR-L. These results suggest that there is trade-off between the number of capable joint policies and policy dissimilarity. When using a larger population size, a small Axp should be used to avoid producing incompetent agents, while bigger Axp should be used with smaller population sizes to ensure the dissimilarity between joint policies.
4.3 TRADE-OFF BETWERN COMPUTATION COST AND DIVERSITY
Not only is using bigger values of N more likely to produce incompetent policies, but it is also computationally expen- sive. Formally, the computation complexity of approximating Jxe(-, P) is O(Nasp) where nzp is the number of XP pairs used to approximate JXP(TA, P). So, we investigate way reduce the cost of calculating JXP(TA, P) by reducing nap- According to Eq. 5, the default value is Rzp = N - 1. When
NN
Discovered Poovereedoo conventions
--
PMPPR -MR-
2
Nap < N 1, Map policies are chosen randomly from P-a by sampling without replacement.
Cross-play pairs per iteration
We observe that, while being computationally cheaper, using Nap < N tends to produce less diverse populations as
Figure 5: Number of learned solu- tions using various 72pp-
shown in Fig. 5. Thus, Nep can be considered a hyperparameter trade-off. However, shown by the dashed lines, the effect
that controls the crmpputatiio-iiversiii of nap on population diversity is less prominent in PMR-C, where solutions are equally likely to be found. We use nep N in all other experiments. See App. J.1.2 for results in CMG.

---

Published as a conference paper at ICLR 2023
4.4 EFFECT OF THE MUTUAL INFORMATION OBJECTIVE
(a) Amn 0.5
(b) AMHI 0.5
(c) AM 0.5
(d) Am 0.5
(c) Asu 0
(f) Amn 0
(g) Asu
(h) AMI 0
Figure 6: The top and bottom rows show four joint policies produced by a single run of LIPO training with and without the MI objectíve, respectively. Different colors of the trajectories correspond to different values of z. The orange and green circles show the starting positions. The blue circles represent the landmarks.
Fig. 6 shows the behaviors of the policies produced by LIPO with and without the MI objective in PMR-C. We can see the effect of the MI objective in the variety of the trajectories. Overall, each agent exhibits larger variations given small MI regularization Ami 0.5. This result aligns with our motivation of using the MI objective to learn variations of each solution. With or without the MI regularization, LIPO discovers all the landmarks with N 4.
4.5 DISCOVERING RECIPES IN MULTI-RECIPE OVERCOOKED
Overcooked, a collaborative cooking game, has been used to study the cooperative ability of learned agents in prior works (Carroll al., 2019; Charakorn et al., 2020; Strouse et al., 2021; McKee al., 2022). To in- vestigate the usefulness of LIPO in a high-dimensional environment, we implement more complex version of the game based on the work of Wu
Chapped
Plagere
Torato
Chopping
Delivery
Carnot
Lettics Salad
Torata carree Salad
Couster
Plate
al. (2021) I); players have to complete and serve one of the six pre-defined recipes fast possible, opposed to delivering a single menu item re- peatedly. We emphasize that this environment is much more challenging than the ones in the previous experiments because of various aspects: First, it has a sparse reward signal. Second, there are multiple sub-tasks. Third, different recipes have different sub-tasks. Each of these characteristics of the environment complicates the process of finding diverse solutions. Furthermore, we note that recipes containing carrot or a tomato are harder to complete than other recipes as they involve an additional coordination step. Particularly, carrot and tomato have to be sent over by the agent on the right, unlike lettuce and onion. Fig. shows an overview of the game.
(a) sample initial state
(b) Possible recipes
Figure 7: An overview of the multi-recipe Overcooked game.
The goal in this experiment is to learn population of behaviorally diverse agents. We choose to quantify the diversity of population based on the entropy of its recipe distribution. For a population
Σ. m;(AA) Er m.(x]]
P. we approximate the probability recipe i being completed as P(recipe,
where m,(TT, denotes the frequency of recipe under a joint policy ~A. The recipe frequencies, (m,(TA)]1 < é 6), for each joint policy TA P are measured by counting the completed recipes from 1,000 self-play episodes. For Multi SP. TrajeDi and LIPO, we set N 8. For Multi SP MI and
Multi MAVEN, Nsseed 8 and |z| 8.
7

---

Published as a conference paper at ICLR 2023
Recipe probabiitt
Reetpe probebllii
Lettvo Owds
Sonate Canar
Tonats
Tethoon
Ove Tanaro
Carier
Toraao
0106
Tonars Ianoto
(a) Multi
(b) Multi SP
(c) Multi MAVEN
0.5
Recipe probabikky
0.5
0.5
probabiltty
0.5
04
04
04
Lethuce
Caroo
Lettuca
Camot
Lettuce
(d) TrajeDi
(c) LIPO (AMr 0.5)
(f) LIPO (AMT
Figure 8: Recipe distributions of generated populations compared to the uniform distribution (dashed
line). We provide reference for the uniform distribution as it has the highest entropy.
Table 1: The means and standard errors entropy of approximated population recipe distributions.
Multi SP
TrajeDi
Multi SPai
Multi MAVEN
LIPO (AMr = 0.5) LIPO (Aur = 0)
1.16 0.03 0.98 + 0.17 1.43 0.05
1,53 0.07
1.58 0.07
1.26 0.17
Results: Quantitatively, Tab. 1 shows that LIPO has the highest population recipe distribution entropy, averaged over five random seeds. This result indicates that LIPO populations all recipes more uniformly than the baseline populations, even though some recipes take longer to complete or are harder to find by random exploration. The recipe distribution of populations produced by each method can be found in Fig. 8. We find that LIPO populations with λMI = 0 still, similar to Ami 0.5, consistently learn to use the hard-to-find Tomato & Carrot Sa lad recipe. However, the frequencies of Chopped Tomato and Chopped Carrot are lowered (Fig. 8f). This means that there are multiple joint policies that learn to complete the same recipe while being incompatible with each other. We suspect that using AMT > 0 alleviates this problem by regularizing each joint policy to represent policy with broader state-action coverage (e.g.. learn multiple ways of completing recipe), indirectly pushing other joint policies to different recipes in order to be incompatible. Qualitatively, we can see in App. J.2 that the baselines produce agents with similar recipe frequencies. The resulting populations, thus, contain agents with similar recipe preference. In contrast, LIPO produces agents with distinct recipe frequencies, collectively making the population more diverse than the baselines. We also visualize the behaviors learned by LIPO in App. J.3.
4.6 TRAINING GENERALIST AGENTS WITH GENERATED POPULATIONS
In addition to evaluating the diversity of agents in Overcooked, quantify the usefulness of the produced agents by using them as training partners of a generalist agent and test the agent with held-out populations. Intuitively, more diverse training partners would enable the agent to generalize and coordinate with unseen agents better. Additionally, we include a population of six specialized SP policies where each policy is trained complete a specific recipe by adjusting the reward function. The specialist population is created for evaluating the agent when the partner has strong preference.
Fig. 9a shows that all generalist agents perform similarly when tested with held-out baseline populations. However, the agents trained with baseline population perform poorly when matched with held-out L.IPO and specialist populations. In contrast, those trained with LIPPO population perform better in both situations. Specifically, they have significantly higher success rate when paired with the specialist with strong preference for Tomato & Carrot Salad as shown in Fig. 9b. We attribute the success rate difference to the fact that this recipe has lower completion probability in all except LIPO populations. As a result, generalist agents trained with a LIPO population perform better in terms of the overall success rate when tested with specialist agents. Overall, training with a LIPO population helps the generalist agents to better coordinate with more partner types indicated by the harmonic means.
8

---

Published as a conference paper at ICLR 2023
Test populattio
spectalist
09.00
M220
79.80
Teenato
Tamaeto
Harrice
Traech
5P
12.25)
12.001
00.01
00151
901-4
12112
X0.10
18.10
3:141
Training popustion
Toge0k
13.50
3.87
(2.0)
1.57
10012
0.99
(2.00)
2:56
S0.01
0.51
0220
0.57
(2.23
2.55
[000]
0.06
9004
0.33
12
0:40
10/02
0.33
10:22)
0.59
30.01
0.34
R001
0.47
08
07
Malu
3.88
3.64
1.00
0.45
0.63
2:33
6:90
0.22
2:44
0:31
0:78
0:53
0.51
0%p
13.22)
13
10002
03:00)
0.22
(1011
88
10.00
1:30
31.00
30.0g
3.54
3.8
0.99
8:59
0:43
0.81
8:79
0.16
0.22
8:41
0:33
0.67
827
0.00
13.00
30.00
03.01
0000
01//2
18:10
0.22
2.2
0.23
3:28
DAY
0.58
0.24
8:23
0.47
0.28
0.00
(1)
(h)
Figure 9: The mean success rates and standard errors (in parentheses) of trained generalist agents when matched with unseen test agents. Each generalist agent is trained with only one population and evaluated with all test partners, each with 300 episodes. The result in each row is averaged over five generalist agents trained with an independently generated population from the corresponding method. A held-out population from each method is used as test population. The right-most column shows the harmonic mean of the success rate of all held-out populations (a) and specialist agents (b).
5 RELATED WORK
Learning a collection of diverse agents has been utilized in various contexts (Parker-Holder et al., 2020; Sun et al., 2020; Zahavy al., 2021: Zhou et al., 2021). In the cooperative domain, Canaan et al. (2019; 2020) use the Quality Diversity (QD) algorithm (Mouret & Clune, 2015; Pugh et al., 2016) to produce a population of behaviorally diverse agents. QD, however, requires domain knowledge to encode different types of behaviors. For example, in CMG, the algorithm requires the mapping between actions and corresponding solutions. In Overcooked, needs to know all possible recipes beforehand. Without such a domain knowledge, it would be difficult to use QD to produce a diverse population. TrajeDi (Lupu et al., 2021) produces a diverse population of agents based on the trajectory distribution. Finally, MEP (Zhao et al., 2021) trains population of agents with an auxiliary reward based on population entropy. Like TrajeDi and MEP, ours does not require Nomain-specifie knowledge. However, promote behavioral diversity, LIPO utilizes the expected returns of different policy pairs as opposed to state-action information.
The idea of diversifying the empirical return has been explored in the context of finding diverse solutions in non-transitive competitive games (Liu et al., 2021; Balduzzi et al., 2019; Perez-Nieves al., 2021). In particular, Liu et al. (2021) share some similar ideas with our work. They propose to use the expected returns, when encountering different opponents, and state-action information to promote diversity of agents. A concurrent work by Rahman et al. (2022) applies a similar idea of diversifying the expected joint return to generate diverse partners in cooperative settings. LIPO can be thought of as a special case designed specifically for cooperative environments (see App. E).
MI objectives have been used in RL to Iearn diverse behaviors (Eysenbach et al., 2018; Sharma et al., 2019; Kumar et al., 2020; Osa al., 2022). In cooperative MARL, MAVEN (Mahajan et al., 2019) optimizes both RL and MI objectives to encourage the agents to explore in committed manner and discover diverse solutions. Also, Any-play (Lucas & Allen, 2022) uses a similar objective to produce training partners with many solutions for generalist agent. In contrast, our approach uses the MI objective to regularize each policy to learn local variations of each solution.
6 CONCLUSION
We propose LIPO, simple and generic method that can create a population of diverse agents in cooperative multi-agent environments. Unlike previous work that uses state-action information from joint trajectories, LIPO utilizes the concept of policy compatibility to create diverse policies. This alternative view of quantifying diversity makes LIPO more robust to state and action spaces. Also, LIPO uses the MI objective to learn local variations of each solution. Empirically, LIPO consistently produces more diverse populations than the baselines across a variety of three multi goal environments. Finally, in multi-recip Overcooked, LIPO produces populations of diverse partners that help the generalist agents to generalize to unseen agents better. We include further discussions and limitations of LIPO in App. F and G.

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENT
This work partially supported by King Mongkut's Institute of Technology Ladkrabang [2566-02- 06-002]. We thank Natchaya Sricom for drawing Fig. and 7. We thank Supasorn Suwajanakorn, Sucha Supitayaponnpong and Maytus Piriyajitakonijj for their suggestions on early draft versions. We also thank anonymous reviewers for their constructive feedbacks.
REPRODUCIBILITY STATEMENT
We have include additional information to reproduce the experimental results in the supplementary text:
Environment details (App. C)
Pscudocode and implementation details (App. D)
Hyperparameters used in all experiments (App. H and I)
The source code is available at httpp:///gtthab..cm//6666/ mariiipporr-io.....oo
REFERENCES
Stefano V Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66-99 2018.
David Balduzzi, Marta Gamelo, Yoram Bachrach, Wojciech Czamnekii, Julien Perolat, Max Jader- berg, and Thore Graepel. Open-ended learning in symmetric zero-sum games. In International Conference on Machine Learning, pp. 434-443, PMLR, 2019.
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Marc Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280:103216 2020.
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27)) 319-840, 2002.
Rodrigo Canaan, Julian Togelius, Andy Nealen, and Stefan Menzel. Diverse agents for ad-hoc cooperation in hanabi. In 20/9 IEEE Conference on Games (CoG), pp. 1-8. IEER, 2019.
Rodrigo Canaan, Xianbo Gao, Julian Togelius, Andy Nealen, and Stefan Menzel. Generating and adapting to diverse ad-hoc cooperation agents in hanabi. arXiv preprint axXiv:2004.13710, 2020.
Micah Carroll, Rohin Shah, Mark K Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32. 2019.
Rujikom Charakom, Poramate Manoonpong, and Nat Dilokthanakul. Investigating partner diversifica- tion methods in cooperative multi-agent deep reinforcement learning. In International Conference on Newral Information Processing, pp. 302-402. Springer, 2020.
Rujikom Charakorn, Poramate Manoonpong, and Nat Dilokthanakul. Learning to cooperate with unseen agents through mete-reiinoorcement learning. In Proceedings of the 20th International Conference Autonomous Agents and MultiAgent Systems, pp. 1478-1479, 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity all you need: Learning skills without a reward function. In International Conference on Learning Representations, 2018.
Ahana Ghosh, Sebastian Tschiatschek, Hamed Mahdavi, and Adish Singla. Towards deployment of robust cooperative ai agents: An algorithmic framework for learning adaptive policies. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pp. 447-455, 2020.
10

---

Published as a conference paper at ICLR 2023
Aditya Grover, Maruan Al-Shedivat, Jayesh Gupta, Yuri Burda, and Harrison Edwards. Learning policy representations multiagent systems. In International conference on machine learning, pp. 1802-1811. 1. PMLR, 2018
Pengjie Gu, Mengchen Zhao, Jianye Hao, and Bo An. Online ad hoc teamwork under partial observability, In International Conference on Learning Representations, 2021.
Jerry Zhi- Yang He, Zackory Erickson, Daniel $ Brown, Aditi Raghunathan, and Anca Dragan. Learning representations that enable generalization in assistive tasks. In 6th Annual Conference on Robot Learning, 2022.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "other-play" for zero-shot coordination. In International Conference on Machine Learning, pp.
PMLR, 2020.
Jiechuan Jiang and Zongqing Lu. The emergence of individuality. In International Conference on Machine Learning, pp. 4992-5001. PMLR, 2021.
Michael I Jordan, Zoubin Ghahramani, Tommi S Jaakkola, and Lawrence K Saul. An introduction to variational methods for graphical models. Machine learning, 37(2):183-233, 1999.
Paul Knott, Micah Carroll, Sam Devlin, Kamil Ciosek, Katja Hofmann, Anca Dragan, and Rohin Shah. Evaluating the robustness of collaborative agents. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '21, pp. 1560-1562, Rich- land, SC, 2021. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978 1503833333.
Jakub Grudzien Kuba, Ruiqing Chen, Munning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforcement learning. arXiv preprint arXiv:2109.11251, 2021.
Saurabh Kumar, Aviral Kumar, Sergey Levine, and Chelsea Finn. One solution is not all you need: Few-shot extrapolation via structured maxent rl. Advances Neural Information Processing Systems, 33:8198 188899882101 2020.
Joel 7. Leibo, Edgar A Dueñez-Guzman, Alexander Vezhnevets, John P Agapiou, Peter Sunehag, Raphael Koster, Jayd Matyas, Charlie Beattie, Igor Mordatch, and Thore Graepel. Scalable evaluation of multi-agent reinforcement learning with melting pot. In International Conference Machine Learning, pp. 6187-6199, PMLR, 2021.
Xiangyu Liu, Hangtian Jia, Ying Wen, Yaodong Yang. Yujing Hu, Yingfeng Chen, Changjie Fan, and Zhipeng Hu. Towards unifying behavioral and response diversity for open-ended learning in zero-sum games. Advances in Neural Information Processing Systems, 34, 2021.
Ryan Lowe, Yi Wu, Aviy Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed cooperative-commertiive environments. Advances in neural information processing systems, 30, 2017.
Keane Lucas and Ross E Allen. Any-play: An intrinsic augmentation for zero-shot coordination. arXiv preprint 2rXiv:2201.1242 2022.
Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In International Conference on Machine Learning, pp. 7204-7213, PMLR, 2021.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. Advances in Neural Information Processing Systems, pp. 7611-7622, 2019.
Anuj Mahajan, Mikayel Samvelyan, Tarun Gupta, Benjamin Ellis, Mingfei Sun, Tim Rocktäschel, and Shimon Whiteson. Generalization in cooperative multi-agent systems. arXiv preprint 2rXiv:2202.00104 2022.
Kevin R McKee, Joel Z Leibo, Charlie Beattie, and Richard Everett. Quantifying the effects of environment and population diversity in multi-agent reinforcement learning. Autonomous Agents and Multi-Agent Systems, 36(1):1-16, 2022.
11

---

Published as a conference paper at ICLR 2023
Reuth Mirsky, Ignacio Carlucho, Arrasy Rahman, Elliot Fosong, William Macke, Mohan Sridharan, Peter Stone, and Stefano V Albrecht. A survey of ad hoc teamwork research. In European Conference Multi-Agent Systems, pp. 273-293. Springer, 2022.
Baptistttite Mouret and Jeff Clune. Illuminating search spaces by mapping elites. arXiv preprint arXiv:1504.04909 2015.
Darius Muglich, Luisa M Zintgraf, Christian A Schroeder De Witt, Shimon Whiteson, and Jakob Foerster. Generalized beliefs for cooperative ai. In International Conference Machine Learning, pp. 16062-16082. PMLR, 2022.
Takayuki Osa, Voot Tangkaratt, and Masashi Sugiyama. Discovering diverse solutions in deep reinforcement learning by maximizing state-action-based mutual information. Neural Networks, 2022.
Sagar Parekh, Soheil Habibian, and Dylan P Losey. Rili: Robustly influencing latent intent. arXiv preprint arXiv:2203. 12705, 2022.
Jack Parker-Holder, Aldo Pacchiano, Krzysztof M Choromanski, and Stephen J Roberts. Effective diversity in population based reinforcement learning. Advances in Neural Information Processing Systems, 33:18050-18000 2020.
Nicolas Perez-Nieves, Yaodong Yang, Oliver Slumbers, David H Mguni, Ying Wen, and Jun Wang. Modelling behavioural diversity for learning in open-ended games. In International Conference on Machine Learning, pp. 85 14--455. PMLR, 2021.
Justin K Pugh, Lisa B Soros, and Kenneth O Stanley. Quality diversity: A new frontier for evolutionary computation. Frontiers Robotics and AI, 3:40, 2016.
Arrasy Rahman, Elliot Fosong, Ignacio Carlucho, and Stefano V Albrecht. Towards robust ad hoc teamwork agents by creating diverse training teammates. arXiv preprint arXiv:2207. 14/0., 2022.
Muhammad A Rahman, Niklas Hopner, Filippos Christianos, and Stefano V Albrecht. Towards open ad hoc teamwork using graph- policy learning. In International Conference on Machine Learning, pp. 8776-877 PMLR, 2021.
Archit Sharma, Shixiang Gu, Sergey Levine, Vikash Kumar, and Karol Hausman. Dynamics-awar unsupervised discovery of skills. arXiv preprint arXiv: 1907,0167, 2019.
Andy Shih, Arjun Sawhney, Jovana Kondic, Stefano Ermon, and Dorsa Sadigh. On the critical role of conventions in adaptive human-ai collaboration. In International Conference on Learning Representations, 2020.
Peter Stone, Gal A Kaminka, Sarit Kraus, and Jeffrey S Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. Twenty-Fourth AAAI Conference on Artificial Intelligence, 2010.
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34: 14502-14515, 2021.
Hao Sun, Zhenghao Peng, Bo Dai, Jian Guo, Dahua Lin, and Bolei Zhou. Novel policy seeking with constrained optimization. arXiv preprint arXiv:2005. 2020.
Zhenggang Tang, Chao Yu, Boyuan Chen, Huazhe Xu, Xiaolong Wang, Fei Fang, Simon Du, Yu Wang, and Yi Wu. Discovering diverse multi-agent strategic behavior via reward randomization. arXiv preprint ar arXiv:2103.04560 2021.
K Terry, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sulivan, Luis Santos, Rodrigo Perez, Caroline Horsch, Clemens Dieffendahl, Niall L Williams, Yashas Lokesh, Ryan Sullivan, and Praveen Ravi. Pettingzoo: Gym for multi-agent reinforcement learning. arXiv preprint a0Xiv:2009. 14471, 2020.
12

---

Published as a conference paper at ICLR 2023
Woodrow Zhouyuan Wang, Andy Shih, Annie Xie, and Dorsa Sadigh. Influencing towards stable multi-agent interactions. In Conference on Robot Learning, pp. 1132-1143. PMLR, 2022.
Sarah A. Wu, Rose E. Wang, James A. Evans, Joshua B. Tenenbaum, David C. Parkes, and Max Kleiman-Weiner. Too many cooks: Coordinating multi-agent collaboration through inverse planning. Topics in Cognitive Science, n/a(n/a), 2021. doi: https::/doi..rr/10000 www.o//.///....1.. 12525. URL https: https///onnineee www.inneibbarryy www.bbrrr..wwlee.e.............e com/ doi abs 10.
12525.
Annie Xie, Dylan Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent representa- tions to influence multi-agent interaction. In Conference on Robot Learning, pp. 575-588, PMLR, 2021.
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of mappo in cooperative, multi-agent games. arXiv preprint arXiv:2 2021.
Chao Yu, Jiaxuan Gao, Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang, and Yi Wu. Learning zero-shot cooperation with humans, assuming humans are biased. In The Eleventh International Conference on Learning Representations, 2023. URL https: http://pppp/eeeeeeee net/ forum?
id=TrwE81 9aJzs.
Tom Zahavy, Brendan O' D'Donoghue, Andre Barreto, Sebastian Flennerhag, Volodymyr Mnih, and Satinder Singh. Discovering diverse nearly optimal policies with successor features. In ICML 202/ Workshop on Unsupervised Reinforcement Learning. 2021.
Rui Zhao, Jinming Song, Hu Haifeng, Yang Gao, Yi Wu, Zhongqian Sun, and Yang Wei. Max- imum entropy population based training for zero-shot human-ai coordination. arXiv preprint 11X11:2112.11711, 2021.
Zihan Zhou, Wei Fu, Bingliang Zhang, and Yi Wu. Continuously discovering novel strategies via reward-switching policy optimization. In International Conference on Learning Representations, 2021.
Luisa Zintgraf, Sam Devlin, Kamil Ciosek, Shimon Whiteson, and Katja Hofmann. Deep interactive bayesian reinforcement learning via meta-learning- In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pp. 1712-1714, 2021.
13

---

Published as a conference paper at ICLR 2023
A PROOF FOR THEOREM 3.1
We prove the relationship of similar policies and compatible policies in Theorem 3.1 under the following assumptions.
Assumption A.1 (All joint trajectories are supported by TB). P( (1))BT))))))))))))))) > 0; VT E T
Assumption A.2 (Shared e). A common - 6 S is used for Def. 2.1 and 2.2 Assumption A.3 (Positive return). G(T) > 0; ET Theorem. If A is similar to TB. then n'A is compatible with TB.
P(r Because T is similar to Tyz. we know that 1 S 1+ Vr importance sampling write J( T3)) in relation to
Proof. Let r(t) p + from Def. 2.1. Consequently, J x3z):
J(T xb) E, =b) G(T)
1. P(T TB)(()
I. P(r|TT Tod P( T (())
P(TITA
B B G(T)G(T)
Because 1 - S r(7) < + and G(T) > VT E T, the expected return J (B) has the following upper and lower bounds:
(1 T) Top (p) G(T) SJ T S (1
(~p(sg T G(r) G(T)
(-c))sP(TB) <J (T) T S (1 + (TBB)
This means that v4 is compatible with T8 (Def. 2.2). If IT's is similar to r'g. then IT' is compatible with TB.
Remark: Assumption A.3 can be satisfied by offsetting the joint return of all trajectories such that min, G(r) > O. However, since we use MAPPO as the base algorithm, the expected return is subtracted by baseline to compute the advantage during the policy update, which removes the effect of the offset. In practice, even under environments that do not satisfy G(T) > 0, LIPO can still discover diverse solutions effectively, shown in the experiments.
B DERIVATION OF THE LOWER BOUND OF THE MI OBJECTIVE
We provide derivation of Eq. here. Let p(z')(o', af}) be the true posterior of 2 and qo be the approximation of p parameterized by c. The lower bound [(fo, at af)))) can be derived follows:
H(2²) H(='lHo',a'))
-H(z') + Egr ) [log pog (oooo'"""" a'1)l
= -H(2") + E [logpp> (0 log (2) + loggo(>")(oo a'))l
= -H(=') + +E ot [ (o', (it (=)l)a(
T((o', (((0))a');=)) 2 H==²) EE o [lo ,,, (2 ( ())]
14

---

Published as a conference paper at ICLR 2023
C ADDITIONAL ENVIRONMENT DETAILS
C.I ONE-STEP COOPERATIVE MATRIX GAME
game of CMG is defined by tuple (M, {kmm:: (rm]), where M is the number of solutions. For m E {l...., ........, J. km is the number of compatible actions and I'm is the reward of a solution m. The game is stateless and terminate immediately after both players simultancously choose an action. By choosing the same solution, both players get reward associated with the chosen solution. This means that the solutions are not equally optimal if the values {rm } are not identical. Similarly, the values in {km} } are not identical then the solutions are not equally likely to be chosen by uniform joint policy. We consider two setups of CMG: sub-optimal (CMG-S) and hard-to-find (CMG-H).
For CMG-S, we set (M = 32, kim = 8, T' 0.5 * (1 + w-))). which causes each solution to have different reward, ranging from 0.5 to 1. There are 32 solutions, each with 8 compatible actions. There are 32 256 possible actions for each player.
For CMG-H, we use (M- = 32, km m, I' 1), which makes solutions with smaller number of compatible actions harder be found by random exploration. There are 32 solutions, each with different number of compatible actions, ranging from to 32. The number of available actions for each player is
7h 32
528.
mm=
C.2 POINT MASS RENDEZVOUS (PMR)
PMR is based on the the Multi-Agent Particle Environment (Lowe et al., 2017: Terry et al., 2020). The observation of each agent includes absolute position, current velocity, and the relative distance to the landmarks and the other agent. These features are concatenated as a 1-D vector of length 14, The possible actions are: no op, move, {up, down, left, right). }. In PMR-C, the start positions of the agents are (00.3.0) (-0.3,0) and the landmarks positions are {(1.59, 1.59), (1.59, -1.59), (-1.59, 1.59), (-1.59, -1.59)). For PMR-L, the start and the landmark positions are ((1.)).(0,)) and
An episode will be terminated after 50 timesteps. The agents are incentivized to go to the same landmark and stay close together with the reward function
rt = d(p', c)
min d(I,c).. c), LEL
where d((,.)) .) is the euclidean distance between two points, p' is the 2-d coordinate of agent i, c is the average coordinate of all agents, and the set of all landmarks.
C.3 MULTI- OVERCOOKED
We implement multi-recipe of the game based on the work of Wu et al. (2021). In this version of Overcooked, there are four ingredients: lettuce, onion, tomato, and carrot. The ingredients are randomly placed at pre-defined positions in the layout. Particularly, the lettuce and the onion randomly placed on the left or the middle counter. The tomato and the carrot are randomly placed on the right or the middle counter. These ingredients can be composed into different recipes making each ingredient unique: four recipes (LettuceSalas) ChoppedOnion on) require only single ingredient, while the other two (TomatoLettuceallee Tomat oCaarotttalleee
ComatoSala,, ChoppedCarrot,
require two ingredients. The ingredients have to be chopped at the chop- ping station before placing on the plate. After the required ingredients are put on the plate, they must be delivered to the delivery station.
Both players have the same egocentric observation and action spaces. The observation is a set of hand-crafted features that represent a local view of the environment. Specifically, we use the following features: absolute position and facing direction, relative distance to the objects and the other agent, state of the ingredients, four booleans indicating if the agent is next to counter in four cardinal positions, currently held items, the state of the held foods, and the type and state of the items in front of the agent. These features concatenated as 1-D vector of length 54. At every timestep, each player has to choose one of the six possible actions: no op, move {up, down, left, right] }. and interact.
An episode lasts at most 200 timesteps and terminates immediately after a successful delivery. An episode without delivery is considered unsuccessful. We incentivize the agents to interact with the
15

---

Published as a conference paper at ICLR 2023
objects and deliver as fast possible with the following reward function:
Ft
Finteract
progress
Tzomplete
P.
here Pinteract shaped reward given when an agent interacts with an object for the first time in an episode, "progres is given when the players progress toward recipe completion (i.c., chopping required ingredients or putting chopped ingredients on the plate), Tcumplete is given upon successful delivery, and p is a penalty. We use Tinteruct 0.5, pprugress 1.0, cccmpllte 10, and = 0.1. We note that recipes with more than one ingredient will give only slightly higher rewards (Finteract + prrogress) but are significantly harder to be discovered by random exploration than those with one ingredient.
Additional experimental details: For specialist agents, the rewards are given when interacting, progressing, or completing a specific recipe. For the held-out populations, we remove incompetent policies with the expected return of less than zero from the test populations created by TrajeDi and LIPO. We do not remove those in the training populations. We do this because testing with an incompetent policy does not give any meaningful information, almost all episodes will be unsuccessful.
D IMPLEMENTATION DETAILS
Algorithm 1: Training process of LIPO (on-policy)
This pseudocode based on self-play. highlighted in green.
text is related the MI objective. LIPO specific code
Input: Population p= f < A < N}. the number of XP pairs used approximate (mp( A, P) (Rep)) number of players in an episode (m4), and number SP XP episodes per iteration (Esr
and Exr).
while not done do
for A [I, do
B c- GGeeEppsooeeollloutttTAaaaa TA, Ese, m)
Compute a using B"
8 < GeiCroosPlyyollloostta. P, Raps Exp; m)
Compute Jxr(T A P) using 8 (Eq. 4)
Compute Taa using F0 B (Eq. 9)
0 + A Veal txp
64 A AMn Vog
Algorithm 2: Rollout collection functions
Function Det.pisodeRRe louts (TA, TB, F)
B + 0
for E [L...., m) do
for episode E {I,.., E do
" ~ p(z 2m
x111)2/)) ley (11,, (1) ~~ (1),)))))))) m 21)
20)
B - uU (r)
return 8
Function SetCroossPayyRllouss (T P, nap, Exp)
B + }
p' A SampeeWihorteepaacmeeetP
Bap)
for TB P' do
B - Get.ctEppiin -=Gttppsoddecloottt (TTA,
Exp
PLA!
E R UB
return R
The pseudocode for LIPO is shown in Algorithm and 2. If there are more than two players (m > 2). I would represent the joint policy of all players except player i, x (a |T!) II T (af (zt). We note that scaling LIPO to more than two players does not increase the training time. It is the same the two-player setting as long the numbers of SP and XP episodes the same. In practice
16

---

Published as a conference paper at ICLR 2023
Table 2: Hyperparameters used by the MAPPO algorithm.
Hyperparameters
Value
0.003 (CMG and PMR)
Learning rate
0.005 (Overcooked)
100 (CMG)
Batch size
2,500 (PMR) 10,000 (Overcooked)
10 (CMG and PMR) 15 (Overcooked)
Epochs
2 (CMG and PMR) (Overcooked)
Number of mini-batches
0.0 (CMG) 0.03 (PMR and Overcooked)
Entropy coefficient
Discount factor (?)
0.99
GAE lambda
0.95
Value loss coefficient
0,5
PPO clipping parameter
0,3
Gradient clipping
0.5
Adam epsilon
le-5
however, more XP episodes might be needed to better estimate Jxp. We use the parameter sharing technique for better sample efficiency and faster convergence (Tan, 1993; Foerster et al., 2018; Rashid et al., 2018) 8). Assuming that policy ' is neural network parameterized by 0¹ this means that for a joint policy (T),, 7) we have 0) = 6%. Still, T and 7 can behave differently as they observe different parts of the environment and have different player indicator concatenated with their local observations.
All methods implemented on top of MAPPO except MAVEN. The critic, policy, and discriminator are
neural networks with two hidden layers, each having 64 units. For a fair comparison, use the same more environment steps in the policy update of the baselines compared to LIPO. Common hyperparameters of methods based on MAPPO are shown in Table. 2.
D.1 MAPPO
MAPPO the base MARL algorithm for all baselines except MAVEN. The policy parameters are shared among all policies. The critic takes a state of the environment and outputs an expected return of given global state. The global state is provided by the environment and only used during training. For the complete training objectives of MAPPO, we refer the reader to Appendix of Yu al. (2021).
D.2 MULTI SP
A simple but effective way produce diverse agents by training multiple SP agents with different neural network initializations and random seeds. Specifically, each run produces a joint policy πa that maximizes Jsp(TA) using MAPPO.
D.3 SPMI
A single run of SP agent trained with added MI objective I(2]o', a'). SP shared for both policies and considers each z as a different joint policy. We train SPar using the same training
procedure as LIPO by setting N 1, Axp and 21 22. The discriminator takes local
17

---

Published as a conference paper at ICLR 2023
observation o' and action distribution (-lo") inputs and outputs the discrete probability of the latent variable. The latent variable of all policies is shared during an episode.
D.4 MAVEN
MAVEN (Mahajan al., 2019) explicitly designed for learning diverse solutions in cooperative multi-agent environments. A joint policy is represented as (-|I,, 2), and each mode of behavior is represented by the latent variable 2. Similar to SPMI MAVEN uses a shared for all policies. We use the same network architecture presented in Mahajan et al. (2019) with recurrent neural networks. However, we do not use the hierarchical policy but sample from the uniform distribution. The latent variable of all policies is shared.
D.5 MULTI SPMI AND MULTI MAVEN
population containing joint policies from multiple runs of SPMI and MAVEN. Like Multi SP, each run has different neural network initializations and random seeds. The population size is [P] = ) where seed is the number of runs. Notably, this baseline uses the training data differently from the base algorithms. Instead of training long single run, this approach allows the policy to "restart" by using different initialization of neural networks. For example, training single run with |2| Need might not discover as many solutions as training Uleed runs with |a| Reed even though the population size the same. Empirically, we find that multiple shorter runs can find more solutions compared single long run of the corresponding algorithm. Thus, omit the results of the base algorithms in multi-recips Overcooked.
D.6 TRAJEDI
TrajedDi produces a population of diverse agents that also maximize the expected return in cooperative environments. The diversity measure of this method is based the -hannnnnn divergence (ISD) between the trajectory distribution of each policy. Different from the original implementation. remove the best-response (BR) policy from the population. Since the BR policy might work well with only subset of solutions, removing BR potentially increase the number of variations in the population. Our modified loss is:
C Σ (JsP(TA)) + (TS AN)],
(11)
where JSD the proposed diversity objective of TrajeDi, and a and are the hyperparameters of TrajeDi.
D.7 LIPO
LIPO uses the same implementation as SP except LIPO uses independent latent variable for each policy and Axp 0. Additionally, LIPO uses extra crities for the XP trajectories. In total, LIPO has an SP critic sp and N XP critics {V/A.-T Tg E P-A}- In each training iteration, LIPO collects SP and XP trajectories of all policy combinations. MAPPO used for both maximizing JsP and minimizing Jxp. The critics are trained using SP trajectories and all of XP trajectories, while the policy is trained using SP trajectories and XP trajectories from the XP pair that has the highest joint return, max(
D.8 GENERALIST AGENT
The policy and critic networks of a generalist agent use two 256-unit GRU layers (Cho et al., 2014) followed by a linear layer. The input also includes the reward and action of the previous timestep. We use MAPPO for training generalist agent. We train both the policy and critic with the batch size of 320,000 timesteps using truncated backpropagation through time (BPTT). The samples are reused for 15 epochs. Each minibatch contains 1,600 sequences with maximum length of 50 timesteps. We also use learning rate annealing, specifically the generalist agent starting from 0 005 to 0. 003 with linear scheduling. Other hyperparameters shared with other methods (Tab. 2). A training
18

---

Published as a conference paper at ICLR 2023
partner for generalist agent is sampled uniformly from the training population at the beginning of an episode.
E RELATIONSHIP WITH RAHMAN ET AL. (2022)
Rahman et al. (2022) propose to optimize the self play returns while maximizing the diversity term Div(C), where C is N x N cross-play payoff matrix. Specifically, they propose to learn diverse policies via the following objective:
max Tr(C) + Div(C)
(12)
C
They propose to use Div(C)= = (Ct( where x(C) is an N x N matrix with Ki. (C)( being similarity between policy π. and т.. The radial basis function (RBF) kernel of the empirical returns used to measure the similarity between two policies:
(C) exp( -
(13)
r²
where Ct is row 7th of C. In other words, C. is the vector containing the empirical return of æ; when matched with other policies the population. Intuitively, this objective diversifies the policies via the expected returns similar to LIPO. Using the same notation of the cross-play matrix, we can write the objective of training a LIPO population as:
max Tr(C)) - Axp Σ max Ct3 C
(14)
18JEN
This objective wants the diagonal (Jsp) to be maximized and the off off- dfagona entries of the cross-play matrix (Jxp) to be minimized. This is special case of Eq. 12 where Div(C") is based on policy compatibility.
F DISCUSSIONS
Agents trained with LIPO incentivized to act adversarially toward agents that behave differently from itself. This behavior might not be desirable for certain downstream tasks. For example, agents produced by LIPO might not be suitable for interacting with humans as they would refuse conform with the user. However, as shown in Sec. 4.6, training generalist agent with these agents would have the opposite effect: the generalist agent would try to comply with the current partner's preference.
LIPO produces a population of netrmoptimal solutions, generalist agent trained with a LIPO population might not coordinate well with significantly sub-optimal agents. In a prior work, Strouse et al. (2021) show that augmenting the training population with past checkpoints (FCP) helps the trained generalist agent to effectively coordinate with sub-optimal agents. Since LIPO and FCP are orthogonal, populations created by LIPO can be also augmented in the same way as FCP.
Previous works find incompatible policies to undesirable since they are generally results of coordinated symmetry breaking (Bard et al., 2020; Hu et al., 2020; )021); these policies perform poorly when interacting with unseen partners. However, we show that learning incompatible policies can be useful for generating behaviorally diverse agents in various scenarios. The produced agents can then be used as training partners for generalist agent. Using LIPO in environments where many solutions equivalent may produce such undesirable that LIPO can be combined with other techniques, e.g., other-play (Hu et al., 2020) and equivariant coordinator (Muglich et al., 2022), to avoid learning arbitrary symmetry-breaking We leave the study of the combination of LIPO and these techniques for future work. A concurrent work by Cui al. (2023) proposes an extension of LIPO by combining insights from off- learning (Hu et al., 2021) to avoid "sabotaging" behavior of LIPO agents.
-breakkn conventions. We believe
G LIMITATIONS
LIPO requires an additional hyperparameter Axp. Axp is too big, it is possible that the main RL objective, Jsp, would be interfered which will result in incompetent joint policy (Sec. 4.2). An
19

---

Published as a conference paper at ICLR 2023
adaptive mechanism that selects suitable value for Axp at different stages of training could help increase training stability.
Although LIPO can be fully parallelized, it requires more computation than the baselines to get accurate approximation of Jxp, which makes it harder to scale up to bigger population sizes. Instead of collecting all policy pairs, sampling a portion of policy pairs to approximate Jxp could reduce computation cost and training time at a potential cost of diversity (Sec. 4.3). Instead using a uniform sampling, mechanism that selects the best pair to sample (e.g., bandit algorithm) might help mitigate the diversity loss from using lower Nap-
In this work, only investigate the effectiveness of LIPO under ments with multiple discrete solutions. Investigating LIPO in other well-knows environments where the solution space might be continuous, e.g., continuous control (Peng et al., 2021), or environments with only a single goal, e.g.. SMAC (Samvelyan et al., 2019), might yield interesting results and pose different challenges. We leave this further investigation as our future work.
set of cooperative environ-
H HYPERPARAMETERS (CMG AND PMR)
We provide the searched values each method Tab. 3, 4, 5, 6, and 8. The hyperparameters searched individually for each population size. We use three random seeds for each set of hyperparameters. We do not use any validation method. Instead, we present the results using the best parameters in the main paper. For LIPO, set AMı 0.0 and 0.5 in CMG and PMR, respectively.
Table 3: The values of Ası used by SP in all environments, The best values shown in bold.
Population size
Hyperparameters
Values (CMG-S)
Values (CMG-H)
Values (PMR-C)
Values (PMR-L)
[0.5.1.5,10]
[0.5.1.5,10]
2
[0.5,1.5.10]
[0.5.1.5.10]
[0.5.1.5.10]
[0.5.1.5.10]
Asa
[1,5,00500]
[1,5,10,50]
[0.5.1.5,10]
[0.5.1,5,10]
16
[1.5.10.50)
[1.5.10.50)
[1,5,10,50
[1,5,10,50|
64
[1.5,10.50]
[1.5.10.50)
Table 4: The values of AMI used by MAVEN in all environments.
The best values are shown in bold.
Population size
Hyperparameters
Values (CMG-S) (CMG-H)
Values (PMR-C)
Values (PMR-L)
[1.5.10.50]
[1.5.10.50]
[1.5,10.50]
[1.5.10.50]
[0,5,10.50]
[1,5,10,50]
AMa
[1.5,10.50]
[1.5.10.50]
[1,5,
[1.5.10.50]
16
[1.5.10.50]
[1.5.10.50]
[1.5.10.50]
[1.5.10.50]
[1,5,10.50]
[1,5,10,50]
Table The values of AMI used by Multi SP in all environments. The best values are shown in bold.
Population size
Hyperparameters
Values (CMG-S)
Values (CMG-H)
Values (PMR-C)
Values (PMR-L)
2
8
Asn
[1.5,10.50]
[1.5.10.50]
[0,5,10.50]
[1,5.10.50]
16
[1,5,10,50)
[1,5,10,50]
[0,5,10.50]
11,5,10,50]
[1.5,10.50)
[1,5.10.50]
[1,5,00.50]
[1,5,10.50]
64
[1.5,10.50]
[1.5.10.50]
20

---

Published as a conference paper at ICLR 2023
Table 6: bold.
The values of λMT used by Multi MAVEN in all environments. The best values are shown in
Population size
Hyperparameters
Values (CMG-S)
Values (CMG-H)
Values (PMR-C)
Values (PMR-L)
Ava
[1.5.10.50]
[1.5.10.50]
[1,5.10.50]
51.5.10.50]
16
[1,5,10.50]
[1,5,10,50]
[0,5,10,50]
[1,5,10,50]
[1,5,10,50]
[1,5,10.50]
[1,5,10,50]
11,5,10,50]
[1.5,10,50]
[1,5.10.50]
Table 7: bold.
The values of a and used by TrajDi in all environments. The best values are shown in
Population size
Hyperparameters
Values (CMG-S)
Values (CMG-H)
Values (PMR-C)
Values (PMR-L)
[1,5.10.50]
[0,5,10,50]
[1,5,10,50]
[1.5.10.50]
[1.5. 10,50
[1,5,10,50]
[0.0...05.0..0..1
[0.01.0.05.0....2]
[1,5,10.50]
[1.5.10.50]
16
[0.01.0.05.0.1.0)
10.01.0.05.0.1.... 1.0.2
(0.01.0.05.0.1.0.
(0.01 1.02..
(0.01.000.00..0.2]
(0.01.0.05.0....22.
[0, 0.1, 0.5]
[0, 0.1,
2
[0. 0.1. 0.5]
[0, 0.1, 0.5]
[0. 0.5]
10, 0.1, 0.5]
7
0
10, 0.5]
[0, 0.1, 0.5]
16
0
64
Table 8: The values Axp used by LIPO in all environments. The best values are shown in bold.
Population size
Hyperparameters
Values (CMG-S)
Values (CMG-H)
Values (PMR-C)
Values (PMR-L)
10.10..2.00.51]
[0.1.0.25.0.5.] [0.1.0.25.0.5.]]
[0.1.0.2.00.5..
10..2..5.
[0.1...25.0.5.]] [0.1.0.25.0.5.]]
Axp
005.1]]
[0.5.1]
[0. 1002..0.)).551
[0.5.1]
[0.5.]]
32
[0.5,1]
[0.5.1]
[0.5.1]
[0.5.1]
I HYPERPARAMETERS (OVERCOOKED)
For each method, the parameters that give the highest entropy generate five populations for Sec. 4.5, and Sec. 4.6. The searched values each method are:
TrajeDi: We perform grid search with following hyperparameters: ? E {0, 0.5). We use a = 5 and 7y = 0.5 for the results in the paper.
{5, 10} and
{5, 10). We use AMI for the results in Multi SPMI: We perform grid search λvι
the paper.
Multi MAVEN: We perform grid search of Am E (5, 10). use AMI 5 for the results in the paper.
LIPO: We perform grid search with following hyperparameters: Axp E {0.2, 0.3] and AmI E {0.1,0.5), We Axp 0.3 and AvT 0.5 for the results in the paper.
J ADDITIONAL RESULTS
J.1 ADDITIONAL ABLATION RESULTS
We provide additional results numbers learned solutions with varying N and Axp (Fig. 10), numbers of competent agents in CMG with varying N and Axp (Fig. 11), and numbers of learned solutions with varying Ryp in CMG (Fig. 12) here. These results are consistent with the analysis presented in Sec. 4.2 and 4.3.
21

---

Published as a conference paper at ICLR 2023
J.1.1
ADDITIONAL RESULTS WITH VARYING N AND XXP
++
Ideal
-deaa
-.055
0.5
32
32
Discovered solutions
Discovered solutions
8
16
8
16
32
Population size
Population size
(a) CMG-S
(b) CMG-H
Ideal
Ideal
0.5 0.1
0.5 - 0.1
Discovered solutions
Discovered solutions
2
Population size
Population size
(c) PMR-C
(d) PMR-L
Figure 10: Numbers of learned solutions with different values of Xap
Ideal
- 0.5
- 0.5
Competent agents
Competent
32
32
16
16
32
64
32
64
Population size
Population size
(a) CMG-S
(b) CMG-H
Figure 11: Numbers of competent agents using various combinations N (x-axis) and Axp (colors) in CMG.
J.1.2 ADDITIONAL RESULTS WITH VARYING 1sp
CMGS -MGHH
Discovered conventions
16 Cross-play pairs per iteration
32
Figure 12: Numbers of learned solutions with varying Nopp- The result is consistent with the results in PMR provided the main text: The lower Rep is, the fewer solutions are learned.
22

---

Published as a conference paper at ICLR 2023
J.2
RADAR PLOTS OF RECIPE FREQUENCIES
Chopped Carrot
Chopped
Chopped Carrot
Chopped
Tomato
Tomato
Chopped
Chopped Lettuce
Chopped
Onion
Onion
Lettuce
Tomato
Tomato
Tomato
Tomato
Lettuce Salad
Lettuce
Salad
Salad
Salad
(a) Multi
(b) Multi SPMI
Chopped
Chopped
Chopped
Chopped
Carrot
Tomato
Carrot
Tomato
Chopped
Chopped
Chopped
Chapped
Lettuce
Carrot
Lettuce
Carrot
(c) Multi MAVEN
(d) TrajeDi
Chopped
Chopped
Chopped Carrot
Chopped
Carrot
Tomato
Tomato
Chopped
Chopped Lettuce
Chopped
Chopped
Onion
Onion
Lettuce
Tomato
Tomato
Tomato
Tomato
Lettuce
Lettuce
Salad
Salad
Salad
Salad
(e) LIPO
LIPO (AMr =
Figure 13: Recipe frequencies of generated populations. Each colored line represents recipe frequen- cies of an agent in a population. Red thick lines connect the highest frequency each recipe. The outer ring represents the frequency of 1,000.
Fig. 13 shows recipe frequencies of the population with highest recipe entropy (out of five runs) from each method. The frequencies are calculated based on completed recipe from 1,000 self-play episodes of each agent described in Sec. 4.5. Qualitatively,
that agents in LIPOO
23

---

Published as a conference paper at ICLR 2023
population have more distinct recipe frequencies. That is, agents are different from each other in terms of recipe preference.
J.3 VISUALIZATION OF BEHAVIORS
We visualize the behaviors joint policies produced by LIPO in PMR and Overcooked https //sites. google. com/vvivviooooooo www/ww/rwwppppeoeeeeeeeeeeeeeeee 2023. Here, we show snapshots of four joint policies that have distinct recipe preference Overcooked.
Recipe preference: Tomato
Carrot Salad
(b) Recipe preference: Chopped Lettuce
(c) Recipe preference: Chopped Onion
(d) Recipe preference: Tomato
Lettuce Salad
Figure 14: Four joint policies from population of eight joint policies produced by single run of LIPO. Each row shows snapshots of joint policy illustrating a distinct recipe preference.
24

---

Published as a conference paper at ICLR 2023
REFERENCES
Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, H Francis Song, Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280: 103216, 2020.
Kyunghyun Cho, Bart Van Merriënboer, Dzmitry Bahdanau, and Yoshua Bengio. On the properties of neural machine translation: Encoder-decoder approaches. arXiv preprini arXiv: 409.,259, 2014.
Brandon Cui, Andrei Lupu, Samuel Sokota, Hengyuan Hu, David J Wu, and Jakob Nicolaus Foerster. Adversarial diversity in hanabi. In International Conference on Learning Representations, 2023.
URL httpt:tt/////c http://pppeeeeeeeeeeeeeeeee net /wtt/e///oreeeeeeeeeeeeeeeeeeeee forum?
Jakob Foerster, Gregory Farquhar, Triantafyllos
Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "otther-pay" for zero-shot coordination. In International Conference on Machine Learning, pp. 4399-4410, PMLR, 2020.
Hengyuan Hu, Adam Lerer, Brandon Cui, Luis Pineda, Noam Brown, and Jakob Foerster. Off-belief learning. In International Conference Machine Learning, pp. 4369-4379. PMLR, 2021.
Ryan Lowe, Yi I Wu, Aviv Tamar, Jean Harb, OpenAl Pieter Abbeel, and Igor Mordatch. Multi-agent actor-critic for mixed coopeeiive environments. Advances in neural information processing systems, 30, 2017.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. In Advances in Neural Information Processing Systems, pp. 7611-611111 7611-7622, 2019.
Darius Muglich, Christian Schroeder de Witt, Elise van der Pol, Shimon Whiteson, and Jakob Foerster. Equivariant networks for zero-shot coordination. arXiv preprint arXiv:2210.12124, 2022.
Bei Peng, Tabish Rashid, Christian Schroeder de Witt, Alexae-Ae Kamienny, Philip Torr, Wendelin Böhmer, and Shimon Whiteson. Faemac: Factored multi-agent centralised policy gradients. Advances in Neural Information Processing Systems, 34, 2021.
Arrasy Rahman, Elliot Fosong, Ignacio Carlucho, and Stefano Albrecht. Towards robust ad hoc teamwork agents by creating diverse training teammates, arXiv preprint 2rXiv:2207.14133 2022.
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shi- mon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 4295-4304. PMLR, 2018.
Mikayel Samvelyan, Tabish Rashid, Christian Schroeder De Witt, Gregory Farquhar, Nantas Nardelli, Tim GJ Rudner, Chia-Man Hung, Philip HS Torr, Jakob Foerster, and Shimon Whiteson. The starcraft multi-agent challenge. arXiv preprint arXiv: / 1902.04043, 2019.
DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34: 14502-14515, 2021.
Ming Tan. Multi-agent reinforcement learning: Independent vs. cooperative agents. In Proceedings of the tenth international conference on machine learning, pp. 330-337, 1993.
J. K Temr, Benjamin Black, Nathaniel Grammel, Mario Jayakumar, Ananth Hari, Ryan Sulivan, Luis Santos, Rodrigo Perez, Caroline Horsch, Clemens Dieffendahl, Niall L Williams, Yashas Lokesh, Ryan Sullivan, and Praveen Ravi. Pettingzoo: Gym for multi-agent reinforcement learning. arXiv preprint X009 /4471, 2020.
Sarah A. WU, Rose E. Wang, James A. Evans, Joshua B. Tenenbaum, David C. Parkes, and Max Kleiman- Weiner. Too many cooks: Coordinating multi-agent collaboration through inverse planning. Topics in Cognitive Science, n/a(n/a), 2021. doi: http::/ddoi.org//.1111oooooo... URL ntttss://onl inel ibrrrr.rr..r wi ley. com htmm//////b/////// 11100.1111111 12525.
25

---

Published as a conference paper at ICLR 2023
Chao Yu, Akash Velu, Eugene Vinitsky, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of mappo in cooperative, multi ageett games. arXiv preprint arX1v:1003.01955, 2021.
26


Reviews:
[{"review_id": "n2Pm9pp02Lz", "reviewer": "Reviewer_CTS6", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper presents the LIPO framework for learning diverse sets of policies for cooperative multi-agent reinforcement learning tasks.  Given an existing population consisting of tuples of joint policies, the LIPO objective maximizes the performance of a new joint policy, while penalizing it for performing well when paired with other policies within the population.  LIPO also, optionally, maximizes the diversity of the trajectories generated by an individual team of policies using policies parameterized by a randomly sampled latent variable, and an additional a mutual information objective.\n\nExperiments demonstrate that LIPO is better than several recently developed alternatives at identifying the space of solutions to cooperative games, including simple matrix games, the cooperative particle environments, and the much more complex Overcooked! environment.  They demonstrate that LIPO is able to more reliably identify the full set of possible solutions, rather than just a subset of these solutions.  The also evaluate \"generalist\" policies trained against the population of policies generated with LIPO (or one of the baseline methods).  They demonstrate that policies trained against the LIPO population were significantly more robust (compared to the baseline methods) in terms of their performance when paired with members of a \"test\" populations.", "summary_of_the_review": "The decision to accept is based on the empirical success of the proposed LIPO framework in training policies for ad hoc cooperative settings.  The method also represents something of a departure from existing approaches to learning diverse policies, which previously have focused on maximizing direct measures of the diversity of the policy space.", "strength_and_weaknesses": "The main strength of this paper is the intuitive nature of the LIPO approach, and the empirically demonstrated performance in both solution enumeration and as the basis for training generalist cooperative policies.\n\nOne potential weakness of LIPO, and related approaches, is the fact that they focus on enumerating the space of near-optimal solutions.  This means that generalist policies trained against these populations may perform poorly when paired with significantly sub-optimal partners.  In real-world settings, using LIPO-like methods could therefore lead to policies that are not robust to realistic partners who are unlikely to behave in a truly optimal way.\n\nAnother potential weakness is that there is no guarantee that the space of behaviors represented by a single policy parameterized by a latent state will represent a unique class of solutions.  Without the ability to recognize unique solutions, LIPO may not be able to recognize solutions that are unlikely to occur in a ad hoc setting (since other agents would not choose strategies that are likely to mis-coordinate).", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ahfsU7_qii", "reviewer": "Reviewer_obTi", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work introduces LIPO (Learning Incompatible Policies), a MARL method that uses a policy compatibility-based objective and a mutual information regularizer in order to learn a diverse set of behaviours in cooperative tasks. LIPO is evaluated in a wide set of environments and shows a superior capacity to discover policies against the compared baselines. Additionally, the work also includes a comprehensive study of the parameter values and their effect on the learned behaviours.", "summary_of_the_review": "All in all, I think this is an interesting and well-presented contribution.", "strength_and_weaknesses": "Strength:\n- Diverse of benchmarks and baselines\n- Comprehensive study of the impact of the parameters\n- Ample discussion of results, as well as limitations of the method\n\nWeakness:\n- A missing more detailed discussion on scalability, in terms of number of agents, since the current work assumes always 2 agents.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "ZtYa7QyVeb", "reviewer": "Reviewer_cwpF", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors develop a training procedure for two-player cooperative games that generate diverse policies. The idea is to generate policies that are incompatible when paired together and train agents with this loss function. This is tested in two toy environments and an Overcooked grid world. In all cases, LIPO finds diverse solutions (when possible) as well as diverse solutions. \n", "summary_of_the_review": "The paper presents a new algorithm that is clearly motivated with theoretical and empirical support. I recommend the paper is accepted if my above comments can be addressed. ", "strength_and_weaknesses": "\nThe paper is very clearly written and well organized it is easy to read and follow the logic. The toy environments are highly diagnostic and give strong insights into why this algorithm outperformer baselines and other approaches. It was nice to see the algorithm tested both for its ability to generate diverse populations and show that those populations matter for behavior. \n\nOne key weakness is that the algorithms were all tested on new benchmarks created just for this manuscript. Since there are now many competing approaches to this kind of diversity generation, it would have been nice to see this algorithm outperform others on a task designed by others. \n\nFurther, I would like to see some ablation experiments to understand the role of the \\lambda_MI in the Overcooked environment. It is further not clear in what contexts this term will be important. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "VohtFpNWo2", "reviewer": "Reviewer_PWGa", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper introduces the LIPO (Learning Incompatible Policies) algorithm for generating diverse policies via a mutual information objective. This is then demonstrated empirically in two simple environments (a cooperative matrix game and a 2D navigation task), as well as in the Overcooked domain.", "summary_of_the_review": "Overall, this is a strong paper with good results. It is likely to be useful to researchers seeking to design cooperative agents with diverse behaviors, as well as possibly for designing agents in non-cooperative and mixed cooperative-competitive settings.", "strength_and_weaknesses": "Strengths\n- While conceptually simple, the proposed framework for generating diverse populations of agents is a powerful idea that could have applicability beyond cooperative domains, e.g., for designing agents that can compete effectively against agents or where optimal strategies are difficult to find.\n- The inclusion of the anonymized source code is appreciated, and all of the details in the appendices aid reproducibility.\n- The paper is written clearly and includes solid theoretical justification for the implementation of an additional MI objective.\n\nWeaknesses\n- The related work section could be earlier in the paper, and possibly compared with LIPO in more detail. For example, what would the \"domain knowledge\" required by QD look like in the experimental settings described here?\n- Figure 8 could include more context; is the goal to generate something that looks like the uniform distribution? Or are some choices clearly better than others in this environment? Essentially, interpreting this requires additional background knowledge about Overcooked and about what an ideal solution would look like.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: notable-top-25%