Generating Teammates for Training Robust Ad Hoc Teamwork Agents via Best-Response Diversity
Authors: Arrasy Rahman
Elliot Fosong
Ignacio Carlucho
Stefano V Albrecht
Venue: Accepted by TMLR ()

Abstract: Ad hoc teamwork (AHT) is the challenge of designing a robust learner agent that effectively collaborates with unknown teammates without prior coordination mechanisms. Early approaches address the AHT challenge by training the learner with a diverse set of handcrafted teammate policies, usually designed based on an expert's domain knowledge about the policies the learner may encounter. However, implementing teammate policies for training based on domain knowledge is not always feasible. In such cases, recent approaches attempted to improve the robustness of the learner by training it with teammate policies generated by optimising information-theoretic diversity metrics. The problem with optimising existing information-theoretic diversity metrics for teammate policy generation is the emergence of superficially different teammates. When used for AHT training, superficially different teammate behaviours may not improve a learner's robustness during collaboration with unknown teammates. In this paper, we present an automated teammate policy generation method optimising the Best-Response Diversity (BRDiv) metric, which measures diversity based on the compatibility of teammate policies in terms of returns. We evaluate our approach in environments with multiple valid coordination strategies, comparing against methods optimising information-theoretic diversity metrics and an ablation not optimising any diversity metric. Our experiments indicate that optimising BRDiv yields a diverse set of training teammate policies that improve the learner's performance relative to previous teammate generation approaches when collaborating with near-optimal previously unseen teammate policies.

Published in Transactions on Machine Learning Research (05/2023)
Generating Teammates for Training Robust Ad Hoc Teamwork Agents via Best-Response Diversity
Arrasy Rahman
arrasy@cs.utenas.ck
Deportment of Computer Science
University of Tezas at Austin
Elliot Fosong
e.fosong@eed.co www.ggieedd.......
School of Informatics
University of Edinburgh
Ignacio Carlucho
ignacio. waw.ooocararcccoooooooooooooo uk
School of Informatics
University of Edinburgh
Stefano V. Albrecht
s.albrecht@ed.com ac.
School of Informatics
University of Edinburgh
Reviewed on OpenReview: https: // openreview. net/ forum? id= 15B2f R0RR01
Abstract
Ad hoc teamwork (AHT) the challenge of designing a robust learner agent that effectively collaborates with unknown teammates without prior coordination mechanisms. Farly ap- proaches address the AHT challenge by training the learner with diverse set of handcrafted teammate policies, usually designed based on expert's domain knowledge about the policies the learner may encounter. However, implementing teammate policies for training based on domain knowledge is not always feasible. In such cases, recent approaches attempted to improve the robustness of the learner by training it with teammate policies generated by optimising coommation-thooretee diversity metrics. The problem with optimising existing information-thoordtii diversity metrics for teammate policy generation the emergence of superficially different teammates. When used for AHT training, superficially different leammate behaviours may not improve learner's robustness during collaboration with unknown teammates. In this paper, present automated teammate policy generation method optimising the Best-Response Diversity (BRDiv) metric, which measures diversity based on the compatibility of leammate policies in lerms of returns. We evaluate our approach in euvironments with multiple valid coordination strategies, comparing against diversity metrics and an ablation not optimising
methods optimising information- any diversity metric. Our experiments indicate that optimising BRDiv yields a diverse of training teammate policies that improve the learner's performance relative to previous teammate generation approaches when collaborating with near-optimal previously unseen teammate policies.
1 Introduction
Ad hoc teamwork (AHT) is the challenging problem of creating a single agent, called the learner, which can robustly collaborate with a of unknown teammates without prior coordination mechanisms (Stone al., 2010 Mirsky al.) (2022). Although teammates in AHT are assumed to be working together achieve a common goal, they may exhibit. different behaviours assume different roles in the team. Attaining optimal

---

Published in Transactions on Machine Learning Research (05/2023)
A
B
A
B
(b) Generated teammate policies by optimizing best-response di-
(a) Superficially different generated team- mate policies with high trajectory diversity.
versity,
Figure Improving learner robustness by reducing superficial differences between generated teammates. Figures Ta and show an illustrative example of training an AHT learner with different sets of generated teammate policies. In this example, the learner (red dot) and its teammate (blue dot) must move to the same landmark (green rectangles) to get rewarded. Following the larger variation within the trajectory generated by different policies (blue arrow), Figure Ta shows leammate policies with higher trajectory diversity. At the same time, the illustrated policies also contain high superficial differences since common best-response policy can effectively deal with each policy. Training a reward-maxxmiiing learner against teammates from Figure La will result in a learner that only moves towards landmark A, which will cause the learner to produce highly suboptimal returns when dealing with teammates moving towards landmark B. Meanwhile, Figure 1b shows teammate policies with fewer superficial differences, whose trajectories require two best-response policies corresponding to the movement lowards landmarks A and B. An AHT learner trained with these teammates will learn to follow their teammate to any landmark. This is symbolised by the red arrows pointing to landmark A and B. Figure la and 1b illustrate how reducing superficial differences between leammate policies intended for AHT training can improve the learner robust indicated by the learner's ability to achieve high returns when collaborating with a broader range of teammates.
collaboration with teammates with different policies and roles may require the learner to use distinct policies. To solve AHT's challenge of optimal collaboration with unknown teammates, robust AHT learner must then adapt its policy based on the teammates' displayed behaviour and the current team composition.
Prior AHT approaches train learner by allowing it to interact with different teammates during training, These interaction experiences are utilised to approximate the best-response policy to each teammate by means of reinforcement learning. The best-response policy to each encountered teammate enables the learner to maximise its returns when collaborating with that particular teammate. Alongside the best-response policies, AHT approaches identify unique characteristics that differentiate the learnates behaviour. The learner then decides its action at evaluation time by initially inferring whether the unknown teammates' behaviour displays specific characteristics identified during training. Based their identified characteristics, learner approximates the optimal policy for collaborating with unknown teammates by generalising the best-response policies learned for teammates encountered in training. Existing AHT approaches often use policy mixtures (Albrecht et al., 2016 Barrett et al. 2017) or neural networks {Rahman et al. 2021 Papoudakis et al. 2021a Zintgraf et al. 2021 as generalisation models that extrapolate the best-response policies designed for training teammates towards new teammates with unknown policies.
Following its importance for AHT training, designing a diverse collection of teammate policies is crucial for training robust learner. Careful design of such a collection is especially required in problems where there are different possible teammate policies which require different best-response policies. Failure to identify the tcammate policies requiring different best-response policies for AHT training can cause the learner to be less robust. A learner's lack of robustness stems from existing AHT approaches only learning the best-response policy to teammates encountered during training. The learner's returns are more likely to be suboptimal when paired with teammates whose best-response policies are not learned during training. Although its generalisation model alleviates this problem, training learner to collaborate with all teammate policies requiring different. best-response policies remains more reliable way improve its robustness.
2

---

Published in Transactions on Machine Learning Research (05/2023)
Previous approaches for designing teammate policies for AHT training fall into two categories. First, early AHT approaches (Albrecht & Ramamoorthy 2013 Barrett et al. 2014 Albrecht et al. 2016: Barrett et al. 2017] formulate training teammate policies based on experts" knowledge regarding reasonable teammate
behaviours an agent may encounter in an environment. Second, more recent AHT approaches (Xing et al.)
2021: Lupu el al. 2021 Lucas & Allen 2022) generate diverse learmale policies by optimising information-
thcoretic diversity metrics, which encourage an increased difference of the trajectory or action distribution
between generated teammate policies.
In terms of facilitating the emergence of robust learners through AHT training, existing teammate policy generation methods face significant problems. Methods that rely on an expert's domain knowledge to formulate reasonable teammate policies have limited applicability since such knowledge is often unavailable or difficult to elicit in many real-world AHT problems. Meanwhile, teammate generation methods that maximise information--heortiic metrics may produce teammates with distinct trajectory distributions despite having the same best-response policy (Lupu et al., 2021). We refer to differences between these teammate policies that the same best-response policy can optimally deal with as superficial differences. Teammates with superficial differences are redundant for improving robustness because training with them will encourage AHT lcarners to Iearn the same optimal policy. In a typical tcammate generation process where only a finite number of teammates can be generated, such redundancy should be reduced to encourage learners to become more robust by instead learning a broader range of best-response policies during training. We further illustrate this need to reduce superficial differences between generated teammate policies in Figure
In this work, we present teammate generation method which discourages the emergence of teammate policies with superficial differences by optimising diversity metric called Best-Response Diversity (BRDiv D Instead of assessing diversity in terms of informationn-hherrele measures, BRDiv measures diversity based on returns. To compute diversity, we measure the returns obtained from pairing any policy in the set of generated policies (IItrain) with policies from the set of best-responses to I1train, defined as BR( We empirically demonstratc that BRDiv prevents the emergence of tcammate policies with superficial differences in their behaviour. This improvement is achieved by optimising the best-response policy for a teammate policy to produce low returns when collaborating with other teammate policies. The BRDiv metric can be optimised using off-the-shelf MARL techniques to produce tcammate policies with minimal superficial differences. Our experiments compare the returns of learner trained with teammate policies generated by BRDiv, previous teammate generation approaches based on action and trajectory diversity maximisation (Lupu et al. 2021 Lucas & Allen 2022), and an ablation of BRDiv that trains teammates' policies independently. We empirically demonstrate the robustness of a learner trained with teammate policies generated by BRDiv, by showing that it achieves higher returns than other evaluated baselines when paired against near-optimal previously unscen tcammate policies.
2 Related Work
Ad Hoc Teamwork (AHT). AHT was defined as formal challenge of developing a learner capable of collaborating with unknown leammates by Stone et al. (2010). Since then, previous works (Mirsky al. 2022 have explored AHT under different application areas and alternative names, such as zero-shot coordination (ZSC) (Hu et al.] 2020) which explores AHT in problems where unknown tcammates are optimal agents optimising the same reward function as the learner. Many of these works utilise type-based methods (Albrecht ltt al. 2016 Barrett et al., 2017 Albrecht & Stone, 2018 Rahman et al. 2021 2022). A limitation of
type-based approaches that they assume access to predefined teammate policies for learning. This entails the manual specification of all possible types, which is often an infeasible process. Our work soeks to bridge this gap by providing ways to automatically generate leammates.
Multi-agent Reinforcement Learning (MARL). The objective of MARL is to jointly train a set of agents to maximise their returns in the presence of each other (Papoudakis et al. 2021b
Zhang et al. 2021).
Unlike ad hoc teamwork, these methods assume full control of all members of the team. Current methods in the literature have shown great success in solving complex tasks (Vinyals et al.
2019 Christianos et al.,
2021), and have been shown to be able to adapt to novel tasks (Schafer et al. 2022
However, a drawback of
1 "mmllementation code is provided here: http:::/ggttubb.cc//oeeaaaeeeeep www.soooocoooeeeeeeeee hawpa///c/coeeeaaaae/////// agents/BRDiv
3

---

Published in Transactions on Machine Learning Research (05/2023)
joint training is that the resulting agents have low performance when interacting with agents that are not encountered during the joint training process (Vezhnevets et. al. 2020: Hu et al. 2020: Rahman et al. 2021
Teammate Policy Generation. Diverse teammate policy generation has been previously explored in problems that are closely related to AHT, such as in zero-shot coordination (ZSC) (Hu et al., 2020). Several works in this area formulate diversity terms of information-theoreiic measures defined over the generated
policies' trajectories (Xing al., 2021 Lupu et al. 2021 Lucas & Allen 2022]. J. Despite its prevalence, previous works (Lupu et al., 2021 Liu et al. 2021] highlighted that training with teammates generated by trajectory diversity-based methods does not always lead to improved learner's robustness, which we also demonstrate through our experiments. This is because many teammate behaviours producing distinct. trajectories entail the same learner's best-response policy. While Liu et al. (2021) also proposed an approach based on the best-response policies' performance, their approach is limited to zero-sum games.
Diversity in Reinforcement Learning. In singlo-agent reinforcement learning, diversity maximisation is mainly utilised as way for agents to increase exploration (Pathak et al. 2017 Hong al.) 2018 Parker-Holder
et al. 2020) or discover reusable skills (Eysenbach et al. 2019). For example, Eysenbach et al. (019)) proposed method to learn a diverse set of reusable skills by only maximising an information-theores
objective.
Similarly, in multi-agent reinforcement learning (MARL), works such as MAVEN (Mahajan et al. 2019).
have aided exploration by maximising a mutual-informattin ion metrie ween the trajectories and a latent, space. Another recent work also utilised reward randomisation to achieve diverse behaviours in multi-agent settings (Tang et al. 2021). As another application of diversity optimisation in RL, Li et al. (2021) proposed a method optimising an information-theoredt objective to facilitate agents' specialisation towards diverse range of roles for solving MARL problem. Note that unlike when inducing diversity for teammate policy generation, these techniques are not designed to create a diverse set of teammates to improve the robustness of learner.
Population- based Training (PBT). Our method aims to train a population of agent policies that optimise a specific metric, similar to existing works on population-based training. Population based training was proposed by Jaderberg et al.] (2017) as a way to speed up the optimisation process of neural networks. This asynchronous algorithm jointly optimises a population of models and their respective hyperparameters, through an alternating process of parallel training and hyperparameter tuning. Further work from Li et al. (2019) then introdueed framework that enables population-based training in more general settings. Unlike our method which optimises the diversity of the entire population, note that PET methods optimise an objective function defined over a single individual. PBT then uses its population of agents to iteratively generate new population members having more optimal objective function values, which is different from our method's s use of MARL techniques for optimisation.
3 Background and Setting
In this section, we formalises the problem of teammate policy generation. We first start by formalising the interaction between agents in our AHT problem. We then provide details on the main objective of a teammate generation process given our previous formulation of agents' interaction.
3.1 Decentralised Partially Observable Markov Decision Process
We model the interaction between agents in a AHT environment as a decentralised partially observable Markov decision process DDec-POMDP (Bernstein et al. 2002). Dec-POMDPss are formally defined as an 8-tuple, (N, S, (A))) P, R, ( O,7). Within a Dec-POMDP, N, S, and 7 denote the set of agents, state space, and discount rate, respectively. A' and Ω' represent the action and observation space of agent i, respectively. The transition function of a Dec-POMDP is denoted by P: x A' x... x A + AS, where AS represents the set of all possible probability distributions over S. Similarly, the reward function is denoted by R: S x A x ..... x Al + R. and the observation function as O: + A(Ω* x .... x N
Each episode in Dec-POMDP starts from an initial state, 8p E S. At timestep t, cach agent i E N receives an observation of ~ O(s)) and selects an action of according to its policy (HH)), which is conditioned on its observation-action history H; (oger containing the sequence of observation and actions observed up to
4

---

Published in Transactions on Machine Learning Research (05/2023)
timestep t. Agents then jointly execute their selected action in the environment. After execution of the joint action ae, the state of the environment changes according to the transition function 8++1 P(*,, at), and each agent is rewarded with at). This reward is common to all agents due to the cooperative nature of AHT problems.
3.2 Teammate Policy Generation
A teammate generation process aims to design a set of K teammate policies, I = { x² that when being used for AHT training maximises the robustness of the learner. Formalising this goal as a quantitative learning objective requires a measure of robustness for a given Dec-POMDP. Once such a robustness measure formally defined, a learning objective can be formulated by defining how the generatod teammate training policies affect the learner's robustness.
We characterise learner policy robust if it achieves high returns when collaborating with teammates from an unknown evaluation set, II Given a Dec-POMDP where the learner is assigned fixed index E N during the teammate generation, AHT training, and AHT evaluation process, our proposed measure of robustness is defined below:
Myooaal (x') = E (a a of P,O Σ yat R(se,
(1)
with dz (a/, a) x², distribution over set X and the joint policy of the [N] - agents other than the learner respectively. It is important to note that II in Equation | may consist of policies not encountered during AHT training, highlighting the need for a robust learner for effective collaboration.
(X) and π denote agents' joint action, the policy of the learner, uniform
Since the proposed measure robustness depends on the set of policies in 11 we outline assumptions regarding the policies that can appear in II As formulated by Stone al. (2010), we assume that II consists of feasible teammate policies. For x to be considered feasible, there must be a policy n' that can achieve expected returns above expert-defined threshold when collaborating with क-ं Note that this threshold can be decreased if we want to increase the number of feasible policies considered in II Similar to the motivation behind its definition by Stone et al. (2010), the feasibility criteria behind π e II reflects how encounters with highly suboptimal teammate policies that no one can collaborate with is improbable in many practical applications of AHT.
As the missing piece to formalise the goal of the teammate generation process, we now define how I affects the robustness of a learner produced by AHT methods based on MI Given an AHT method to train learner, Iraaann is utilised to learn an optimal AHT policy, (IItaa)))) that maximises the expected returns of the learner when collaborating with teammates from I Given a Dec-POMDP the optimal policy given I|train is defined below:
Σ o R(se. an)
train = argmax Ex, of of P.O
(2)
1=
Later during the AHT evaluation process, n (TTural))) is the policy whose robustness when collaborating with tcammates from Ivval will be mcasured.
Based on the definition of (IItmin)))) the goal of a teammate generation process is find an optimal of training teamnates, train that maximises the robustness of an AHT agent. Given a Dec-POMDP and an unknown II Itttiain is formally defined as:
II trainn argmax MI (Itaan))))))).
Mesii
While setting train ovvl provides an optimal solution to the above objective, note that the teammate generation problem operates in a setup where II is unknown during training. Therefore, the main challenge in the teammate generation problem arises a result of optimising for I without knowing Nevall
5

---

Published in Transactions on Machine Learning Research (05/2023)
Self-play interaction
Cross-play interaction
x
of
Minimised via MAA2C
of
H2
Shared citte
Vira (i m,.))
Cnttc Loss
Vy-u (1 1)
Shared antio
Cross-play matrix
G (1 .22)
G)
G (12) .f
G2 (1t) .82)
3 (z)) (aa
1 (it) (ae
(2), [2)
(8),
Maximised via MAA2C
Maximised MAA2C
BRDi-[n (7))
Figure 2: Teammate Generation By Optimising BRDiv. This figure visualises our teammate generation method assuming that we generating [ = for AHT environments with two players. Our method utilises MAA2C (Papoudakis et al.) 2021b) to generate a set of teammates that maximises the BRDiv divorsity metric. The MAA2C algorithm trains a separate actor network (purple rectanglos) to represent the policies each generated leammaae,, x² E [Ttrain, and their associated best-response policies, ा -- Assuming x², x* E ITtrain, shared critic network (green box) is trained estimate expected returns from the interaction between any possible pairs of (T', The shared critic network's return estimates for all pairs are then compiled into cross-play matrix (blue bordered box), which serves a basis to compute the BRDiv diversity metric (red box). Finally, a diverse I is produced by optimising the actor networks to maximise the cross-play matrix-based BRDiv metric by minimizing the actor loss outlined in Equation g
4 Best- Rest-Response Diversity Metric
This section provides the details of Best- Rest--eesponse Diversity (BRDiv), the diversity metric that is optimised by our leammate generation method. Section 4.1 starts by outlining a desirable characteristic for I set of teammates policies generated for AHT training. Section 4.2 then formally defines BRDiv as diversity metric optimised by our teammate gencration approach to encourage the crcation of a desirable I
4.1 Desirable Diversity for AHT
A good diversity metric to generate I[train must consider the effect of I[train on learner's learning process and their robustness when collaborating with policies from II Current AHT methods train learner to model the encountered teammates and approximate the best-response to each teammate policy in I by optimising Equation 2 Although generalisation models used by AHT methods can alleviate this issue, a learner is less likely to achieve high returns when collaborating with eeall E IIvall whose best-response policy, x" different to the best-response policy to any train e I The adverse effects of encountering oval whose best-response is never learned during training why setting I Io is the ideal solution to maximise learner robustness as defined by Equation However, we often do not know what II is.
Without knowledge regarding II way to improve learner's robustness is to increase the mumber of best-response policies that it learns from interacting with trrein E raaa This idea intuitively aims to reduce the likelihood of a learner being unprepared by not knowing how to best respond to an unknown teammate. Increasing the number of best-response policies learned during training is equivalent to reducing superficial differences between generated teammates, as illustrated in Figure !
Our teammate generation method aims to reduce the appearance of redundant policies for AHT training in taain characterised by their superficial differences. Teammates with superficial differences are indicated by
6

---

Published in Transactions on Machine Learning Research (05/2023)
their common best-response policies. Assuming a sufficiently small number c, pair of policies X', x E II are formally deemed to be superficially different if they share the same best-response policy as defined below:
Σ y R(s an) - -Eat as PPO P.O
Σ (e) Sc.
π * x² A E Batrot az , P.O
with az = (a,; at) and x being the best-response policy lo ne' defined as:
Σ 7 R( a1)
π = argmax E ux of ~T, P.O
Through the interaction with each n² from I with minimal superficial differences, the learner will learn as many best-respouse policies possible to interact with a possible teammate. This equips the learner with a more comprchensive library of bchaviours to effectively collaborate with any tcammate policy. Consoquently, the learner's robustness should improve by reducing the likelihood of it having no adequate strategies to effectively collaborate with unknown teammate from 1leval.
4.2 BRDiv Metric
Following our desired characteristic of diversity metric, this section defines a diversity metric that can be maximised to generate ITtrain with minimal superficial differences between the generated tcammates. The description of BRDiv assumes that only two agents exist. in the environment. Assuming a Dec- where learner is assigned a fixed index from N during teammate generation, AHT training, and AHT evaluation, extending our proposed diversity metric and optimisation mothod to environments with more than two agents is straightforward.
BRDiv aims to generate a set of diverse policies for AHT training, 11train = x= :..,, n "), where similar best-response policies cannot be used to effectively collaborate with different generated teammnte types from I|minn Therefore, defining a metric that quantifies the effectiveness of two agents' policies when collaborating with each other is a crucial first step in formulating our diversity metric. We measure the effectiveness of two policies when collaborating via their expected returns, which is inspired by our notion of robust collaboration int roduced in Section 3.2 Assuming that agent, j and & are interacting with each other based on policies, x (a (H() and (a [H)), that are conditioned on their respective observation-action history H and H this return-based effectiveness mcasure is defined as:
VH H,) was a
Bat
Σ TT-R((sT- ar))
Hi. H
(4)
This condiy--ond return-basee effectiveness measure provides a foundation for defining an optimised diversity metric to achieve the goal of BRDiv. Demoting the best-response policy to x by π -k, and the set of best-response policies to each policy in I[train by (I ), we use Equation 4 to evaluate the effectiveness of æ -h; BR(II))ain) when collaborating with n E IItrainn Given a pair of observation-action historics, H, and H&; we arrange the measured cooperative effectiveness between all possible (N* ->.))) M (R(Tuaaii into a K x K cross-play matrix, C BR(I""")) (11,, H!?). Elements of this cross-play matrix are defined as:
Vj,k (1,2,... ..., K}.Cll train tRi (H. H2) V),(-R.) H?))
(5)
In this work, the way we compute C I (B(II illustrated by the blue arrows and table in Figure
(H., HP) and assemble it into the cross-play matrix are
The BRDiv metric is based on the intuition that a good train to ensure the learner's robustness must possess two characteristics. First, the cross-play matrix of Iaaain must have high values on its diagonal elements ensure that each n² train interacts effectively with its associated best-response policy, -- (II This characteristic also prevents the emergence of teammate policies producing low returns, which
7

---

Published in Transactions on Machine Learning Research (05/2023)
reward-optimising agent would have reason to use in environment. Second, the off-diagonal elements of C (BR(((")))))) must also have low values to discourage a best-response policy - E BR( from being effective for collaborating with x E (I1train - {T²)). }). By optimising the incompatibility of a best-response policy when dcaling with other policies in Iltrain, we aim to induce the need for different collaboration strategies to deal with each policy in I
Based on these two characteristics, we define our diversity metric as:
BRDiv (H) Hx)) Tr (cultiig BRiII
(H, BR(I
+
Σ
(H, H?) - Ch.
Ci BB((I (H, H??)
i ()...k ins
(6)
Σ
mrain BR(II
(H), H?) - C (R مز
(H; Hi:)
1,
(. K}.
The maximisation of the first term in Equation enforces the first characteristic. Meanwhile, maximising the remaining terms produces a cross-play matrix with low off- diagonal values, encouraging the generated policies to fulfil the previously mentioned second desired characteristic.
5 Maximising BRDiv with Multi-Agent Reinforcement Learning
We now describe an optimisation technique that maximises BRDiv to generate 11train, Although a wide range of multi-agent RL algorithms can be used to maximise BRDiv, we propose an optimisation technique based the Multi-Agent A2C (MAA2C) algorithu (Papoudakis et al. 2021b) due to the straightforward modifications required to utilise it for maximising BRDiv. We use the centralised critic of MAA2C to estimate the elements of the cross-play matrix defined in Equation 5) Mcanwhile, the policies in ITtaain alongside their associated best-response policies in BR(IItrain) are treated as actors that MAA2C trains. A detailed pseudocode of our MARL--based diversity optimisation technique is provided in Algorithm D in Appendix A visualisation that summarises our proposed teammate generation method is also provided in Figure 2
Data Collection. Before each update to the actors and centralised critic, we separately collects two types of interaction data for training. First, we collect self-play experiences where let a policy, x* E Iitrrin interact with its associated best-response policy, x E BR(II(s)m). The second type of data is cross-play experiences which we collect by letting a policy, T² E IItrain, interact with the best-response policy of a different policy, x E BR( - (T') )). Both self-play and eross-play interaction data are then stored in separate storage denoted by DSP and DXP respectively. Note that assuming we also record the idcntity of the agents generating the experience, which is j and -k; each experience stored in the storage is then defined as 7-tuple, [[H],H,] af a,"" {Re), (Hf Hz41), j, j--k) with Hi and H2 denoting the observation-action history from using policies n² and T up to timestep t. After the models are updated, DSP and DXP are emplied and new self-play and cross-play interaction data are collected for the subsequent updale.
Actor and Centralised Critic Architecture. As we mentioned at the beginning of Section 5 the actors in our optimisation method correspond to the generated teammate policies in I and their associated best-response policies. For each x' E (ITtrsin u (R(Tusi))) this policy is represented as neural work parameterised by 8¹, In the remainder of our description of BRDiv, note that we denote the set of actor parameters from I U (R([]trii)) O.
Like the actor networks, the centralised critic used in this optimisation process is also implemented neural network. The centralised critic network is specifically responsible for estimating elements of the cross-play matrix, - _BR(I"=*"))) based on Equation 5. As shown in Figure 2 the shared critic network inpul consists of a sequence of observation-action history from both players in the environment. In the remainder this document, note that we drop Irtrain as parameters to the cross-play matrix since evaluating each clement of this matrix row and column j does not involve æ and π -i+ Instead, we evaluate (H, H,")) by also concatenating one-hot, identifiiation of and -j to the centralised critic's input as indicated by Figure
8

---

Published in Transactions on Machine Learning Research (05/2023)
Learning Objective. The centralised critic network is trained to minimise the n-step return loss. As in many deep RL methods, we incorporate a target critic network parameterised by ø to compute the target values for the critic network. Using the collected experiences from DSP and DXP, the centralised critic loss function defined below:
Σ r Ress - -- - H (lmm)) ) k=U
Lø(DSP, DXP)=
E DAPUDDP
Ve (H, Hi)
2
Given a stored experience from D or DXP, the actor networks in 11train and (I( are trained maximise the BRDiv-based advantage function, (H; Hi, (Be+k )a-) Hitw Hign), defined below:
BRDiv(C,,-3 (. ( H! HB)) BRDiv base, (st)).
(8)
In the above expression, C. (H) H?, (Re+k) (Re+k)RZ0 18-1 H!, H?) is a cross-play matrix which has its entry at row and column j replaced by n-step return estimate resulting from the interaction between o' and m- Meanwhile, (sase..(ss) is a baseline cross-play matrix whose elements only depend on H; and H²
Similar to its role in the optimisation process of methods based on A3C (Mnih et al.) 2016), we use an n-step return-based estimate for one of the elements of this cross-play matrix to reduce the bias of gradients associated with the actor loss updates, which is commonly used method single-agent actor-critic methods. We then subtract baseline function from the n-step return estimates to reduce the variance of the gradient updates for the actor networks. Finally, note that n-step advantage function estimate highly resembles the advantage function in MAA2C (Papoudakis et al., 2021b), except that we define the advantage function in terms of the best-response diversity of cross-play matrices.
Given stored experiences from D and DXP, this results in the use of the following loss function to optimise the actor networks:
Lo(DSP (DXP)
Σ
- -log (Taa (:) (H;
DDPUDXP
(9)
A A (1, H1? (Re4k Ji Ultt ((itz)
where,
V
Voig((t) (H,
if (p. ④) # (i.j)
Cpree (H,, H? ( 10-1 Hit H2+m)
En= x Retk V V (H H2 H2 otherwise
Obase. (H; H²) V (H,
(10)
are the crosss-play play matrices computed to evaluate the advantage function defined in Equation g This objective function that multiplies agents' actions log-likelihood with the advantage function is similar to how actor networks are trained in MAA2C. Minimising Equation updates the actor networks to encourage the emergence of actor networks that assign higher probabilities towards actions leading to trajectories with higher BRDiv values.
6 Experiments
We evaluate the offectiveness of BRDiv in improving the robustness of an AHT learner when dealing with previously unseen leammale lypes. First, we provide details of the environments used in our teammate generation experiments in Section 6.1 This is followed by an overview of our experiments' AHT training and evaluation process Scction 6.2 Scction 6.3 then details the bascline approaches we compare BRDiv against. We then present and analyse the results of the teammate generation experiments in Section 6.4 Finally, Section 6.5 analyses the behaviours of teammate types generated by BRDiv.

---

Published in Transactions on Machine Learning Research (05/2023)
0.75
0.75
(a) Cooperative Reaching.
(b) Level-Based Foraging.
(c) Simple Cooking.
Figure Environments for Teammate Gencration Experiments. Figure visualizes an example state of the Cooperative Reaching euvironment. In this visualisation, the red circle, blue circle, and grids with texts denote the teammate, learner, and reward-providing coordinates. Meanwhile, level-based foraging environment is visualised in Figure 3b) The white and red icons represent the players and the objects that exist in the environment. The level of each player and object is then visualised in the bottom right corner of their respective icons. Finally, an example environment state for the Simple Cooking environment is provided in Figure 3e The kitchen layout in this environment is such that the chefs are inside a decagon kitchen with a table the middle, symbolised by the red decagon with a plate on top of it. The required cooking items and ingredients to finish the recipe are then placed on top of the green counters in this kitchen. To finish the task, all processed ingredients and the plate must be placed the serving counter which is visualised green side of a decagon with star top.
example state of
6.1 Environments
Our experiments evaluate BRDiv and the baseline approaches three multi-agent environments. All environments used in our experiments have two agents, one of which will be controlled by a leammate policy during an interaction episode. A visualisation of an example state from each environment is shown in Figure 3 Further details of the environments used in our experiments
provided below:
Cooperative Reaching. Cooperative reaching is a simple environment situated in a 5x5 grid world. Each agent has five actions corresponding to staying at a particular grid and moving into the four cardinal directions. The goal of all agents is to reach and jointly stay in grid cell whose location belongs the set of reward-providing coordinates, (0, 0) and (4,4) 4) provide reward of 1 to both agents once they are in the same grid cell with this coordinate. Meanwhile, the grid in (0,4) and (4, only provide a reward of 0.75 once both agents arrive. In this environment, the collaboration strategies correspond to the distinct ways teammate may select a destination grid within F. robust AHT learner should ideally learn to follow their teammates towards any seward-providing coordinates.
coordinates, F {00, 0), 4), (4, 4)). Within these reward-providhng
Level-based Foraging (LBF): In this environment, agents must retrieve three objects that are randomly scattered in a 6 x 6 grid world. Agents can move in either of the four cardinal directions and have a special action that allows them to collect adjacent objects. However, note that agents cannot. be positioned in the same grid. At the beginning each episode, each object and each agent are assigned a level that determines whether an agent may collect an object. To successfully pick up an object, the total level of agents choosing the collection action from grids adjacent the object must be al least. the same the level of the collected object. We then enforce the need for collaboration between agents by setting the level of each object as the total level of agents in the environment. For every successful collection of object, agents will then be given reward of 0.33.
Simple Cooking: Simple Cooking an environment where two chefs must collaborate create a simple dish with chopped tomatoes and blended carrots. Following Figure 3c the two chefs only be positioned
10

---

Published in Transactions on Machine Learning Research (05/2023)
on 10 empty spaces between the cooking counter and the table in the middle of the kitchen. However, they cannot be positioned in the same empty space in the kitchen. Fach chef is then equipped with eight actions that enable them to (i) stay still, (ii) move clockwise, (iii) move anti-clockwse,, (iv) retrieve an ingredient from a counter, (v) put an ingredient to a counter, (vi) retrieve an ingredient in the middle table, (vii) put an ingredient on the middle table, or (viii) use cooking tools placed on a counter. A chef must be positioned in the space closest to the target counter to collect or put an ingredient from or to a counter. On the other hand, a chef can put or collect items on the table at any time. Using blender or knife to blend carrots or chop tomatoes requires an agent to be positioned in the space closest to the tool and have the right ingredient placed on the same counter as the tool. In this environment, a reward of 0.25 is provided to both agents right after (i) the tomato is chopped, (ii) the carrot is blended, (iii) both chopped tomato and blended carrot are placed on a plate, and (iv) a plate containing chopped tomatoes and blended carrols has been placed on lop of the serving counter.
When deciding the environments used in our experiments, we consider whether the environment has multiple cooperation strategies that do not share the same best, response policy for optimal collaboration, which means the learner really needs to adapt to the behaviour of the other agent. All three environments in our work fulfil this criterion as many different tcammate strategies can solve the underlying collaboration task. In Cooperative Reaching, multiple strategies correspond to the distinct locations teammates can move towards to get rewarded. The different strategies in LBF correspond to different orderings followed by teammates when collecting the scattered objects. Simple Cooking environment also has different subtask allocation and completion strategies for effective collaboration.
The environments in our experiments are also related to existing environments used for AHT evaluation. Level-based Foraging (Albrecht & Ramamoorthy 2013 Papoudakis et al.
2021b Mirsky et al. 2022 Rahman
et al. 2021) is a commonly used environment for MARL and AHT evaluation. Simple Cooking is also similar
to environments based on the Overcooked game (Wu et al.) 2021 Yu et al. 2023). We argue that Simple
Cooking is as complex as existing Overcooked environments following the many subtasks that need to be completed by agents, alongside the constricted hallway that, restricts the movement of agents.
6.2 Experiment Protocol
Our process to evaluate the compared tcammate generation methods can be divided into three stages. In the first stage, we run BRDiv and other baseline teammate generation methods to create a set of training teammates I The second stage utilises the resulting tcammates from the first stage to train an AHT learner. We then evaluate the performance of the robustness of the learner when collaborating with set of previously unseen teammate types from IIeaal
In the first stage, we run cach evaluated teammate generation method to produce K tcammate types. Each leammate generation method is run for five experiment seeds and learns for total timesteps. We utilised different K and T for each evaluated environment. In Cooperative Reaching, each teammate generation method is trained for 16 million timesteps to produce four teammate types due to the simplicity of the environment. Meanwhile, each method is trained for 200 million timesteps to produce six and eight different teammate types for LBF and Simple Cooking, respectively. Under each experiment seed, we save I produced by each compared algorithm at the end of the teammate generation process.
The second stage utilises traii generated from the first stage to train a learner policy through AHT training. To enable a fair comparison between results from each teammate generation method, our evaluation protocol uses the same AHT algorithm to train Iearner based on cach Iraii. We specifically use the PLASTIC Policy algorithm (Barrett al.) 2017) due to its ease of use for computing a learner policy given I produced by our teammate generation methods. In particular, PLASTIC Policy agent's 's decision-makinn process only requires the policy of each teammate and their associated best-response policies, both being a by-product of the teammnate generation process contained in the resulting I|train and (R(an)))
Using the learners produced in the previous stage, the final stage of our experimental protocol evaluates the learner's robustness when dealing with agents from I[ To evaluate robustness we construct Ie based on two different scenarios. In the first scenario, II consists of policies generated by the different teammate generation methods evaluated in this work. In the second evaluation scenario, to construct eevll
11

---

Published in Transactions on Machine Learning Research (05/2023)
Table 1: Experiment Baselines. This table outlines the differences between our method and the baselines compared in our experiments. The comparison between these different tcammate generation methods is basod on their optimised loss functions, self-play and cross-play data for training, and the use of a policy classifier to produce intrinsic rewards for MAA2C training.
Method
Loss Function
Self- Play Data
Cross-Play Data
Policy ID Classifier
BRDiv (Our method)
Equations
&
Yes
Yes
No
Independent
Equations
Yes
No
No
Equations
TrajeDi
Yes
No
No
with auxiliary loss
7 with intrinsic rewards
Equations
Any-Play
Yes
No
Yes
we define policies based on the predefined heuristics (defined in Appendix B Note that when evaluating against teammates generated by the same teammate generation method in the first scenario, we also measure the learner's robustness against teammate policies generated from different teammate generation experiment seeds. Such evaluation remains challenging since teammates generated by the same method under different seeds may have different behaviours that cause difficulties in effective collaboration.
As a mcasure of robustness, our evaluation process procoeds by evaluating the returns of an AHT learner when dealing with tearmate policies in I[eval, For each environment and evaluation scenario, we then compute an aggregated statistic of the returns achieved by each method when dealing with policies from II Using the stratified bootstrap confidence interval method (Agarwal et al., 2021), we compute a 95% confidence interval over the interquartile mean returns of AHT learners resulting from training with I from cach tcammate generation method. This confidence interval allows us to argue over the significance of the difference in robustness between teammate generation methods. We then also report breakdown of the mean returns achieved by the Iearner across the different policy types in I[ The resulting returns of a learner trained through generated teammate types produced by BRDiv and baseline approaches are reported and analysed in Section
6.3 Baselines
We compared our proposed method with two types of baselines. The first type of baseline comprises previous methods for automatically generating tcammates in AHT related problems, such zero-shot coordination. Meanwhile, the second type of baseline consists of ablation of our method, which removes parts of it responsible for encouraging ineffective collaboration between generated teammate policy and the best- response policy associated with another generated tcammate type. Since our experiments operate on fully observable environments, our loss functions are optimised such that Equations 7 and g auxiliary loss functions, and intrinsic rewards optimized by the compared methods are defined over the state of the environment. Further details of these methods and their implementation are summarised in Table ! and the remainder of this section. Appendix C then provides the value of cach methods' hyperparameters used in our experiments.
Prior teammate generation methods. Among methods under this category, we choose TrajeDi (Lupu et al. 2021h h and Any-Play (Lucas & Allen, 2022) as representative baselines. We choose TrajeDi following its usage of the action discounting term, which provides additional flexibility when defining the optimised information-theoredii diversity metric. Prior teammate generation methods other than TrajeDi define their optimisod diversity metric in terms of an agent's overall trajectory or its selected action at each timestep, which both have their drawbacks. TrajeDi's action discounting term enables users to tune the resemblance of its optimised diversity metric to action diversity and trajectory diversity- based approach. In the plots that we report in this work, we denote these TrajeDi-based basclines as TrajeDio, TrajeDi025, TrajeDio5, Tr07s and TrajeDi1, which use the action discounting term of 0, 0.25, 0.5, 0.75, and respectively.
We also add Aly-Play as a baseline following the results from Lucas & Allen 2022) that demonstrated its improved performance over TrajeDi a few environments. This bascline will be denoted in our analysis as AnyPlay. Unlike BRDiv and TrajeDi, Any-Play's teammate generation process adds an intrinsic reward that the actor networks also attempt to maximise aside from the original rewards from the environment.
12

---

Published in Transactions on Machine Learning Research (05/2023)
This intrinsic reward specifically evaluates the log- log-likelhhood assigned by classifier that distinguishes the different policies in I Intuitively, Any-Play optimizes different actors to produce trajectories that are distinguishable from each other. Thus, comparing BRDiv's performance against Any-Play also delivers insights regarding the gains from using a different optimisation technique to induce diversity.
We implement TrajeDi and Any-Play based on our MAA2C-based teammate generation method by first removing the evaluation of loss functions based on cross-play data. Compared to our proposed approach, the absence of any training based cross-play data prevents TrajeDi and Any-Play from minimizing the non-diagonal elements of the cross-play matrices defined in Equation [0] Maximising Equations X and g solcly based on self-play data has the effect of encouraging cach generated policy in I to achieve the highest possible returns when interacting with its best-response policy.
TrajeDi and Any-Play also add additional auxiliary losses or intrinsic rewards to encourage diversity in IItrain Using DSP for its evaluation, TrajeDi adds an auxiliary loss that minimiscs the negative Jensen-Shannon Divergence between the trajectory distributions of the actor networks being trained. For Any-Play, we add a loss function that trains classifier that identifies the population teammate belongs based on observed state and its action. The output of this classifier is then used at each timestep to compute an intrinsic reward that is added on top of the environment rewards during training.
A Ablations of BRDiv. We also compare our method against an ablation which independently trains K teammate policies with MAA2C (Papoudakis et al. 2021b) without maximising any policy diversity metrics. Our experiments denote this baseline as Independent. Comparing our method's performance against this ablation helps us identify the impact of optimising our proposed diversity metric on the resulting learner's robustness when dcaling with previously unscen tcammates. We train this ablation to maximisc Equations and g solely based on self-play data, which similar to TrajeDi and Any-Play encourage the generated policy to maximise their returns against their respective best-respouse policies. However, we do not add any intrinsic rewards or optimise auxiliary losses to encourage diversity among the generated teammate policies.
6.4 AHT Evaluation
This section provides the results of using the generated teammate policies for training an AHT learner based on the experimental protocol outlined in Section 6.2 The aggregated performance of learners trained with I produced by each teammate generation method outlined in Figure 4 Figure show show the performance of a PLASTIC Policy-based learner when interacting with teammates that follow one of the previously unseen hcuristics defined in Appendix B The performance of the same learner when dealing with previously unseen teammates generated by different. Leammate generation methods is then provided in Figure G
5a and 5b BRDiv provides a more reliable way to generate robust learners than the bascline methods when co collaborating with unseon hcuristics that are near optimal. When comparing the resulting returns between BRDiv-based learners with the Independent. baseline, we see that BRDiv achieves higher returns in all but three heuristics throughout the entire types of teammates used in the evaluation process. Except for interactions against tcammates using hcuristics H08 and H10, learner trained with BRDiv-based teammates consistently achieves the highest average returns compared to the other evaluated teammate generation methods in Cooperative Reaching. Meanwhile, BRDiv also consistently yields more robust learners than compared baselines in LBF except for interactions against teammates using heuristic H02. In experiments against teammates using policies generated by other teammate generation methods whose results are provided by 4d and ba 6c learners trained using BRDiv-based teammates consistently achieve the highest average returns compared to other bascline methods in all environments except for Simple Cooking. Finally, note that in cases where specific baseline methods outperform a baDiv- learner in terms of the resulting average return,, the difference in performance between BRDiv and these baselines is insignificant except for some evaluation scenarios based on Simple Cooking.
As shown in Figures 4a
4b
Another substantial evidence of BRDiv's reliability in training robust. learners can be found by comparing the confidence interval of returns between compared methods. In Figures 4 5 and we observe BRDiy's tendency to produce more compact confidence intervals in its returns indicates lower variance in a BRDiv-basod learner's returns across different training scods. The baseline methods' larger variance in returns results from their generated trtaiin having high variance in best-response diversity across different experiment seeds.
13

---

Published in Transactions on Machine Learning Research (05/2023)
Independent
Independent-
Independent
TrajeDil
TrajeDil
TrajeDi075
TrajeDi1
TrajeDi075
TrajeDio75
TrajeDio5
TrajeDi05
TrajeD005
TrajeDi025
TrajeDi025
TrajeDi025
TrajeDio
TrajeDio
TrajeDio
AnyPlay
AnyPlay
AnyPlay
BRDIv
BRDiv
BRDiv.
0.45
0.60
0.75
0,90
0.4
0.5
0.7
0.15
0.45
Returns
Returns
Returns
(a) Cooperative Reaching Heuristic Teammates
(b) Level-Based Foraging
(c) Simple Cooking
Heuristic Teammates
Heuristic Teammates
independent
Independent
Independent
TrajeDil
TrajeDil
TrajeDil
TrajeDi075
TrajeDi075
TrajeDi075
TrajeDi05
TrajeDi05
TrajeDi05.
TrajeDi025
TrajeDi025
TrajeDi025
TrajeDio
TrojeDi0
TrajeDiO
AnyPlay
AnyPlay
AnyPlay
BRDiv
BRDiv
BRDiv
0.4
0.6
0.8
0.45
0,60
0.75
0.45
0.60
0.75
0.90
Returss
Returns
Returss
(d) Cooperative Reaching Generated Teammates
(e) Level-Based Foraging Generated Teammates
(f) Simple Cooking Generated Teammates
Figure Aggregated Learner Performance During Collaboration With Policies From I1 This figure visualises aggregate statistic of the episodic returns achieved by learner trained with traain when collaborating with policies from ITeall, We show results for each baseline and evaluation scenario detailed in Sections 6.1 and 6.2 The learner interacts with each teammate type that constitutes I[ for five episodes. Using the stratified bootstrap confidence interval method (Agarwal et al. 2021 we report the 95% confidence interval of the interquartile mean episodic returns the learner achieves. Results show that our method significantly outperforms all basclines across the two evaluation scenarios Cooperative Reaching and Level- wese-BBaedd Foraging. Meanwhile, our method obtained lower returns than the baselines in the Simple Cooking environment against heuristics.
Across some seeds, the baseline methods still discover I[train with a high BRDiv value even without optimising BRDiv. These baseline methods can discover 11train with high BRDiv values since policies in Iltrain with high BRDiv also cxhibit high diversity in the trajectorics they generate. However, since high trajectory diversity does not imply high BRDiv, a few seeds the baseline methods also discover Iqtrain with lower BRDiv that yields learners with lower returns due superficial differences between generated policies. When comparing the BRDiv value of Iraaain by different teammate generation methods and the lcarner's episodic returns in Cooperative Reaching and LBF against heuristics in IIeval we found these measures to have strong Pearson correlation coefficient of 0.7664 and 0.7961 respectively. While TrajeDi has an action discounting hyperparameter that can be tuned to minimise the emergence of uraain with superficial differences (Lupu et al. 2021). our results indicate that tuning this hyperparameter is less effective in preventing the emergence of superficial differences between generated teammates compared to directly optimising BRDiv.
Important insights are also obtained from evaluating learners when collaborating with more suboptimal teammates. Against haammate-followin heuristics HO8 and H10 in Cooperative Reaching, BRDiv ceased to become the best pest-performing teammate generation method to improve the robustness of the learner. The same trend is seen in Figure 5c where the learner must collaborats with H01-H12 whose expertisc only spans parts of lasks in the environment, such as processing the ingredients, assembling them into a dish, and delivering it to a serving counter. This echoes with the results of BRDiv when dealing with other Simple Cooking teammate policies generated by other bascline tcammate generation methods, which we discuss in Section 6.3 have generated suboptimal policies. All these results point towards the inadequacy of BRDiv-based generated policies to improve learner robustness when dealing with suboptimal teammates.
14

---

Published in Transactions on Machine Learning Research (05/2023)
Evalustion Cooperative Reaching Against Hearistic Agents
Againss Agents
Evallation in Simple Cooking Again Heunstti Agents
20
Silee 110 we es 1:30
M
Evelyakon Teamiuu Heurbetie Nore.
Eodlunn Teursmen Heure hame
Paaiil www. wwl.i
mill
Af
63
alkiin
14
.401 Aww. 2..
22
All No.
4.00 20 427
27
1/
withe
22
M8
Al
20
83
204
wy' haiie
Training Teammate Generator Algonihim
Training Teammate Generater Algorithm
Teammite Generator Agorithm
(a) Cooperative Reaching
(b) LBF
(c) Simple Cooking
Figure 5: AHT Evaluation Results Against Heuristic-based Teammates. We provide the average returns resulting from the interaction between I1 consisting of heuristic- teammates and the learner, which is trained using PLASTIC Policy (Barrett et al. 2017) and IItrain produced by the evaluated tcammate generation methods. Labels on the x-axis of the heatmap visualisation indicate the teammate generation method used to produce 11train, Labels on the y-axis highlight the heuristics followed by agent policies from [! Within each entry of the healmap, the first number provides the average returns from the collaboratio tion between learners trained with Iltrain generated by the method indicated in the x-axis and teammates following heuristics labelled in the y-axis. The numbers in the parentheses provide a 95% confidence interval of the returns based on teammate generation experiments conducted across five seeds. Figure 5a show the results in the Cooperative Reaching environment where training learner with BRDiv-based teammates produces more robust agents that can deliver higher returns than the baselines, except for interactions against H08 and H10. Mcanwhile, the LBF enviromment results also mirror the findings from the Cooperative Reaching environment, where a BRDiv-based learner yields higher average returns than all baselines except. for interactions against H02. Finally, Figure 5c show the Simple Cooking environment results where BRDiv did not achieve the best performance compared to other methods due its inadoquacy when dcaling with suboptimal tcammate heuristics.
6.5 Behaviour Evaluation
In this section, we provide additional empirical evidence regarding the effectiveness of BRDiv in generating Iraainn for AHT training. First, we show example of IIttain exhibiting superficial differences based on the results of running one of our baseline teammate generation methods in the Cooperative Reaching environment. We then show how BRDiv successfully avoids generating mitainn exhibiting superficial differences, which then leads to improved learner robustness when I is used for AHT training.
An example of I with superficial differences discovered by one of our baseline methods can be seen in Figure Ta In this visualisation, multiple policies in Iraiin move towards the same swwar-providing coordinates. Effective collaboration with these policies can be achieved through the same best-response policy of moving towards policies is reflected in Figure 7b which shows the cross-play matrix resulting from the interaction between the policies in traain and BR (I
reward-providiig grid the teammate moves towards. This commonality in best-response
Training a Cooperative Reaching learner based on train in Figure 7a will not provide a robust learner. This is because certain teammate behaviours are not present in the training set, such as teammates that move towards the lpfer-left or bottom-right reward-providing corners. A better I for training learners in Cooperative Reaching is visualised by the BRDiv-based teammate policies in Figure Sal In this case, I
15

---

Published in Transactions on Machine Learning Research (05/2023)
Coeperat Seathing Acalmet Generated Anwnrs
Evaltarian LBT Againe Geperated
Evaluat on Simple Cooking Agaisst Giene ated Agents
cakilies
dildoo is iiliinn ruliion
is dilee
Dosteeboo Teaminen Greenter Mper Mhe
0o Grventi Aly Aly--mm
clltt is Balos iubiinn
cabsn is Baan
Teamnni
ch
20w
557 37h The hhw
céllion
sub
waee vide video vuen vedern vol Teammate Algeritin
Trariig Teammats ganttm
Algeritmm
(a) Cooperative Reaching
(b) LBF
(c) Simple Cooking
Figure AHT Evaluation Results Against Previously Unseen Generated Teammates. Given I generated by a tcammate generation method, we also report the average returns achieved by the learner when dealing with I[ consisting of leammates generated by the different evaluated leammate generation methods. In the figures above, the labels the heatmap's x-axis, y-axis, and numbers have similar semantics with their respective counterparts in Figure 5 Note that when dealing with Ieeal generated by the same algorithm producing I it is possible that effective collaboration cannot be achieved since [Teval also consists of policies generated through experiments using different seeds from which being used to produce trainn In Cooperative Reaching and LBF, BRDiv-based learners produce higher average returns when dealing with previously unsoen tcammates. For certain teammate generation methods, the difference between a BRDiv-based learner's and its mean returns is even statistically significant. For Simple Cooking, method occasionally struggles to deal with suboptimal teammate policies generated by some teammate generation methods. Further discussions regarding the suboptimality of policies produced by certain methods are provided in Section 6.5
0.75
Mole
035
0.56
0.75
(a) Example of superficial differ- ences between generated team- mates.
(b) Cross-plas matrix between poli- cies generated following Figure 7a
Figure Example of Superficial Policy Differences in Cooperative Reaching. From one of the Ittaain resulting from bascline teammate generation method in our experiments, we example of leammates with superficial dilferences in Cooperative Reaching. Figure 7 show that superficial difference is characterised by different teammate policies that move teammate towards the same reward-providing corner. Since an effective collaboration with teammates having superficial differences can be achieved using the same best-response policy, the cross-play matrix from Figure 7b demonstrates the compatibility of some best-response policies with multiple policies from tuaiin
16

---

Published in Transactions on Machine Learning Research (05/2023)
0.75
0.09
0.4
0.92
0.17
0.00
0.75
Palne
ITtrain generated by BRDiv in Cooperative Reaching.
(b) Cross-play matrix between the policies generated by BRDiv in Fig- ure
Figure 8: Ideal l|train for Cooperative Reaching. An example I generated by maximising BRDiv is provided in Figure Sa Compared to the leammale policies displayed in Figure 7 a robust learner is likely to be produced from training with this 1[train since it contains different policies that move teammates towards all the reward-providigg coordinates in Cooperative Reaching. Teammate policies that move towards the upper left and bottom right corners are specifically better handled if the learner trains against policies visualised in Figure Sa. Following Figure Sb this I[train is characterised by the distinct best-response policies required for effective collaboration against each generated policy.
0.8
0,04
0:
0317
07
0.50
8:36
622
0,07
0.11
8:25
0.2
0:00
0.002
02
0.1
5
(a) Trajectories produced by two ran- domly sampled policies from 11 1ltain
(b) Cross-play matrix from 11srain pro- duced by BRDiv.
Figure 9: Iitrain Generated by Optimising BRDiv in LBF. Assuming that the level one agent is the leammate, example trajectories from two randomly sampled policies generated by optimising BRDiv for LBF are displayed in Figure Da In this visualisation, each sequence of the same coloured arrows starting from the teammate's s position corresponds to a trajectory of a single policy from I Different policies in I|trinn specifically correspond the distinct orderings that teammate may follow collect objects in the environment. Since effective collaboration with leammates that follow a specific object collection ordering requires best-response policies that follow the same object collection ordering, the best-response to every policy in I[ttain is distinct and incompatible for collaboration with other policics. This results in the cross-play matrix displayed in Figure 9b
produces a more robust learner by equipping it with a more comprehensive set of strategies against teammates moving towards any reward-providing grid. We can consistently find this desirable ITtrain since each policy I[tainn requires different best-response policies, which makes it highly likely to be discovered by optimising BRDiv.
17

---

Published in Transactions on Machine Learning Research (05/2023)
B
(a) Tomato Preparation.
(b) Carrot Preparation.
(c) Food Assembly & Delivery.
Figure 10: Teammate Policy Generated By Optimising BRDiv and Its Best-Response Policy. From left to right, we show an example of a leammate policy alongside its best-response policy generated by optimising BRDiv. The generated policy and its best-response policy learn to quickly divide the ingredient preparation dish assembly, and delivery tasks between themselves. In Figure [0a] the generated policy collects the tomato and puts it on the table another teammate closer to the knife and positioned on the opposite side of the kitchen can retrieve and chop it. After chopping the tomato, Figure TOb then shows that the best-response policy learns to move towards the carrot and puts it on the table so that the agent controlled by the generated policy can collect and blend it. After the generated policy puts the blended carrot on the table, the rest-response policy collects the plate and carrot to combine with its chopped tomatoes. The best-response policy eventually delivers this combined food to the serving counter as seen in Figure 10c
Optimising BRDiv also enables the discovery of a I that encourages the emergence of robust learners in the LBF environment. As seen in Figure Da. each policy in I generated by optimising BRDiv for LBF corresponds to the distinct orderings that an optimal agent may take to collect objects in the environment. Since any optimal or near-optimal teammate should follow one of the six possible orderings when collecting objects, the discovery of I containing policies that follow each ordering prevents the learner from not having adequate strategy to deal with optimal ncar-optimal tcammates. As in the with Cooperative Reaching, note that the discovery of good quality leammate policies for LBF is made possible by each policy in Irtrain requiring different best-response policies, which makes it likely to be discovered by optimising BRDiv.
Besides highlighting why optimising BRDiv facilitates improved learner robustness in Cooperative Reaching and LBF, analysing the behaviour of policies generated by our method and the baselines also provides insights into why method does not yicld the most robust learner in Simple Cooking. As displayed by Figure [0] teammate policies generated by optimising BRDiv highly optimal. The generated policies alongside its best-response policy quickly learn to divide and execute the available subtasks among themselves. Between different generated policies in I|trin,, difference emerges due to different task assignments between agents and different orderings to complete the subtasks. In general, URDiv-basee generated policies and their best-response policies tend finish an episode of Simple Cooking in 17-20 timesteps. Learning from such highly optimal policies makes learner unprepared when facing highly suboptimal policies during evaluation.
This result from BRDiv highly contrasts with the results from optimising alternative diversity metrics tested in this work. A detailed breakdown of the number of timesteps required by each compared teammate generation method to solve Simple Cooking in self-play is provided in Table 2 The better-performing basclines for generating robust learners for Simple Cooking against heuristic-based Leammates particularly produce suboptimal teammate policies that solve the enviromment in 30-195 timesteps. Throughout interaction, Any-Play and TrajeDi-basod policies often exhibit suboptimal behaviour such as (i) going back and forth between putting an item on the counter and retrieving it again or (ii) stopping working on subtasks and doing nothing. The availability of such suboptimal policies in Iaaii makes the learner more prepared to
18

---

Published in Transactions on Machine Learning Research (05/2023)
Table 2: Required Timesteps to Solve Simple Cooking in Self-Play. The number of timesteps required by each method to solve Simple Cooking provided in their respective entries in the second row.
BRDiv 17-20
Ala-Play 30-35
TrajeDio
TrajeDi025
TrajeDi05
TrajeDi075
TrajeDil 20-28
Independent
18-25
65-158
85-233
53-161
18-29
complete a task on its own in case teammates are performing poorly in the task. While this inability to deal with highly suboptimal teammates presents potential research directions to further improve BRDiv, note that such suboptimal teammates are rarely encountered in many realistic applications of AHT. As originally formulated by (Stone et al., 2010), encountered teammates are normally assumed to be capable of achieving specific return threshold at the given task.
7 Conclusion & Future Work
In this work, we discussed the importance of generating a collection of training teammate policies, IItrain that require different best-response policies to improve the robustness of an AHT agent. To achieve this, we proposed a teammate generation method that optimises BRDiv, a diversity metric designed to prevent the emergence of superficial differences between policies in Mraain Based on a comparison against TrajeDi (Lupu et al. 2021), ; Any-Play (Lucas & Allen 2022), and a baseline that independently trains different teammate policies via MARL, our experiments show that optimising BRDiv achieves higher average returns when dealing with near-optimal previously unseen leammate policies. At the same time, we also see a smaller variance in the returns achieved by learners trained with I|taann produced by optimising BRDiv.
The conducted analysis of the generated teammates' behaviour showed that optimising BRDiv avoids generaling leammates with superficial differences. At the same time, I generated by optimising BRDiv covers a comprehensive set of reward-maximising tcammate behaviours. Training against this set of teammates eventually produced teammates that can perform a wider range of strategies to collaborate against previously unseen teammate policies.
Although results in the teammate generation experiments show that optimising BRDiv can generate leammate policies that require different strategies for effective collaboration, note that this is not the only type of diversity displayed by decision decision-making agents in real-world problems. In many applications of AHT, a learner also has to deal with teammates that vary in their ability to maximise the teams' returns. For example, even with different teammates that prefer a specific role such as being a striker, we see a wide range of skill levels between polential leammates a pick-up soccer game. A leammate's ability may range from having the skills of an amateur player to possessing elite skills displayed by top-division professional players. Currently, this diversity cannot be discovered solely based on optimising BRDiv. The first term the right-hand side of Equation 0 encourages the creation of teammates with near-optimal policies when we optimise BRDiv. By only training learner against teammates generated by optimising BRDiv, this limitation potentially results in a learner yielding suboptimal returns when dealing with teammates with a low skill level. The results of our experiments in the Simple Cooking environment also confirmed the need for further developments in this direction.
The proposed method to optimise BRDiv also faces challenges when dealing with problems other than two- two-player games. In many real-world problems such as those addressed in open ad hoc teamwork (Rahman et al. 2021), generating team of multiple agents with different policies is desirable. While our proposed optimisation method can be modified to generate a team of tcammates, many such teams must be generated at once to improve the robustness of the learner. After all, the number of generated training teams must. match the exponential increase in the space of possible team confiuraatons. Since training agents via MARL may require millions of experiences even simple domains, the computational resources required by our proposed method to generate large collection of teams can quickly grow impractical as the size of a generated team increases.
19

---

Published in Transactions on Machine Learning Research (05/2023)
Acknowledgments
This research received financial support from various sources. A.R. received funding from the Edinburgh Enlightenment scholarship. E.F. was supported by the United Kingdom Research and Innovation (grant EP (EP/S02308811 /1,, EPSRC Centre for Doctoral Training in Robotics and Autonomous Systems (RAS). I.C. and S.A. were recipients of funding from the US Office of Naval Research (ONR) via grant N00014-20-1-2390, and the Google Cloud Research Credits program award.
References
Rishabh Agarwal, Max Schwarzer, Pablo Samuel Castro, Aaron C Courville, and Marc Bellemare. Deep reinforcement. learning at the edge of the statistical precipice. Advances in Newral Information Processing Systems, 34, 2021.
Stefano V. Albrecht and Subramanian Ramamoorthy. A game-theoretic model and best-response learning method for ad hoc coordination in multiagent systems. In Proceedings of the 12th International Conference on Autonomous Agents and Multiagent Systems, St. Paul, Minnesota, USA, May 2013.
Stefano V. Albrecht and Peter Stone. Autonomous agents modelling other agents: A comprehensive survey and open problems. Artificial Intelligence, 258:66 95, 2018. DOL: 11t.://...tinn.211....... 2018.00.00.
Stefano V. Albrecht, Jacob W. Crandall, and Subramanian Ramamoorthy. Belief and truth hypothesised behaviours. Artificial Intelligence, 235:63 94, 2016.
Samuel Barrett, Noa Agmon, Noam Hazon, Sarit. Kraus, and Peter Stone. Communicating with unknown teammates, In Proceedings of the 2014 International Conference on Autonomous Agents and Auett-Agent Systems, AAMAS '14, pp. 1433 1434, Richland, SC, 2014. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450327381.
Samuel Barrett, Avi Rosenfeld, Sarit Kraus, and Peter Stone. Making friends on the fly: Cooperating with new teammates. Artificial Intelligence,
2017.
Daniel S Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4):819 8409, 2002.
Filippos Christianos, Georgios Papoudakis, Arrasy Rahman, and Stefano V. Albrecht. Scaling multi-agent reinforcement learning with selective parameter sharing. In International Conference Machine Learning (ICML), 2021.
Benjamin Eysenbach, Abhishek Gupta, Julian Ibarz, and Sergey Levine. Diversity is all you need: Learning skills without reward function. In International Conference on Learning Representations, 2019. URL
https:://oenrrevie http:///pppppeeeeeeeeeeeeeertras
Zhang- Chang-Wei Wei Hong, Yun Shann, Shih- Yang Su, Yi-Hsiang Chang, Tsu-Jui Fu, and Chun-Y Yi Lee. Diversity- driven exploration strategy for deep reinforcement learning. Tn S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018. URL https://proceeeiing. http://a.....rrppp...c////////// pdf
Hengyuan Hu, Adam Lerer, Alex Peysakhovich, and Jakob Foerster. "other-play" for zero-shot coordination.
In International Conference on Machine Learning, pp. 4399 4399-4410, PMLR, 2020.
Max Jaderberg, Valentin Dalibard, Simon Osindero, Wojciech M. Czarnecki, Jeff Donahue, Ali Razavi, Oriol Vinyals, Tim Green, lain Dunning, Karen Simonyan, Chrisantha Fernando, and Koray Kavukeuoglu. Population based training of neural networks. CoRR, abs/1711.00944 2017. URL http://arxiv.orz org/
hit/1711109946
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 141: 6980, 2014.
20

---

Published in Transactions on Machine Learning Research (05/2023)
Ang Li, Ola Spyra, Sagi Perel, Valentin Dalibard, Max Jaderberg, Chenjie Gu, David Budden, Tim Harley, and Pramod Gupta. A generalized framework for population based training- In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, KDD 2019, pp. 1791-1799, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450362016. doi:
10.1145//299500088336666 URL htttps://dde http:////oo.org//... 1111///9.1145//995000.0 3330649
Chenghao Li, Tonghan Wang, Chengjie Wu, Qianchuan Zhao, Jun Yang, and Chongjie Zhang. Celebrating diversity in shared multi-agent reinforcement learning. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan (eds.), Advances in Newral Information Processing Systems, 2021. URL https:
//openreview.co http://www...tttoooooooooo---SE.
Xiangyu Liu, Hangtian Jia, Ying Wen, Yujing Hu, Yingfeng Chen, Changjie Fan, ZHIPENG HU, and Yaodong Yang. Towards unifying behavioral and response diversity for open-ended lcarning in zero-sum games. In M. Ranzalo, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan (eds.), Adounces in Newral
Information Processing Systems, volume 34, pp. 941 952. Curran Associates, Inc., 2021. URL https ;
Keane Lucas and Ross E Allen. Any-play: An intrinsic augmentation for zero-shot coordination. arXiv preprinl ar 2rx:::2201.12/ 12436, 2022.
Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In Proceedings of the 38th International Conference Machine Learning, 2021.
Anuj Mahajan, Tabish Rashid, Mikayel Samvelyan, and Shimon Whiteson. Maven: Multi-agent variational exploration. In Advances in Neural Information Processing Systems, pp. 7611-7622, 2019.
Reuth Mirsky, Ignacio Carlucho, Arrasy Rahman, Elliot Fosong, William Macke, Mohan Sridharan, Peter Stone, and Stefano V Albrecht. A survey of ad hoc teamwork research. European Conference on Multi-Agent Systems (EUMAS), 2022.
Volodymyr Muih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukeuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937. PMLR, 2016.
Georgios Papoudakis, Filippos Christianos, and Stefano Albrecht. Agent modelling under partial observability for deep reinforcement learning. Advances in Neural Information Processing Systems, 35, 2021a
Georgios Papoudakis, Filippos Christianos, Lukas Schäfer, and Stefano V. Albrecht. Benchmarking multi- agent deep reinforcement learning algorithms in cooperative tasks. In Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS), 2021b. URL http://arxiv. wwtrrivv.orrr/ooo org/abss/
07869
Jack Parker-Holder, Aldo Pacchiano, Krzysatof Choromanski, and Stephen Roberts. Effectiive diversity population based reinforcement learning. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS'20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Deepak Pathak, Pulkit Agrawal, Alexei A. Efros, and Trevor Darrell. Curiosity-driven exploration by self-supervised prediction. In Proceedings of the 34th International Conference on Machine Learning Volume 70, ICML/17, pp. 2778 2787. JMLR.org, 2017.
Arrasy Rahman, Niklas Höpner, Filippos Christianos, and Stefano V. Albrecht. Towards open ad hoc teamwork using graph-baue based policy learning. In International Conference on Machine Learning, volume 139. PMLR, 2021.
Arrasy Rahman, Ignacio Carlucho, Niklas Höpner, and Stefano V. Albrecht. general learning framework for open ad hoc teamwork using graph-based policy learning. arXiv preprint ar Xrxv:22000.0 210.022000 2022.
21

---

Published in Transactions on Machine Learning Research (05/2023)
Lukas Schäfer, Filippos Christianos, Amos Storkey, and Stefano V. Albrecht. Learning task embeddings for teamwork adaptation in multi-agent reinforcement learning. arXiv preprinl ar Xrx::2007 02249, 2022.
Peter Stone, Gal A. Kaminka, Sarit Kraus, and Jeftrey S. Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. In AAAT Conference on Artificial Intelligence, pp- 1504-1509, 2010.
Zhenggang Tang, Chao Yu, Boyuan Chen, Huazhe Xu, Xiaolong Wang, Fei Fang, Simon Du, Yu Wang, and Yi Wu. Discovering diverse multi-agent strategic behavior via reward randomization, 2021. URL /ttpss//arxiv... http:////wwii..oo////////// ptgg/ba//103.....m
Alexander Vezhnevets, Yuhuai Wu, Maria Eckstein, Rémi Leblond, and Joel Z Leibo. Options as responses: Grounding behavioural hierarchies in multi-agent reinforcement learning. In International Conference on Machine Learning, pp. 9733 -733-9742. PMLR, 2020.
Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. Grandmaster level in StarCraft. TI using multi-agent reinforcement learning. November 2019. ISSN 1476-4687. doi: 00088//415500011-1722444 URL
Nature, 375 attpps//wwer http::///ww..aauue..e nature. www/ cttpr///rrriieee//11---0000--1-- Number: 7782 Publisher: Nature Publishing Group.
Sarah A. Wu, Rose E. Wang, James A. Evans, Joshua B. Tenenbaum, David C. Parkes, and Max Kleiman- Weiner. Too many cooks: Coordinating multi-agent collaboration through inverse planning. Topics in Cognétive Science, n/a(n/a), 2021, doi: https:/ /ttp:////oooooo11111111111111111 12525. URL httLittttss//mmllebrrrry wiley.coo//bii/bb//000 http/////aaaab//..////....///// 12525
Dong Xing, Qianhui Liu, Qian Zheng, and Gang Pan. Learning with generated teammates to achieve type-frec ad-hoc leamwork. In IJCAI, pp. 472 478, 2021.
Chao Yu, Jiaxuan Gao, Weiling Liu, Bo Xu, Hao Tang, Jiaqi Yang, Yu Wang, and Yi Ming Wu. Learning zero-shot cooperation with humans, assuming humans are biased. ArXiv, abs abss/2302.0000 03330.00605, 2023.
Kaiqing Zhang, Zhuoran Yang, and Tamer Başar Multi-Agent Reinforcement Learning: A Selective Overview of Theories and Algorithmss, pp. 221-384. Springer International Publishing, Cham, 2021. ISBN 978-3-030- 60990-0. doi: (0.100779888---00--0009-- 12. URL httpptt///o...orgee/ - 000-0300-9990000
Luisa Zintgraf, Sam Devlin, Kamil Ciosek, Shimon Whiteson, and Katja Hofmann. Deep interactive bayesian reinforcement learning via meta-learning. arXiu preprint vrXio:2101.0880, 2021.
A BRDiv Pseudocode
We complete the description of our method by providing a pseudocode for the teammate generation process undergone in BRDiv, shown in in Algorithm [ An essential part of Algorithm [ is a call to the COM- PUTE LOSS function that evaluates the loss functions minimised by BRDiv. How BRDiv utilises the gathered self-play and cross-play experience to compute the minimised loss functions is then described Algorithm 2
PUTE_LOSS
B Heuristic-based Teammates
As we mentioned in Section 6.2 we use Ivvl consisting of heuristic-based policies evaluate the methods used in our experiments. The details of heuristics followed by each policy for the Cooperative Reaching
22

---

Published in Transactions on Machine Learning Research (05/2023)
Algorithm 1 BRDiv-based Teammate Generation Process
Require:
Number of training episodes, Neps.
Episode length, T.
Update period, Lupdate.
Number of generated teammate types, K.
Initial population actor network parameters, θ (0,,02,- 02,0000.0.00 0x- 0BR ,0 00R Initial centralised critic parameters, ø. Target centralised critic parameters, Ø.
0BR) ).
Learning Rate, a.
Target network update coefficient, a. Environment for SP and XP interaction, en & en
for Peps do
++0
DSP DXP f,,f
4: IDSP ~ Uniform( (1,.... K})
Sample Population ID for SP Sample Population ID for XP 1XP 2XPP from en and envXP respectively.
5: ID .D Uniform (f (f(i. j) E El. i
Observe H = =op] 1.PP 2PP and H (o
.0
H..PP. H2SPP Hoo HDXPP .HP.P (op 1 b fo }. 10o
LSP 2SP
1.PP
10o
xxP)
while < T do
// Self- Data Collection 1.SP
10:
at ( HiSPP
1.SP
2SP and dz
~
BR
(1
2.SP
1
SP
ID&r
TH4 HPP Sn (H, SP
11:
a
12:
DSP + ( ar SP
H4, IDSP)
13:
// Cross-Play Data Collection
14:
dr 1XP
a4
1H ;0DNP and d
I.XP
2,XP
BR
(1
2,XP
HH2 ;
XP H t env en a
15:
DXP + D a²"", 2it H., ID "DD"")
XP
16:
17:
if t mod tupdate 0 then
18:
// Parameter Update
(DSP, DXP) + COMPUTE COMPUTE_LOSS(DE DNP O,
19:
20:
for 0, E 2 do
0, +- GRADIENT o a, V Le, (DSP, D2P))
21:
22:
end for
Φ - GRADIENT _ENCENTTEEEETT.C. VoL (D
23:
24:
① -- (1 ā)Φ + +ā
D + (
25:
26:
DXP 0
27:
end if 28: + t+1 1
29:
end while
30: end for
31: Return:
environment are provided in Scction B.1 policies in IIvaal for LBF, while Section B.3 outlines the heuristic followed by agents in the simple cooking environment.
Mcanwhile, Scction B.2 outlines the heuristics followed by the
23

---

Published in Transactions on Machine Learning Research (05/2023)
B.1 Cooperative Reaching
For Cooperative Reaching, we implement 11 heuristics part of I]evall, Each heuristic differs from others in coordinates to move towards. Some heuristics also
terms of their way of selecting which encourage teammates to follow the Iearner towards one of the existing roward-providing coordinates. The details each heuristic used in Cooperative Reaching are provided below:
Heuristic H01. This heuristic selects the action that gets teammate closer to the closest. reward-providing coordinate.
Heuristic H02. This heuristic selects the action that gets a teammate closer to the furthest. reward-providing coordinate from its initial position at the beginning the episode.
Heuristic H03. A teammate under this heuristic moves towards the closest optimal reward- coordinate.
Heuristic H04. H4 moves an agent towards the furthest optimal reward-providing coordinate from & leammate's initial location in an episode.
Heuristic H05. Same H4, except that the learner only considers the suboptimal reward-providing coordinates instead of the optimal ones,
Heuristic H06. Same H5, except the leammate goes towards the closest suboptimal reward- providing coordinate.
Heuristic H07. At the beginning of the episode, agents under this heuristic randomly select reward-providing coordinate and move towards it.
Heuristic H08. This heuristic moves teammate towards the reward-providie coordinate closest to its counterpart agent's location.
Heuristic H09. Same H8, but only optimal reward-providing coordinates are considered as the teammate's destination.
Heuristic H10. This heuristic moves the teammate towards its counterpart agent's location.
Hcuristic H11. This hcuristic always randomly selects an action from the tcammate's possible actions.
B.2 Level-Based Foraging
Like Cooperative Reaching, we create diverse teammate heuristics requiring a learner adapt their policies to achieve optimal collaboration. The ten heuristics used for LBF generally correspond to different ways of deciding the ordering to collect objects scattered in LBF's grid world. Details of each heuristic are provided below:
Heuristic HO1. The teammate attempts collect whichever object is closest to its current location.
Heuristic HO2. At each timestep, the teammate computes the midpoint between the learner and its location. This teammate then attempts to collect whichever object is closest to this midpoint.
Heuristics H03-HO8. For heuristics H03 H08, we assign distinct random index from 2, 3} to each object at the beginning of each episode. Heuristics HO3-HOS then collect the objects according to one of the 6 distinct possible orderings of the object index.
Heuristic H09. The teammate always attempts collect food closest to the learner's location.
Heuristic H10. At the beginming of each episode, H10 identifies the object furthest from location and attempts to collect it. Each time its target item is collected, H10 then attempts to collect the remaining object at the furthest distance from the current location of the controlled teammate.
24

---

Published in Transactions on Machine Learning Research (05/2023)
B.3 Simple Cooking
As the layout of our Simple Cooking is ring, consider two movement directions around the ring: clockwise and anti-clockwise. Each heuristic agent has a goal, such as "seek and process the nearest food. Once their goal has been completed, the heuristic agent finds counter without any tools on it and stands on the empty space closest to the said counter.
Seeks the nearest. food in the clockwise direction, picks it up, and continues move clockwise to the appropriate food processing counter, where it processes the food.
Heuristic HI:
Heuristic H2: Seeks the nearest food in the anti-clockwise direction, picks it up, and continues to move anti-clockwise to the appropriate food processing counter, where it processes the food.
Heuristic H3: Takes the shortest path to the nearest food item, picks it up, and continues to lake the shortest path to the appropriate food processing counter, where it processes the food.
Heuristic H4: Soeks the furthest away food in the clockwise direction, picks up, and continues move clockwise to the appropriate food processing counter, where it processes the food.
Heuristic H5: Seeks the furthest away food in the anti-clockwise direction, picks it up, and continues to move anti-clockwise to the appropriate food processing counter, where it processes the food.
Heuristic H6: Same as Heuristic H3, except 25% of the time, the agent takes a uniform random action.
Hcuristic H7: This heuristic seeks the nearest processed food in the clockwisc direction. then picks up the processed food and checks whether the plate on one of the counter or not. If the plate on one of the counters, the agent moves clockwise to the plate in order put the processed food. Otherwise, the agent goes towards the serving counter place the food. Once there no processed food to move, this agent moves clockwise to stand in front of an outer counter without any items top.
Heuristic H8: This heuristic is similar to H7 except that agents under this heuristic always move in an anti-clockwise direction.
Houristic H9: This heuristic is similar to houristics H7 and H8 except that agents under this heuristic always decide its clockwise or anti-clockwise movement. based on the shortest distance between its target object or location.
This heuristic is similar to heuristics H7 except that agents under this heuristic will immediately put the processed food the serving counter.
Heuristic 10:
Heuristic H11: This heuristic is similar to heuristics H8 except that agents under this heuristic always put the retrieved processed food on the service counter.
Heuristic H12: This heuristic is similar to heuristics H9 except that agents under this heuristic
will immediately put retrieved processed food on the service counter.
C
Experiment Hyperparameters
This section provides details of the hyperparameters and neural network architectures used in our teammate generation experiments.
When optimising BRDiv, we run 32 parallel threads collect self-play experiences during training. Meanwhile, the remaining methods use 160 parallel threads to gather self-play experiences used during their teammate generation process.
25

---

Published in Transactions on Machine Learning Research (05/2023)
Aside from the threads used to gather self-play experiences, we 128 parallel environments to collect cross-play experiences when optimising BRDív.
All evaluated methods have their actor and critic networks updated every 8 timesteps.
is set to 0.99.
The generated actor networks alongside the critic network are trained using Adam optimiser (Kingma & Ba 2014) with learning rate of 14-4.
We clip the gradients of the model so that it always lies between and 1.
Each actor network corresponding to policies in the generated Il and (I( are implemented as multilayer perceptrons. The size of these networks for each environment is detailed below:
Cooperative Reaching. The model comprises of four hidden layers with 128, 256, 256, and 128 neurons respectively.
LBF. The model comprises of two hidden layens, each consisting of 128 neurons. Simple Cooking. Our network for this environment. has two hidden layers with each layer having 256 neurons.
We associated different weights to the optimised loss functions when generating I[train using our proposed mothod, TrajeDi, Any-Play and the independent bascline. The weights of cach loss function optimised by these methods are detailed below:
For all methods, the critic loss function for SP data is also set 1.0. Meanwhile, BRDiv assigns weight of 1.0 to the loss function that minimizes the critic loss function following cross-play interaction data.
For BRDiv, the weights of the losses optimised for training the actor networks set to 25.
The Jensen-Shannon Divergence term maximised by TrajeDi is given weight of 10-3, We arrive at this value after finding the largest possible weight from {10-1, 1002 10-3, 11-0:110-11111111111111111111111 that still ensures every policy in I to achieve optimal performances in the enviromment when collaborating with its associated best response policy.
The weights of Any-Play's 's intrinsic reward to maximise diversity between populations is tuned in the same way as how tuned the Jensen-Shannon Divergence weights for TrajeDi. This results in the intrinsic reward weights of 10-2, 10-3 and 10-3 for Cooperative Reaching, LBF, and Simple Cooking.
The classifier Any-Play uses to compute the intrinsic rewards uses the same architecture of other methods' critic networks. The term associated with the supervised learning loss utilised to train this classifier is also set to 1.
For TrajeDi, Any-Play, and the independent baseline, the weights associated with the term that maximises the self-play performance between a policy in I and their associated best response policies is set to 1.
We also use = for the polynomial weight weighting algorithm for PLASTIC Policy, which is the algorithu we use for our AHT experiments.
26

---

Published in Transactions on Machine Learning Research (05/2023)
Algorithm 2 Loss Computation
Require:
Self-play and cross-play data, DSP & DXP.
Population actor network parameters, O.
Centralised critic parameters and target centralised critic parameters, & & 5. DNP (,
function COMPUTE (OAMPUTE_LOS(DOOO
Istart first time in the bulfers DSP DXP
tend f- latest time in the buffers D DDP
4: Virggt +- VHH SP
Fead SP ID :))
5: CSP +-
D. Compute Sel(-Play Critic Loss
6:
for t= bend to Estart do
Vpredd f V(H ,IDP .IDPP :))
7:
if episode terminates at
Viarget
SP
+ y Vargeet
otherwise.
1
S
t
CSP
+
(V
pred Viarget)
2
10:
end for
11:
Viarett f- (HH
Foud
IPP IDD :6) XP
12:
LXP
+-
Compute Cross-Play Critic Loss
13:
for t= bend to tstart do
Vpred t V(HPP, .D ID XP :))
XP
14:
rXP.
if episode terminates at t
15:
Viargtt f
xXP + + +Varget:: otherwise.
1
16:
CXP
&
CXP 2
+ PPeedd Viarget) )2
17:
end for
18:
hontstrap
SHHSP
Lend +1 SP .IDSP,
19:
Compute Self-Play Actor Loss
20:
for 1= bend to lstart do
21:
Muaseline + TO XP MATRIX
(4..). E 1, NN))..
SP
if episode terminates at f otherwise.
22:
Voootstrp
ri Woootosrrpp
23:
Mpred f- Mbaseline
24:
M Mpred,ID** .IDSP Vhootstrap SPP
Replace matrix element of interacting populations
25:
CSP
(a |
1.SP
SP
(ppar BB I IDar (BRDivv
2,SP 22SP @BR
BRDiv (ADDv(Masslline))
26:
end for
27:
Vhootstrap
XH
lend lend +1 XPXP IDXP, :
28:
LXP +
Compute Cross-Play Actor Loss
29:
for lend to lstart do
30:
Mbaseline TO XP 1AP_MMARIXX (EVRRIXX(VVEEEE ([V(H"" (ijjo))i,.j I,....NN) NN) if episode terminates at
r XP
31:
Vouotstrap
XP
Vbootssrrap
otherwise.
32:
Mpred +- Mbaseline Mpred.IDXP IDNP + Vbootstrap
33:
Replace matrix element of interacting populations
34:
CXP
CXP
1.P 1.X !H log(T(a(
I.PP, 1DRN BZ 2 2.XP
222.P. OBR (BBD))))RRDi(Mppeee)
(Div(Maasolling end for
35:
36:
Return: CSP LXP
Lg + CXP
37: end function
27


Reviews:
[{"review_id": "CUoCjZecpc", "reviewer": "Reviewer_GwjL", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": " I really enjoyed reading this paper.\n\n# Strengths\n\n1. The manuscript is well written. It lays down enough background such that an expert reader outside of the standard ad-hoc MARL literature (such as myself) can appreciate the imposed constraints over BRDiv, and the various architectural decisions.\n\n2. The BRDiv method is clever, is ultimately a simple objective that can be applied to a large set of MARL algorithms, and seems to be well justified from both intuition-wise as well as from a theoretical perspective.\n\n3. The authors have gone above and beyond in evaluating BRDiv in the chosen environments, and I really commend them in how clear Section 6 is. I wish RL papers were generally setup in such manner.\n\n# Weaknesses\n\n1. I found it puzzling that the authors chose to implement their own toy environments. According to the background section, there exists plenty of literature on ad hoc learning, and that includes many environments. Considering that the manuscript essentially argues for improving a particular kind of diversity (i.e. reducing \"superficial differences\" in policy space), it would have been good to see this an improvement in this diversity metric translating into an improvement over common ad hoc environments.\n\n2. The new diversity metric / requirement is a key concept in this paper, but its presentation is a bit lacking. It is quickly introduced from a high level perspective in Section 1, in which Figure 1 does most of the heavy lifting. However, the manuscript afterward tends to assume its importance without necessarily arguing for it (outside of the positive empirical results). In particular, the \"superficiality\" in policy space is quite narrowly defined, and seems to be highly dependent on the environment (or even subjective!). It feels like a broader discussion and/or presentation might be necessary here.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents a method for automated teammate policy generation in ad hoc teamwork scenarios.\nRecent approaches have attempted to improve the robustness of the learner by optimising information-theoretic diversity metrics to generate teammate policies, however resulting in lack of useful diversity due to naive objectives (generating diversity with \"superficial differences\").\n\nIn contrast, the proposed method optimizes the Best-Response Diversity (BRDiv) metric, which measures diversity using reward distributions across a population of training agents, setting the objective to minmax certain teammate policies vs others in the population.\n\nBRDiv is tested in a few simple environments, demonstrating that learn ad hoc agents are better at dealing with a set of unseen teammate policies, as well as being qualitatively diverse.\n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Overall, I enjoyed the manuscript, and I don't have any strong change requirements. However:\n\n1. I would *love* to understand the reason behind the lack of reutilization of previous ad-hoc environments (even if only some of the ones contained in the literature presented within the manuscript), or even some common MARL ones, since it feels like it should be generally possible to modify them to be compliant to the AHT setting.\n\n2. I would like to see a better intuitive explanation for why it is necessary and worthwhile to minimize this particular interpretation of \"superficial differences\".\n", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "06CBJclgS7", "reviewer": "Reviewer_8v61", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall, I really love the topic of ad-hoc teamwork, and I think that the key motivation of having meaningfully diverse policies to train with is very appealing. \n\nThe paper gives an overview of the topic of ad-hoc teamwork from its early days, which I find very helpful. Similarly, the description of the method is quite clear (with the technical figures adding a lot of value). I think the relation and differences with existing approaches could be presented in more depth (see below). Some of the figures are a bit cumbersome (see below), or could be simplified. \n\nThe empirical analysis is quite convincing but limited in terms of coverage and the simplicity of the environments (see suggestions below - mostly adding more environments, and having slightly more complex ones to show that the approach is not tailored to few specific domains). \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper deals with the Ad-Hoc teamwork framework, where a learner must interact and collaborate with previously unencountered teammates, without any prior coordination. Existing work already addresses this challenge using methods that train the learner given a diverse set of teammate policies (e.g. handcrafted ones tailored manually for the demain). \n\nHowever, such a solution is not scalable, as such a manual intervention is required for each new domain to be captured. One possible solution is applying information theoretic metrics of diversity when generating teammates to train with, but such a simplistic metric can result in teammates with superficially different behaviors that are in reality quite similar, and not adding much robustness to the learner. \nThe authors propose a teammate generation algorithm which optimizes for a diversity metric called BRDiv, best-response diversity. This metric measures diversity based on the compatibility between teammate policies based on the joint returns. \n\nThe authors also provide an empirical evaluation in an environment where multiple valid coordination strategies are present, and contrast them with baselines based on approaches that optimize for information theoretic diversity (and also with the case where no optimization for diversity is done). The results show that the approach results in a more meaningfully diverse set of teammates to train with, hence improving the learners performance and robustness. ", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "In terms of presentation, I do feel like the authors could have done a better job at contrasting their approach with existing methods in a better way. A table comparing the various approaches would have been very useful (e.g. what the metric measures, how it is used, the training pipeline etc.). In particular, MAA2C based optimization with a metric, which is a key building block, should receive more attention. \n\nAlso, figures 4+5 is very busy, and perhaps there is an easier way to convey that message? \n\nIn terms of the empirical analysis, I think the environments presented here already give enough evidence of the value of the approach. However, including more settings and more difficult tasks could help show whether this principle is generally applicable. For instance, you have one environment that is akin to overcooked, but quite a simplistic one. A more difficult environment would be very useful. Possible games with many more agents? Could further multiagent AI gyms be used here? \n", "broader_impact_concerns": "I spotted no broader impact concerts. \n", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "317LkF9VnK", "reviewer": "Reviewer_XuvG", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "# Strengths\nThe authors are tackling an interesting, and challenging, problem that does not receive much attention, but will likely start receiving more as the community develops more capable agents.\nThe method proposed by the authors makes sense, and I am reasonably convinced it generates diverse policies that benefit the learning agent. The authors performed a large set of experiments that help in showcasing the advantages of their proposed method.\n\n# Weaknesses\nMy main concern with this paper is in its clarity. There are many points (detailed below) where the writing could be improved.\nI also have some concerns with the way the baselines were run, which are detailed below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper proposes a methodology for generating diverse teammates so as to obtain improved robustness in the resulting learning when collaborating with new teammates. Diversity is obtained by optimising the Best-Response Diversity (BRDiv) metric, which is also introduced in this paper. The authors evaluate their approach in a number of environments with varying coordination strategies.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "# Comments / suggestions / questions\n1. Throughout the paper you refer to the \"best-response policy\" but it's not clear what you mean by that, nor how it is learned.\n1. Throughout the paper you use the term \"superficial differences\", but it's not clear to me what you mean by that. For example, in Figure 1 you say (a) and (b) only have superficial differences, but they seem quite different to me.\n1. In the caption of Figure 1 you say \"an AHT learner will only acquire the skill to pass...\". This suggests this is the case for _all_ learners of this type, which I doubt is true. Also, why is learning the passing skill bad? Same issue in the main text in the introduction when you say \"A learner trained to collaborate with these policies will only acquire expertise in passing...\".\n1. I'd suggest revisiting Figure 1 and the example provided. It's not clear what message you're trying to convey.\n1. In the introduction you say \"they do not encourage improved learner robustness\", please be more specific as to what kind of robustness you're referring to.\n1. The second to last sentence in page 2 is hard to parse. \"teammate policies\" appears twice in the sentence but it's not clear they are referring to the same thing. Please expand and clarify more.\n1. In the top of page 3, it's not clear what you mean by \"forcing\". It suggests you're artificially making things better for your method?\n1. In the first sentence of section 3.2, how does $K$ relate to $N$?\n1. In the second paragraph of section 3.2, is the \"position\" of $\\pi^i$ (in terms of action selection) is fixed throughout training? Order is important, so stating that assumption would be best.\n1. In equation (1), I assume the size of $\\mathbf{\\pi^{-i}}$ is $N-1$?\n1. In the last paragraph of page 4, when you say \"feasible teammate policies\", are these evaluated with the learner $\\pi^i$ or not? If yes, how ($\\pi^i$ hasn't been learned as far as I understand)? If not, what is used instead? Just a regular learner? Please clarify.\n1. In the first sentence of section 4.1 you say \"a suitable $\\Pi^{train}$ for AHT...\". It's not clear what this means, nor why we want to avoid it. It's phrased weirdly, so it's hard to parse. The equations that come after do clarify things, so lean on that when explaining.\n1. In the second equation in section 4.1, is $a_t$ the combination of $a^1_t$ and $a^2_t$?\n1. In the last paragraph of section 4.1, when you say \"by the existence of a common...\", I'd say your method is in a sense more superficial since it's only comparing based on returns. Thus, two very different policies, but with the same returns, will be classified as \"similar\", when they probably shouldn't.\n1. In the first paragraph of section 4.2, when you say \"Extending our proposed diversity metric...\", I'm not sure about this, in particular with respect to the ordering assumption I mentioned above.\n1. In equation (4) you switch from $j,k$ to $1,2$.\n1. Isn't equation (4) just the same as the second equation in section 4.1?\n1. The last paragraph in page 6 could benefit from a figure, as there's a fair bit of notation, but it should be reasonable to make a figure to clarify.\n1. In **Data Collection** in seciton 5, when you say \"First, we collect self-play experiences...\", these are changing throughout training, right? It's a little confusing as it's worded here as if they're fixed and simply being chosen from a pre-existing set. See also my comment on algorithm below.\n1. In **Action and Centralised Critic Architecture** in section 5, when you say \"the trained actors\", these are already trained, or you're training them? The wording is a little confusing.\n1. In equation (9) it's not clear how you arrived at this loss function from the preceding discussion.\n1. Where are the $C^{pred}$s from equation 10 used? It seems they should be used in equation (9) (from the wording), but they're not there.\n1. In **Level-based Foraging** of section 6.1, when you say \"the total levels of agents...\", are the levels added when both agents are in the same cell?\n1. In the first paragraph of page 10, just to confirm: agents can both be in the same position?\n1. In the third paragraph of section 6.2, just to confirm: Policies in $\\Pi^{train}$ are fixed at this point, yes?\n1. In the fourth paragraph of section 6.2, make it clear that you also evaluate with policies generated by other _methods_, not just different seeds. This becomes clearer later, but you could clarify nmore at this point as well.\n1. In the last paragraph of section 6.2 when you say \"with each unknown teammate policy type from $\\Pi^{train}$\" I think you mean $\\Pi^{eval}$?\n1. In the second-to-last paragraph of section 6.3, when you say \"a classifier that identifies the population... based on an observed state and its action\", given that state and action spaces used are very small, is a state-action pair really enough to classify teammates? Given that this classifier is central to AnyPlay, it seems like it places it at a pretty stron disadvantage.\n1. In **Ablations of BRDiv** of section 6.3, when you say \"Furthermore, when computing MAA2C's actor loss...\", this is still for Independent baseline?\n1. The grids in Figures 4 and 5 are kind of hard to interpret. You should consider using something like interquantile mean (see [RLiable](https://github.com/google-research/rliable) and [paper](https://proceedings.neurips.cc/paper/2021/hash/f514cec81cb148559cf475e7426eed5e-Abstract.html)) to provide a more digestible aggregate summary. It seems scores are normalized, so it should be relatively easy to add with rliable.\n1. In addition to the last point, a useful plot to generate would be a scatter plot of BRDiv score against the final agent performance. This can allow us to determine if there _is_ a correlation between them, and it would further justify your choice of this metric.\n1. Is Figure 7 (b) supposed to be better than 6(b)? It's not clear that this is the case.\n1. In the bottom of page 15, when you say \"Learning from such highly optimal policies makes...\", why was this not an issue with other environments?\n1. In the last sentence of section 6.5 (right before section 7) you say \"a specific return threshold at the given task\". What would be a good threshold here?\n1. In Algorithm 1, where the \"best response policies\" chosen?\n\n\n\n\n# Nits\n1. In the first sentence of section 3, remove the last 's' from \"formalises\".\n1. In the caption of Figure 2 you say \"a shared critic network (green box)\", but I think this should be blue box.\n1. In the last line of page 8 you wrote \"chapter\", but you mean \"section\".", "broader_impact_concerns": "No concerns", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a method for learning policies for ad-hoc team formation,  i.e., situations with new teammates without pre-existing coordination strategies. Previous work has achieved that by placing, at training time, agents into situations where they are asked to interact with as much diverse teammates as possible. As authors state, for that to happen successfully it often requires domain knowledge but most importantly current ways to diversify teammates results in “superficial” diversity where trajectories might be different but policies themselves still correlated. Instead, to promote diversity authors propose to use the Best-Response Diversity (BRDiv) metric which directly operates on the returns of the agents. The authors compare their method for diversifying against previous trajectory diversification methods in 3 environments. Results show that the proposed method yields better returns while also resulting in more diverse behaviours.

All reviewers agreed that this is a strong submission with dense yet clear results or a simple method that can be applied across different setups and environments. Reviewers have some issues wrt to clarify and additional discussions added, and authors have incorporated these into their manuscript. The main outstanding points referenced across reviews was, perhaps, the limited and artificial set of environments used here. Reading the reviews, author response and paper, I agree somewhat with the reviewers -- however, there can always be more complex and more environments and LBF is also an environment used for this type of work. Ultimately, all reviewers agreed that this submission provides valuable insights. I agree with them and so I’m going to recommend acceptance. 
Decision: Accept (poster)