Diversity Is Not All You Need: Training A Robust Cooperative Agent Needs Specialist Partners
Authors: Rujikorn Charakorn
Poramate Manoonpong
Nat Dilokthanakul
Venue: NeurIPS 2024 poster (2024)
Keywords: multi-agent
reinforcement learning
robustness
diversity
specialization
Primary area: reinforcement_learning
TL;DR: We show that partners' specialization is crucial for training a robust generalist and propose a method that reduces overfitness of XP-min partners that already have good specialization

Abstract: Partner diversity is known to be crucial for training a robust generalist cooperative agent. In this paper, we show that partner specialization, in addition to diversity, is crucial for the robustness of a downstream generalist agent. We propose a principled method for quantifying both the diversity and specialization of a partner population based on the concept of mutual information. Then, we observe that the recently proposed cross-play minimization (XP-min) technique produces diverse and specialized partners. However, the generated partners are overfit, reducing their usefulness as training partners. To address this, we propose simple methods, based on reinforcement learning and supervised learning, for extracting the diverse and specialized behaviors of XP-min generated partners but not their overfitness. We demonstrate empirically that the proposed method effectively removes overfitness, and extracted populations produce more robust generalist agents compared to the source XP-min populations.

Diversity Is Not All You Need: Training A Robust
Cooperative Agent Needs Specialist Partners
Rujikorn Charakorn
Poramate Manoonpong
Nat Dilokthanakul KMITL
VISTEC rujikorm.c.s19vvee wtt...s..vevstte....ttt.
VISTEC, SDU
poramate..mgistec..aa m@w.stec..c.tti
nat dw@.mitll.acc.mmmm
Abstract
Partner diversity is known be crucial for training robust generalist cooperative agent. In this paper, we show that partner specialization, in addition to diversity, is crucial for the robustness of downstream generalist agent. We propose a principled method for quantifying both the diversity and specialization of a partner population based on the concept of mutual information. Then, we observe that the recently proposed cross-play minimization (XP-min) technique produces diverse and specialized partners. However, the generated partners are overfit, reducing their usefulness as training partners. To address this, we propose simple methods, based on reinforcement learning and supervised learning, for extracting the diverse and specialized behaviors of XP-min generated partners but not their overfitness. We demonstrate empirically that the proposed method effectively removes overfitness, and extracted populations produce more robust generalist agents compared to the source XP-min populations. This result highlights the importance of considering both the diversity and specialization of training partners while carefully managing their overfitness for training robust cooperative generalists.
1 Introduction
Multi-agent reinforcement learning (MARL) algorithms can generate a team of agents for solving complex cooperative tasks [1, 2]. However, these agents tend to overfit to teammates seen during training and cannot cooperate effectively with unseen team members [3, 4]. This problem is also known as the ad-hoc teamwork problem [5]. A crucial aspect of building robust cooperative (generalist) agent is the diversity of training partners [6-10]. Recent tfate-of-the-ee partner generation methods utilize cross-play minimization technique (XP-min) [11-16] to produce diverse partners.
XP-min approaches generate partners that are behaviorally diverse by learning incompatible policies via some measure of incompatibility, e.g., incompatible with other agents in the same population [12, 14] or best response agents [15, 16]. Fundamentally, XP-min aims to maximize self-play (SP) return and minimize return of cross-play (XP) trajectories, in which policies from different teams interact. However, due to the nature of - objective, generated partners are incentivized to learn to identify the current partner and use the partner's 's identity to decide to cooperate (maximize return) or sabotage (minimize return). This behavior also known as handshaking [17, 18). Importantly, handshakes-ccoventiiig conventions used for bandshaking could be arbitrary and nonsensical, e.g., agent will cooperate only the partner moves in the north direction in the first timestep. Handshaking behaviors undesirable if these partner agents to be used for training generalist agent, simply because they overfit to random handshakes established during training. Thus, handshaking is form of overfitness, and we the words "overfitness" and "handshaking" interchangeably.
The root cause of handshaking induced by XP-min is that the partners can learn to sabotage the game if they identify that the current interaction is an XP trajectory. Recently, the problem of handshaking behaviors has been tackled by [17, 18). The core idea of these approaches is to regularize the
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

diversified partners such that they still have high expected self-play (SP) return under XP trajectories, effectively mixing the SP and XP experiences. That is, the partners should not sabotage the game as they have to maximize the return even under XP trajectories. Therefore, the partners are no longer incentivized to learn handshakes, and these partners should behave in good faith regardless of which partner they interact with. We refer to this class of approaches as mixed-play regularization (MP-reg).
Sarkar al. [18] find that it possible for regree overcorrect the XP-min objective, resulting in a less diverse partner population. We hypothesize that, even with the right amount of experience mixing in reg, the produced partners will have less specialization the quality of an agent capable of solving the problem only in specific ways. We refer to this problem as loss of specialization (LOS).
We hypothesize that generalist agent who trains with diverse but unspecialized partners will not be exposed to diverse experiences. That is because, during training, the generalist agent can pick and choose specific solutions that give higher expected returns, ignoring other potential solutions in the environment. Furthermore, the generalist agent is not required to understand the partner's 's intention when the partners are willing to cooperate regardless of the solution the generalist agent attempts. Thus, we posit that the desiderata of training partners are not only diversity but also specialization.
Our main insight is that XP-min partners not only diverse but also have rich specialized behaviors that are useful for training a generalist agent. So, selectively using diversity and specialization of XP XP-min partners while reducing handshaking could prove valuable for training robust cooperative agent. While MP-reg could increase the diversity and reduce the overfitness of generated partners, it comes with the hypothesized LOS problem. Hence, we are interested in alternative method that satisfies both desiderata of good training partners. We propose a simple yet effective method to transfer the knowledge to another set of newly initialized partners via reinforcement learning (RL). Instead of reducing handshaking from xP-min partners via regularization, we could specifically extract diverse and specialized knowledge from them after the training process. Importantly, we aim to maintain their diversity and specialization but not the sabotaging behaviors. We call this method specialization transfer via reinforcement learning (SpecTRL). Furthermore, we present SpecTRL DAgger-a a combination of SpecTRL and DAgger [19] to stabilize the distillation process and better maintain the diversity of the source population. Our key contributions are as follows:
A set of measures that quantify the quality of training population whole: diversity, specialization, overfitness (Section 3).
Investigation of the interaction between the proposed measures and the robustness of downstream generalist agents. We find that overfitting and lower specialization have a detrimental impact the robustness of downstream generalist agents (Section 4).
A novel method SpecTRL and SpecTRL DAgger that aims to transfer diversity and special- ization while eliminating overfitness of xP-min partners (Section 5). Finally, show that SpecTRL DAgger effectively reduces the number of incapable distilled partners (Section 6).
2 Preliminaries
We focus on cooperative environments described as decentralized partially observable Markov decision processes (Dec-POMDP, Bernstein et al. [20]). An N-player Dec-POMDP is defined by tuple (S, [4'), (2'), T,,,O, 7, H,, where S is the global state space. A = A' and Ω x' are the joint-action and joint-observation spaces of all players. The probability of the next state Players' local
conditioned on a state and a joint action is given by the transition function observations are partial views of the current state given by the observation function O. The reward function R outputs a global reward, 7 the discount factor, and H is the horizon length.
At timestep t, each player observes of. a partial view of the global state st, and outputs an action aj ~ (-(r)) where 7/- {of, ay. To..., }f is the local history of the trajectory of player i. Collectively, all players produce a joint action 0₂ ~ π II; T², where π is the joint policy. The global reward
is given as 7'r,. 7'r. The expected return of a joint policy is (TT) = =))((=) [G(T)), where p(TT) is the distribution of joint trajectories under a joint policy т. We use bold letters to represent joint quantities. We use subscripts and superscripts to indicate different joint policies and players, respectively. We
R(s;,a,) The return of joint trajectory T E T = (9 x A x R)" is G(T) =
use π to represent all other agents except i, i.c., A { " " We define the expected
2

---

return of self-play (SP) trajectories as JsP(TTA) = [ T T and the expected return of cross- play (XP) between two joint policies as (Xp(TTA TB) == EEE [ FF') + J( ग xx'll and the expected return of an ad-hoc team between generalist policy To and a joint policy as (AH T) = 인 Ei= i=!
J(TG π ". Note that #G is not joint policy and does not have any
specific role assigned to it. So, must cooperate with other agents by filling in the missing role.
We use the word "partner" for policies that are used for training or testing a generalist agent. Our main interest training a robust generalist agent &G that can cooperate with unseen partners. Formally, given a population of training partners P, the training objective of TG is
max JAIT Ta; P) == Ex,EP (JAIT(TG: TP)
#G
In this work, we prioritize task completion over literal task performance. Thus, describe the robustness of aG as
R(TG, Pue)) == HM((SR(TG. ,T') FT E Pieu)))
SR(T) == By~o(m) [S(T)),
where HM(-) gives the harmonic mean of set of scalars, SR(T) is the success rate of π and S, identifies whether a joint trajectory is successful.
Cross-Play Minimization (XP-min): XP-min technique has been recently proposed to generate diverse training partners [1 11116.. Here, we describe the variant used by Charakorn et [12], by which the partners learn to be incompatible with others in the same population:
High return
Low XP return
max nin AT P) = JSP( A FA
AXP TT) TTAEP
(4)
argmax AXP(TA.T.) TTEE(((TM)) ((TA))
(5)
where T. is the policy that is the most compatible with πA In short, the XP-min objective optimizes a set of joint policies that are capable while being incompatible with other policies.
Mlxyd-Play Regularization rMP-reg): MP-reg [17, 18] aims to solve a fundamental problem of XP-min, where Eq. 4 incentivizes the partners to overfit or learn handshakes. Here, we briefly explain the MP-reg objective and how helps reduce overfitness following the description of CoMeDi [18].
In addition to SP and XP episodes, re- introduces mixed-play (MP) episodes, which have two stages: state mixing and self-play. In state mixing, the first i' timesteps of episode will evolve according cross-play policy, e.g., лXP (I' A: I лxp (#² " Then, typical SP rollout happens right after the state mixing. The training objective of MP-reg is to maximize the SP return starting from sy produced by Txp: JMP(TTA, π4). Intuitively, this objective regularizes XP-min agents such that they do not "sabotage" the game when interacting under XP trajectories as they still have maximize SP return after timesteps. The rP- objective of each A
High SP returm
Low XP return
Hlgh MP return
max Jup-reg A, P) = -TS((TA) AXP JMP TT VTA EP,
(6)
Mutual Information (MI) Objective: LIPO [12] uses both XP-min and MI to generate diverse partners. Here, we briefly describe how MI objective can be used in XP-min training [12]. A joint policy л is latent-conditions policy with the form T T) = Ez~p(z) II, <a'''r",, where = (=) is joint latent variable and p(z) ) is the prior distribution of The MI objective is to maximize the mutual information between the doservation-acttinn pair and the latent variable cach policy: I(fo', 2'). Since the objective is intractable, we then optimize the variational lower bound of the objective:
I(fo',a'): 2 H(²²) + [log 9 ('(lo',a')] aa))
(7)
where H(z²) is the entropy of the random variable z1, and go is neural network approximating the true posterior K a²)).
3

---

3 Quantifying Partner Qualities
In this section, we present three measures that quantify the different qualities of a population of partners. The purpose of these measures is to allow us to compare populations and predict which ones are better producing more robust generalists. First, we describe the characteristics of joint trajectory and a joint policy as follows. Given a function f : X, we can compute the characteristic of joint trajectory T as x f(T), where X' is the characteristic space. For instance, a could represent the frequencies of certain events under a joint trajectory т. Consequently, we can think of the distribution of random variable X under æ, P(X 17 T as the characteristic of joint policy π. That is, the probability of observing x, P(X= = II πτ) depends on the joint policy л. This approach allows us utilize domain knowledge through well-crafted function for better interpretability or even Iearn the function when expert knowledge is unavailable. Next, we present the first two partner qualities that affect the robustness of downstream generalists based on the concept of the mutual information between X and I7: diversity and specialization.
We define the diversity of population of partners P how diverse the random variable X X distributed under P. Then, the diversity of X under P can be calculated using the concept of entropy:
D(P) H(X) - P(x)
log P(x) =- - ET [P(x) log (E, (P((
(8)
We can see that D(P) has direct implication to the training distribution (Eq. 1) and, consequently. the robustness of the generalist agent (Eq. 2). For instance, if the training population Purain is characteristically diverse (i.e., D (Puaii) is high), it is more likely that some characteristics in Prest will be covered by the training set of the generalists.
Diversity of P is not the only aspect that affects the robustness of a downstream generalist agent. We expect that another quality that affects the robustness of the generalist agent is the specialization of the training partners. We propose to measure the specialization of P by how single-minded each partner π is. Mathematically, we can compute the specialization of a joint policy as the negative entropy of the characteristic of that joint policy: -11 (X 1/ = т). Then, we can calculate the specialization of a population by taking the average specialization of the joint policies:
S(P) = BB [ 1I T) = - II),
(9)
H(X 11 1= = - log P(x
P(x|T)
(10)
We argue that specialization directly impacts the robustness of the generalist agent based on the following rationale. For a generalist agent to effectively cooperate with specialized partners (S(P ( is high), it has understand the partner's intention and learn various strategies because each partner is a specialist who solves the task in specific ways. In contrast, if the partners are not specialized (S train) is low), the generalist no longer needs understand the intention of the training partner and find the easiest or the most rewarding path to complete the task.
Another crucial quality of training partners is their overfitness. We propose way quantify the overfitness of policies in a population by evaluating them against an oracle generalist Tē. which has been trained with set of oracle specialists Pš. The oracle specialists in P3 are assumed to be collectively diverse, individually specialized, and do not use handshakes. This means that gद is capable of solving the task in different ways thanks to the diversity and specialization of the oracle specialists. However, T6 will not be able to cooperate with a partner π if the partner either uses overfitness evaluator of generated partners.
handshakes or is overfit. Thus, we can use To Mathematically, we define the overfitness of a population as
(11) Capable partners with handshaking and overfitness (O(P) is high) are not desirable because they cannot cooperate even with the oracle generalist, which implies that they can only cooperate only if the entire team behaves under specific state-action distribution. As a result, a downstream generalist could be underfit because it might not discover the specific handshakes used by the partners.
O(P) 1 R(T);P)
4 Understanding Effects of Specialization and Overfitness
In this section, we aim to explore the relationship between the specialization and overfitness of population and the robustness of downstream generalist agents. Thus, we perform a control

---

experiment using oracle specialists as the starting population. Then, we derive two additional populations with lower specialization and higher overfitness, respectively, while controlling the diversity of the populations to be similar to the starting population.
We use the multi-recipe Overcooked [12, 21, 22] (MIT License, Fig. 1) as the experimental platform. In short, multi-recipe Overcooked is tWO-
Playens
Chopped Letture (83)
Chopped Onion (R2)
Tonata
Chopped Tomato (83)
Chopped Carret (84)
player cooperative game where agents play as chefs who aim to deliver one
Choppting
Beltwery
Garrot
of six possible recipes as fast as pos-
Deion
Temato Lattuce Saled (83)
Tematt Carret said (RG)
sible. An episode terminates when
dish is delivered or the time hori- zon is reached. We use handcrafted (partial) function f that extracts the completed recipe of joint trajectory. Specifically, z; = f(T) is a one-hot vector representing the completed recipe in and X' = {z, J E R is the completed recipe space. Thus, II = π represents the recipe completion distribution of æ and P(X 2; πτ) is the probability of recipe r; being completed by the joint policy æ. We assume that these partners capable, i.c., they have high SP returns. This means that we only consider partners that can complete the task consistently. More details can be found in Appendices A and B.
Counter Plate
(a)
(b)
Figure I: Overview of the multi-recipe Overcooked game.
To obtain the oracle specialists, we use reward-shaped self- self-play training in which each specialist is trained to complete specific recipe. We train four policies for each of the six specialist types. use three set specialists as starting population ( P's = 18) and the other for robustness evaluation (| P = 6). To generate population with increased overfitness (P oventt)) we train XP-min agents with a modified objective. Instead maximizing SP return, each agent learns to maximize return with specialist. For population with decreased specialization (P unsspee) train 18 generalist policies against P (see Appendix C for training details). Finally, train 8 XP-min partners (Pxp.nin) using only the -i-min objective and put it Table and Fig. 2 for reference.
Table 1: Diversity (D), specialization (S), and overfitness (O) of partner populations. The right- most column shows the harmonic mean of success of downstream generalist agents against Prea- We do not use FCP [8] in this experiment.
Relative Frequency
0.6
0.4
0.2
Populations
D(P)
S(P)
O(P)
R( P Punt)
1.79
-0.01
0.06
0.81 + 0.05
0.0
p
unspes
1.72
-1.64
0.01
0.49 +
Relative Frequency
overtt
1.79
00.01
0.35
0.73 1 0.01
0.6
(+) Pxp-rán
1.31
-0.12
0.51
0.49 + 0.02
0.4
0.00
0.2
-0.25
Decrease specialllinneee specialiration
Specialization
-0.50
0.0
Relative Frequency
-0.75
Starting population (P2)
0.4
Decreased specialization (P Gnsped)
Increased overfitness (P overt))
0.2
-1.75
min
0.0
0.0
0.1
0.2
0.3 Overfitness
0.4
0.5
0.5
R1
R2
R6
Figure 3: The recipe distribution at the last train- ing iteration the generalist agents trained with P (top), Po (middle), and P unspec (bottom).
Figure 2: Relationship of training populations in the 2D specialization -vveriiiss landscape.
5

---

Table I shows that the three control populations have comparable diversity, while P Paverti and Pr have increased overfitness and reduced specialization, respectively. For each population, The right- most column shows the performance of generalist agents trained with each population when evaluated against Plest- As expected, Poverit and p
unspec
unspoc produce much less robust generalists compare to P's.
We can see the root cause of the lower robust- ness of generalists trained with Pinspee in the bottom subplot of Fig. 3. We can see that the recipe distribution during the training general-
15
Average Return
ist agents is less diverse, condensing at recipe R6
(the error bars represent the standard deviations over three random seeds). Furthermore, despite learning to mostly use R5 and R6, the general- ists cannot cooperate well even with specialists that prefer R5 and R6 (see Appendix D for de- tails). In Fig, 4, overfit to the optimal ways for completing the most rewarding recipes (R5 and R6), achieving even higher return than self-play agents thanks to generality and flexibility of training partners in Pinspe (i.e., low ((Paaspe)) )). As result, they are not required to have partner understanding capability, which is crucial for generalization. While o generate similar training recipe completion to PP's, the generalist agents are not as robust the trained with P'-- The orange training curve in Fig. suggests that the generalists trained with averrit is underfit. Thus, has lower performance when matched with test partners. From these results, conclude that unspecialized or overfit partners are not good training partners.
SelF-play
trained with P
trained with P
trained with P coopoc
see that the generalists
100
200
300
400
500
Training Iteration
Figure 4: The average training returns of general- ists trained with different oracle populations.
Specialization Transfer
XP-min Training
XP-min Partners
Specialization Transfer
Distilled Partners
Generalist Training
Generalist
Figure 5: An overview diagram showing the steps in the training pipeline.
In Table 1, we see that Pxp-nin has moderate diversity and high specialization but also high overfit- ness. This observation suggests that XP-min population can potentially be desirable training population if the overfitness is reduced. Therefore, we seek methods to reduce the overfitness while maintaining the diversity and specialization of an XP-min population. We propose SpecTRL and SpecTRL DAgger to further reduce the overfitness after the training process of Xp-min agents.
SpecTRL distills the knowledge from a source population P into distilled population P' with the same number of agents using reinforcement learning. Specifically, each agent a E P' distills the knowledge of reference agent P with the reward maximization objective:
JspecTaL (TT'==
Σ Ha'', x i=1
(12)
Intuitively, distilling via the reward maximization objective (Eq. 12) incentivizes the distilled partners to "nudge" the source partners to perform cooperative behaviors (which gives high return) and away from their sabotaging behaviors (which gives low return). Additionally, when the source partners cooperate, they do so in specialized ways as they have already learned specialized behaviors with XP-min. This means that the distilled partners must learn the preferences and specialization of the XP-min partners but not their sabotaging behaviors.
SpecTRL can be further combined with DAgger [19], using XP-min partners experts, resulting in a knowledge transfer method that utilizes both RL and supervised learning. This combination especially beneficial when XP-min partners heavily utilize complex handshakes that are unlikely to

---

be discovered by random exploration. We refer to this approach as SpecTRL DAgger:
L= N
SspeCTRL DAgger (T =
Σ
(
x + CongerrCA
(13)
1=1
Laggger (T = -R
20 ; log ( (ailT)),
(14)
where at is the expert action, given by the source policy T) and ADAgger 2 0. The auxiliary supervised objective Eq. 14 is useful for stabilizing the distillation process by directly transferring the knowledge from the source policy, unlike SpecTRL that fully relies on random exploration of RL training. A primary assumption of DAgger is that it requires access to experts' policies.
6 Experiments
Table 2: Qualities oracle partner populations and their respective downstream generalists' robust- Green arrows (ft, 4.), red arrows (U), and approximation symbol (=2) indicate the improvement, degradation, and no significant changes over the source population (written between brackets), re- spectively. We only the arrows when the standard deviations do not overlap. + represents the standard deviation over three random seeds.
Populations
D(P) t
S(P) t
O(P) 4
R Plea) t
*S
1.79
-0.01
0.06
0.82 0.05
*Overfit
1.78
0.00
0.35
0.74 + 0.02
*Unspec
1.72
-1.64
0.01
0.49 0.08
1.79
-0.01
0.16
0.78 + 0.01 (f 0.04)
|*Overfit] SpecTRL
(2)
(-)
(4 0.19)
Table 3: Qualities of learned populations and their respective downstream generalists' robustness.
Populations
D(P)
S(P) f
O(P)
R(TG. Pws) 1
SP XP-min
1.08 0.05
0.49 1 0.12
0.08 0.02
0.18 1 0.08
1.31 0.08
0.13 0.05
0.51 0.05
0.56 1 0.07
1,48 0.03
-0.38 0.16
0.47 0.04
0.44 0.04 (4 0.12)
[XP-min] MP-reg
(ft 0.17)
(4 0.25)
(==
1.63 0.03
0.63 0.12
0.54 0.06
0.61 + 0.02
[XP-min] + MI
(ft 0.32)
(4 0.50)
(R)
(~)
1.28 0.05
0.68 + 0.21
0.31 + 0.05
0.55 + 0.04
[XP-min] + MI + MP-reg
(~)
(U 0.55)
0.17)
(~)
1.21 = 0.10
00.11 + 0.08
0.25 0.02
0.58 + 0.08
[XP-min] + SpecTRL
(~)
(=)
(U 0.26)
(~)
1.32 0.09
-0.14 0.04 (-)
0.29 0.06
0.62 0.01
[XP-min] + SpecTRL DAgger
(~)
(U 0.22)
(~)
1.44 0.04
0.45 1 0.06
0.20 +: 0.06
0.62 1.01 (~)
[XP-min + MI] + SpecTRL
(4 0.19)
(1t 0.18)
(4 0.34)
1.60 0.02
0.08 0.56
0.30 0.03
0.64 0.01
[XP-min + MI] SpecTRL DAgger
(=)
(=)
(U 0.24)
(1 0.03)
[XP-min + MI + SpecTRL DAgger] +
1.56 0.02
-055 + 0.09
0.27 0.02 (z)
0.64
SpecTRL DAgger
(2)
(~)
(7)
We aim to empirically investigate how different training objectives (XP-min, MI, MP-reg, SpecTRL, and SRL DAgger) affect the qualities of the generated partners. We use our implementation of recently proposed CoMeDi [18] to represent the MP-reg approach. We also include self-play (SP) as one of our baselines (see Appendix E for more details). Note that partner qualities presented in Section 3 do not depend on the size of populations of interest. In theory, we can compare populations with different sizes. For fair comparison, we compare populations the same size ((P) = 8).
is possible that some generated partners are incapable due to training instability unsuccessful
7

---

0.00
O
Overfit
*5
XP-min
-0.25
1.6
-0.50
Self-play
Specialization
XP-min MI
1.5
-0.75
-1.00
-1.25
-1.50
*Unspec
SpecTRL
SpecTRL DAgger
-1.75
MP-reg
1.2
0.0
0.1
0.2
0.3
0.4
0.5
0.6
Overfitness
Figure 6: The partner quality landscape, representing partner qualities from Table visually. The color bar represents population diversity. The arrows show how different approaches improve partners' qualities. XP-min MI + SpecTRL DAgger population is marked with
distillation. If that is the case, the proposed diversity and specialization measures are undefined for incapable partners, So, we remove those partners from diversity and specialization calculation while still keeping them in their population for training downstream generalist agents. Therefore, for fair comparison, we always incorporate Fictitious Co-Play (FCP) [8] when training generalist. FCP adds weaker partners to a training population, including random and incapable ones, by using past checkpoints of the partners. For populations related to oracle specialists (marked with +), they have population size of 18, and we do not use FCP as they only serve as references.
Table 2 shows the effectiveness of SpecTRL at reducing the overfitness of the overfit oracle specialists. Table 3 shows the qualities of learned populations and the robustness corresponding downstream generalists. We also visualize the populations in the partner quality landscape in Fig. 6, allowing us to visually compare populations generated by SpecTRL, SpecTRL DAgger, and MP-reg.
SpecTRL and SpecTRL DAgger Consistently Reduce Overfitness: SpecTRL consistently re- duces overfitness of the input population as in- tended. It successfully reduces the overfitness of the overfit oracle specialists, thus improving the robustness of generalists (see Table 2). However, SpecTRL significantly reduces diversity when used with XP-min + generated populations (see Table 3). It is likely because of unsuccess- ful distillation, resulting in incapable distilled partners. SpecTRL DAgger fixes this problem effectively (see Table 4). Thus, it maintains the diversity of the source population while reducing overfitness. Although the reduction in overfit-
Table 4: The number of capable distilled partners when using SpecTR, TRL and SpecTRL DAgger. Other populations do not contain incapable partners.
Populations
Capable Partners
XP-min
7.33 - 0.47
XP-min + SpecTRL
6.00 = 0.82
XP-min + SpecTRL DAgger
7.33 0.47
XP-min MI + SpecTRL
8.00 0.00
XP-min + MI + SpecTRL
5.33 1.25
XP-min MI + SpecTRL DAgger
8.00 0.00
ness is smaller than SpecTRL, the robustness of downstream generalists higher, thanks to the
preserved diversity. Interestingly, SpecTRL seems to increase the specialization of the partners while

---

SpecTRL DAgger does not. We believe that the source partners might have multi-moda behaviors, but the distilled SpecTRL partners might not discover all the behavioral modes via random exploration. So, the SpecTRL partners become more single-minded. On the other hand, SpecTRL DAgger does not significantly increase specialization as the DAgger objective helps transfer multi-modal behaviors directly from the source partners through the imitation learning objective.
We can see that up-min populations are much less diverse than the ones that incorporate either MP-reg or MI regularization during the training process. This result agrees with Charakom et al. [12] and Sarkar et [18] that find MI and MP- reg to help increase the diversity of XP-min partners. Interestingly, we find that using both MP- reg and MI simultaneously do not yield a more diverse population. However, both regularization techniques significantly reduce the specialization of the generated partners. Therefore, despite the increased diversity, the robustness of downstream generalists does not increase as the specialization is also significantly reduced. This result confirms our hypothesis that adding a regularization during XP-min training also comes with the loss of specialization (LOSS) problem.
MP-reg and Increase Diversity but Lose Specialization:
We believe that the LOS problem of both regularizations has different root causes. MP-reg incen- tivizes the partners to complete the task despite not being aligned with their preferences, effectively regularizing agents from having strong preferences and reducing how single-minded they are. On the other hand, MI induces the generated partners have multi-modal behaviors, which could correspond to different high-level behaviors, e.g., completing different recipes.
So far, can that SpecTRL DAgger is effective at reducing the overfitness of XP-min partners. This raises the question of whether repeated distillation could further reduce the overfitness. We investigate and find that repeating distillation does not further reduce overfitness (see the last row of Table 3). We hypothesize that the sabotaging behavior has already been significantly reduced in the first round of distillation and that SpecTRL DAgger is effective at removing sabotaging behavior but not other kinds of overfitness, e.g., state distribution overfitness, which remains in all populations as shown in Table 3. Therefore, repeating the distillation no longer reduces the overfitness.
Repeated Distillation Does Not Reduce Overfitness Further:
7 Discussion
Although MP-reg and MI regularization methods have the LOS problem, they are still necessary for increasing the diversity of the XP-min agents. Theoretical understanding of how regularization during XP-min training changes their specialization would lead to better regularization techniques that improve partner qualities, which will be crucial for building robust cooperative agent.
The calculation for diversity (Eq. 8) and specialization (Eq. 9) depends on the probability of each π being drawn from its population. Thus, we can alter the diversity and specialization of a population by changing how the joint policies are drawn from the population. We believe it is possible to positively modify the diversity and specialization of a population by changing how π are drawn. We leave this investigation for future work.
The overfitness of a population (PP) presented in this paper does not separate handshakes and state-action distribution overfitness. Future work could explicitly decouple types of overfitness for further interpretability of the measure. We do not use any of the measures presented in this paper to diversify; rather, we use them as means understand partner qualities and their relationship to the robustness of generalist agents. Using the presented measures as diversification objectives, potentially with domain knowledge, is worthwhile future direction for much more efficient learning algorithms. We will explore this direction in future work.
Finally, training the generalist agent with auxiliary objectives, e.g., opponent modeling, might improve robustness. However, it is unclear how such training objectives affect the relationship between the proposed measures and the downstream robustness. Understanding how these measures interact with auxiliary objectives will be critical for building robust cooperative agents.
8 Limitations
We note that none of the presented measures alone are representative indicators the quality of the partners. For example, two populations could be equally diverse yet yield vastly different levels of

---

robustness of the generalist agent due to their difference in specialization overfitness. There could be other qualities that impact the robustness of generalists. We will investigate this in future work.
In the experiments, we use domain knowledge for the function f (i.e., how to extract the characteristic of trajectory) and for reward shaping of the training of the oracle specialists. Both of which affect the partner quality measures. We acknowledge this as the main limitation of the experiments. Nonetheless, evaluating models without expert knowledge is challenging and is not unique to cooperative multi- agent systems. For example, evaluating LLMs requires domain knowledge to generate test scenarios or human preferences. We do not aim automate such notorious challenge. Instead, our proposed measures give us the option to use domain knowledge to evaluate the qualities of cooperative agents while leaving an option for learning the function future work.
If f is not well-designed, it is possible that Σx P(x π < 1, which is not a valid probability distribu- tion. Consequently, the diversity and specialization of the population containing is undefined. For example, an untrained π might have an invalid P πT πτ) if X' is the space of reachable goals in an environment because the policy is incapable of achieving any goal.
Our experiments are performed under only a single cooperative domain, multi-recipe Overcooked. We acknowledge this limitation and aim to investigate different domains in future work. The performance of SpecTRL and SpecTRL DAgger depends on the quality of the source partner populations. Hence, the methods should not be expected to improve arbitrary partner populations.
9 Related Work
Generating Diverse Partners for Training Robust Cooperative Agents: In recent years, much efforts in the ad-hoc teamwork literature have been put into generating diverse training partners. Using
one can generate diverse partners via hand-crafted policies [23-25], domain-specific reward shaping [26-28], or Quality-Diversion (QD) algorithms [29]. On the other hand, there are several techniques that can be used to generate diverse training partners without using domain knowledge including using past checkpoints [30, 8), population baased training [3, 31], a mutual information objective [32], trajectory-basee diversification [33], or XP-min p-min methods [11- 13, 15, 17, 18]. This paper conveys an important message that there are other qualities of training partners that should be considered for training robust cooperate agents: Diversity is not all we need.
There are studies that explicitly aim to understand variables that affect the robustness of cooperative agents [10, 13]. McKee et al. [10] and Wang et al. [13] conclude that the number of training partners and diversity are critical factors for the robustness of downstream agents. Our work studies a different set of partner qualities and shows that specialization and overfitness are also crucial for training robust cooperative generalist agents. Notably, under the competitive multi-agent setting, Vinyals et al. [34] show that learning against specialized training opponents eases the learning process and increases robustness. Our work formulates the notion of specialization mathematically and identifies that specialization is one of the key qualities of training partners under the cooperative setting.
Partner Qualities That Affect Robustness:
Reducing XP-min Partners' Overfitness: MP-reg methods [17, 18] aim to reduce overfitness of XP-min partners during their training process. The main idea is to mix SP and XP experiences such that xP-min partners do not learn handshaking behaviors. Unlike MP-reg, the proposed method reduces the overfitness of xP-min partners after the training by knowledge distillation.
Using knowledge distillation for reducing overfitness is also well known in the broad machine learning literature [35-37]. Typically, the goal of knowledge distillation is to transfer knowledge of a teacher model to another student model to reduce the overfitness of the model's predictions. We use the same idea of knowledge distillation in the context of extracting diversity and specialization of xP-min training ("teachers")) partners to another set of ("students") partners while reducing their overfitness.
Reducing Overfitness of Neural Networks:
10

---

Acknowledgement
This research was supported by the Vidyasirimedhi Institute of Science and Technology (VISTEC) and Siam Commercial Bank (SCB). Additionally, it was partially funded by the Reinventing University-Al Beyond Modeling project, supported by the Ministry Higher Education, Science, Research, and Innovation of Thailand. We thank anonymous reviewers for their thoughtful feedback.
References
[1] Jakub Grudzien Kuba, Ruiqing Chen, Muning Wen, Ying Wen, Fanglei Sun, Jun Wang, and Yaodong Yang. Trust region policy optimisation in multi-agent reinforeement learning. In International Conference on Learning Representations, 2022. URL https://openrevieet net/forrumPidEEcGFFITA.
[2] Muning Wen, Jakub Kuba, Runji Lin, Weinan Zhang, Ying Wen, Jun Wang, and Yaodong Yang. Multi-agent reinforcement learning is sequence modeling problem. Advances in Neural Information Processing Systems, 35: 15:10911165,,,, 2022.
[3] Micah Carroll, Rohin Shah, Mark Ho, Tom Griffiths, Sanjit Seshia, Pieter Abbeel, and Anca Dragan. On the utility of learning about humans for human-ai coordination. Advances in neural information processing systems, 32, 2019.
[4] Nolan Bard, Jakob N Foerster, Sarath Chandar, Neil Burch, Mare Lanctot, Francis Song. Emilio Parisotto, Vincent Dumoulin, Subhodeep Moitra, Edward Hughes, et al. The hanabi challenge: A new frontier for ai research. Artificial Intelligence, 280: 103216, 2020.
[5] Peter Stone, Gal A Kaminka, Sarit Kraus, and Jeffrey S Rosenschein. Ad hoc autonomous agent teams: Collaboration without pre-coordination. Twenty-Fout Fourth AAAI Conference on Artificial Intelligence, 2010.
[6] Rujikorn Charakorn, Poramate Manoonpong, and Nat Dilokthanakul. Learning to cooperate with unseen agents through meta-reinforcemmnt learning. Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, pages 1478-1479 2021
[7] Paul Knott, Micah Carroll, Sam Devlin, Kamil Ciosek, Katja Hofmann, Anca Dragan, and Rohin Shah. Evaluating the robustness of collaborative agents. In Proceedings of the 20th International Conference on Autonomous Agents and MultiAgent Systems, AAMAS '21, page 1560-1562. Richland, SC, 2021. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 978 450383033
[8] DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34: 14502-14515. 2021.
[9] Darius Muglich, Luisa M Zintgraf, Christian A Schroeder De Witt, Shimon Whiteson, and Jakob Foerster. Generalized beliefs for cooperative ai. International Conference on Machine Learning, pages 16062-16082. PMLR, 2022.
[10] Kevin R McKee, Joel Z Leibo, Charlie Beattie, and Richard Everett. Quantifying the effects of environment and population diversity in multi-agent reinforcement learning. Autonomous Agents and Multi-Agent Systems, 36(1):1-16, 2022.
[11] Rujikomn Charakom, Poramate Manoonpong, and Nat Dilokthanakul. Generating diverse cooperative agents by learning incompatible policies. In ICML 2022 Workshop Al for Agent- Based Modelling, 2022. URL https::/openreviiee www.reeiwwwwwweeeeeeeeeeeeeeeeee (itormmmii==aaLnnGGGT
[12] Rujikorn Charakorn, Poramate Manoonpong, and Nat Dilokthanakul. Generating diverse cooperative agents by learning incompatible policies. In The Eleventh International Conference on Learning Representations, 2023. URL //ttp:://nn httsss://ppeeeeeeeeeeee wwweeeeeeweeeeeeeeeeeeeeeeeeeeee ttte//ffouummddduuUUo00o07
6.

---

[13] Xihuai Wang. Shao Zhang, Wenhao Zhang, Wentao Dong, Jingxiao Chen, Ying Wen, and Weinan Zhang. Quantifying zero-shot coordination capability with behavior preferring partners. arXiv preprini arXiv:2310.05208, 2023.
[14] Arrasy Rahman, Elliot Fosong, Ignacio Carlucho, and Stefano V Albrecht. Generating team- mates for training robust ad hoc teamwork agents via best-response diversity. Transactions on Machine Learning Research, 2023. ISSN 2835-8856. URL
forum?id=15B22001
[15] Lei Yuan, Lihe Li, Ziqian Zhang, Feng Chen, Tianyi Zhang, Cong Guan, Yang Yu, and Zhi-Hua Zhou. Learning to coordinate with anyone. In Proceedings of the Fifth International Conference on Distributed Artificial Intelligence, pages 1 1-9, 2023.
[16] Muhammad Rahman, Jiaxun Cui, and Peter Stone. Minimum coverage sets for training robust ad hoc teamwork agents. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 17523-17530, 2024.
[17] Brandon Cui, Andrei Lupu, Samuel Sokota, Hengyuan Hu, David J Wu, and Jakob Nicolaus Foerster. Adversarial diversity in hanabi. In The Eleventh International Conference on Learning Representations, 2023. URL https:/ /ttp::///ppeeeeiieeee hwtpreeiieiieeeeeeeeeeeeeeeeeeee wwtp://eee/oooominiicoem wttp:/rrnnnnnnnwnrriorr H.
[18] Bidipta Sarkar, Andy Shih, and Dorsa Sadigh. Diverse conventions for human-Al collaboration.
In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https: wwweeereeeew.ne//forreeeeeeeeeee wowtorrrmmiiiiimooo M.yyMM eRycu
[19] Stéphane Ross, Geoffrey Gordon, and Drew Bagnell. A reduction of imitation learning and structured prediction to no-regret online learning. In Proceedings of the fourteenth interna- tional conference on artificial intelligence and statistics, pages 627-635. JMLR Workshop and Conference Proceedings, 2011.
[20] Daniel Bernstein, Robert Givan, Neil Immerman, and Shlomo Zilberstein. The complexity of decentralized control of markov decision processes. Mathematics of operations research, 27(4): 819- 840, 2002.
[21] Sarah A. Wu, Rose E. Wang, James A. Evans, Joshua B. Tenenbaum, David C. Parkes, and Max Kleiman-Weiner. Too many cooks: Coordinating multi-agent collaboration through inverse planning. Topics in Cognitive Science, n/a(n/a), 2021. doi: https://doi.org/10. 111111 1/ttos.. 12525. URL ptttps://oonineeibraae waabbbrry.wwlley.. wiley. http///oo/a///s/// http///....//oos.... 12t.l
[22] David Rother, Thomas Weisswange, and Jan Peters. Disentangling interaction using maximu- mentropy reinforcement learning in multi-agent systems. In European Conference on Artificial Intelligence, 2023.
[23] Ahana Ghosh, Sebastian Tschiatschek, Hamed Mahdavi, and Adish Singla. Towards deployment of robust cooperative ai agents: An algorithmic framework for learning adaptive policies. In Proceedings of the 19th International Conference on Autonomous Agents and MultiAgent Systems, pages 445, 2020.
[24] Annie Xie, Dylan Losey, Ryan Tolsma, Chelsea Finn, and Dorsa Sadigh. Learning latent representations to influence multi-agent interaction. In Conference Robot Learning, pages 575-588. PMLR, 2021.
[25] Woodrow Zhouyuan Wang, Andy Shih, Annie Xie, and Dorsa Sadigh. Influencing towards stable multi-agent interactions. In Conference on Robot Learning, pages 1132-1143, PMLR. 2022.
[26) Joel 7. Leibo, Edgar A Duemiez-Guzman, Alexander Vezhnevets, John P Agapiou, Peter Sunehag, Raphael Koster, Jayd Matyas, Charlie Beattie, Igor Mordatch, and Thore Graepel. Scalable evaluation of multi-agent reinforcement learning with melting pot. In International Conference on Machine Learning, pages 6187-6199, PMLR, 2021.
[27] Zhenggang Tang, Chao Yu, Boyuan Chen, Huazhe Xu, Xiaolong Wang, Fei Fang, Simon Du, Yu Wang, and Yi Wu. Discovering diverse multi-agent strategic behavior via reward randomization. arXiv preprint arXiv:2103.0455 2021.
12

---

[28] Chao Yu, Jiaxuan Gao. Weilin Liu, Botian Xu, Hao Tang, Jiaqi Yang, Yu Wang. and Yi Wu. Learning zero-shot cooperation with humans, assuming humans are biased. In The Eleventh International Conference Learning Representations, 2023. URL httpss://openreeiee. net/forum?2d-TTwE88B00
[29] Rodrigo Canaan, Xianbo Gao, Julian Togelius, Andy Nealen, and Stefan Menzel. Generating and adapting diverse ad hoc partners in hanabi. IEEE Transactions Games, 15(2):228 -41,, 2022.
[30] Rujikorn Charakorn, Poramate Manoonpong, and Nat Dilokthanakul. Investigating partner diversification methods in cooperative multi-agent deep reinforcement learning. In Neural Information Processing: 27th International Conference, ICONIP 2020, Bangkok, Thailand, November 18-22, 2020, Proceedings, Part V 27, pages 395-402. Springer, 2020.
[31] Rui Zhao, Jinming Song, Yufeng Yuan, Haifeng Hu, Yang Gao, Yi Wu, Zhongqian Sun, and Wei Yang. Maximum entropy population- based training for zero-shot human -ai coordination. Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 6145-6153, 2023.
[32] Keane Lucas and Ross E. Allen. Any-play: An intrinsic augmentation for zero-shot coor- dination. In Proceedings of the 21st International Conference on Autonomous Agents and Multiagent Systems, AAMAS '22, page 853-861, Richland. SC, 2022. International Foundation for Autonomous Agents and Multiagent Systems. ISBN 9781450392136.
[33] Andrei Lupu, Brandon Cui, Hengyuan Hu, and Jakob Foerster. Trajectory diversity for zero-shot coordination. In International Conference Machine Learning, pages 7204-7213. PMLR, 2021.
[34] Oriol Vinyals, Igor Babuschkin, Wojeiech M Czarnecki, Michaël Mathieu, Andrew Dudzik, Jun- young Chung, David Choi, Richard Powell, Timo Ewalds, Petko Georgiev, et Grandmaster level in starcraft ii using multi-agent reinforcement learning. nature, 575(7782):350-333 -0444,04-55455 2019.
[35] Cristian Buciluă, Rich Caruana, and Alexandru Niculescu-Mizil. Model compression. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 535-541, 2006.
[36] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in neural network. arXiv preprint arXiv: 503.00553 2015.
[37] Samuel Stanton, Pavel Izmailov, Polina Kirichenko. Alexander A Alemi, and Andrew G Wilson. Does knowledge distillation really work? Advances Neural Information Processing Systems, 34:6906-69) 6919, 2021.
13

---

Reproducibility Statement
We include additional information to reproduce the experimental results in the Appendices:
Environment details (Appendix A)
Experimental details (Appendix B)
Oracle-related populations (Appendix C)
Pseudocode, implementation details, and hyperparameters (Appendix E)
The source code is available at https://aaaoyoosss.eemeccssseeer
Societal Impact
The paper provides additional insight into how specialization can impact the robustness of cooperative generalist agents. Also, the proposed methods aim reduce the overfitness of training partners to better generalize generalist agents. We believe this paper will improve how future research tackles the challenges of cooperative multi-agent systems. We also hope that the proposed methods will be used to train more robust and capable generalist agents in the near future.
The choice of f and r can be biased, which can discriminate certain partners' behaviors in downstream uses of the partners. Finally, having superhuman Al assistant in cooperative games might incentivize human players to actively look for Al companions instead of learning to play with other human players, which could harm their mental health and relationships in the long term.
A Multi-Recipe Overcooked
We use the multi-recipe version of the simplified Overcooked game from Charakorn et al. [1], which based on the work of Wu et al. [2], Rother al. [3] (MIT License). The game has the following cooking ingredients: lettuce, onion, tomato, and carrot. At the beginning of an episode, they are randomly placed pre-defined positions in the game. Specifically, lettuce and onion are randomly placed on the left or the middle counter. tomato and carrot randomly placed on the right or the middle counter. These ingredients can be used in different recipes, making each ingredient unique. Four recipes (LettuceSalad, TomatoSa lad, ChoppedCarrot, ChoppedOnion) require only a single ingredient, while the other two two ingredients. The ingredients must be chopped at the chopping station before placing the plate. After the required ingredients are put on the plate, they must be delivered to the delivery station.
TomatoCarrotSal lad) require
The players have the same egocentric observation and action spaces. The observation is set of hand-crafted features representing a local view of the environment. Specifically, we use the following features: absolute position and facing direction, relative distance to the objects and the other agent, state of the ingredients, four booleans indicating the agent next to counter in four cardinal positions, held items, the state of held items, and the type and state of the items in front of the agent. These features are concatenated as a 1-D vector length 54, At every timestep, each player has to choose one of the six possible actions: no op, move {up, down, left, right}, and interact.
An episode lasts at most 200 timesteps and terminates immediately after a successful delivery. An episode without delivery is considered unsuccessful. Consequently, f(T) is undefined for unsuccessful trajectories. The agents are incentivized to interact with the objects and deliver as fast as possible with the following reward function:
Tt Pinteract
T Trrogress
F Tzomplete
p.
(15)
where Fimtercct shaped reward given when an agent interacts with an object for the first time in an episode, "poogrss is given when the players progress toward a recipe completion (i.e., chopping required ingredients or putting chopped ingredients on the plate), Tcomplete is given upon successful delivery, and p is a penalty. We use Tintermet 0.5, progress 1.0, Tcomplate 10, and 0.1. We note that recipes with more than one ingredient will give only slightly higher rewards (Fimeract + Tprogress) but are significantly harder to discover by random exploration than those with one ingredient.
14

---

B Additional Experimental Details
Since we use handcrafted characteristic function f that maps a trajectory to a one-hot vector represent- ing a completed recipe, mathematically. f(T) is undefined for some т that is unsuccessful. Hence, f is a partial function. From experience, all capable agents always complete recipe. If necessarily, we remove trajectories that the agents fail to deliver a recipe (undefined f(T)) so that P(X 17 = π) valid probability distribution: Ex P(x|m 1. We remove incapable agents from all quality measurements as they do not represent meaningful behaviors, though we still use them for training generalist agents. We make sure that all populations include some weaker agents by always using FCP [4] when training a generalist. We define capable agent as an agent with more than 50% success rate under SP trajectories. We estimate P 1I π for each π using 200 SP episodes. We estimate the overfitness of each π using 200 ad-hoc episodes against the oracle generalist. We evaluate the robustness each generalist using 1200 ad-hoc episodes and 200 episodes for each test specialist. Although the proposed specialization transfer approaches are not computationally expensive, they require additional computation after training the source populations. For example, a single run of SpecTRL or SpecTRL DAgger takes around 12 hours to distill a population of 8 XP-min agents using a desktop PC with an AMD Ryzen 9 5950X 16-Core processor with 64GB of RAM.
The details provided here are used in both Section 4 and Section 6.
C Oracle-Related Populations in Section
Oracle Specialists (P'):: The oracle specialists are trained using self-play with a modified reward function. For each oracle specialist, the completion reward Teomplete is given only when completing an assigned recipe, which is unique for each specialist. We repeat the training of all six specialists three times, resulting the oracle specialists population with size 18 P( 18).
Overfit Oracle Specialists (P (Pover):: We train XP-min agents with a modified objective. Instead of maximizing SP return, each agent learns maximize return with a specialist while still minimizing return when matched with other xP-min agents in the population. Specifically, we train three LIPO populations instead of one with 18 agents because requires lower computation and is easier train. This is justified since the starting population contains three copies of the six specialists. The training objective for each A in Povertic can be written as
max JXP(TA T.A) Xxp T+) ;(TTA ) E P. 3A
(16)
P= ((TA, TSA) A P overtit
TSA EPss and A {1,..., M}} JXP(TT,, T+)
(17)
argmax T+E(Pnee M(TA))
(18)
Optimizing Eq. 16 creates a population Po in which each partner learns to cooperate with a specific oracle specialist while incentivized to use handshakes to reduce the XP term.
Unspecialize Oracle Specialists (Pi Pumppee): We train 18 instances of generalist agents for this population. The objective for training each generalist is
max E Eng
m'') ; VYg E Pi
(19)
unspec
"C
Note that the generalists are capable of completing all recipes, meaning that they have high diversity. At the same time, they are willing cooperate regardless of what the partner is attempting. This means that they have very low specialization. The quantity of the diversity and specialization is given in Table 1.
D Additional Results for The Control Experiment in Section 4
Fig. shows the full evaluation performance of generalists trained with oracle-related populations generated with domain knowledge and an XP-min population. Interestingly, generalists trained with the unspecialized specialists Pinspec are less robust to unseen test partners who prefer R6 despite the training recipe distribution heavily concentrates at R5 and R6 (Fig. 3). This result suggests that unspecialized partners make downstream generalists prone to overfitting the most optimal trajectories that maximize the return. Thus, the generalist cannot effectively cooperate with the test partners.
15

---

Test specialist partner
Spedillets
15,00)
3.98
0.86
(0 022
10.05)
0.86
10,02)
0.93
10.07)
0.52
0.81
00)
Training popelation
Spedialits
(0.0K)
0.80
01.0%)
(0.04)
(0.09)
10.01)
(0.05]
0.55
07
06
0.5
04
0:59
0.3
Figure 7: Test success rate of generalists trained with oracle-related populations.
E Implementation Details
Our implementation based on the codebase and hyperparameters provided by Charakorn
al. [1], which uses the parameter sharing technique [5-7] and MAPPO [8] for training all partners. We provide the pseudocode for the training of the XP-min MI + MP-reg here.
Algorithm 1: Training process of LIPO (on-policy)
This pscudocode is based on self-play. Blue text is related to the MI objective. XP-min specific code is highlighted green. MP-reg related computation orange
Input: Population
{ S A N
while not done do
// This loop is fully parallelizable
for A E NN do
// Collect self-paay data and compute the objective
Esp +- CColleetPPData(TT.)
Compute ( with Bsp
// Find most compatible policy
π4 <- argmaxx, JXP(TA, T+)
I/ Collect cross-play data and compute the objective
Bxx + CollectXPData(TT, πT)
Compute Jxp( (T TT) with 1xr
I/ Collect mixed-play data and compute objective
BMP CollectMPData (TAAT.)
Compute Jar A, ) ) with BxP
I/ Compute mutual information objective
Compute Lau (the MI lower bound, Eq. 7) using all experiences (Bsp U BxP U BRP)
// Update the policy
0 4 0 Ve, [-ps ( (xx +4)+ +Au A Jaar ( mT) // Update MI posterior approximation
AΛ + Φ Asa Voa TAa
Hyperparameters:
We the following hyperparameters:
Axp = 0.3 (for XP-min objective)
AMI 0.5 (for MI lower bound objective)
AMP 0.1 (for MP- objective)
ApAgger 0.1 (for SpecTRL DAgger objective)
For populations that do not use all the training objectives, set the corresponding coefficients to zero. We use Axp and Avi based on the values provided by Charakorn et al. [1]. We hyperparameter search for AMP {0.1, 0.2) and ADAggar {0.01, 0.1, 0.2), and report the best result.
16

---

F Conclusion
We propose a principled way to measure three qualities of training partners: diversity, special- ization, and overfitness. We investigate how these qualities impact the robustness of downstream generalists and find that, in addition to diversity, specialization and overfitness are essential factors for training robust generalist agent. We also observe that XP-min partners have the potential to be good training partners if their overfit reduced. Thus, we propose two simple methods, SpecTRL and SpecTRL DAgger, to effectively reduce the overfitness while maintaining the diver- sity and specialization of the partners. Empirically, the proposed methods successfully reduce the overfitness of the partners. SpecTRL DAgger improves SpecTRL by stabilizing the distillation process, reducing the number of incapable distilled partners. As a result, the generalists trained with XP-min MI + SpecTRL DAgger populations are the most robust. We also observe that using MP-reg and MI regularizations during XP-min training increases diversity. However, they have the LOS problem and, therefore, cannot directly increase the robustmess of downstream generalists. Although the analysis done in this work relies on domain knowledge, the insight presented here is valuable for building more robust cooperative agent in general.
References
[1] Rujikorn Charakor, Poramate Manoonpong, and Nat Dilokthanakul. Generating diverse coop- erative agents by learning incompatible policies. In The Eleventh International Conference on Learning Representations, 2023. URL
[2] Sarah A. Wu, Rose E. Wang, James A. Evans, Joshua B. Tenenbaum, David C. Parkes, and Max Kleiman-Weing Too many cooks: Coordinating multi-agent collaboration through inverse planning. Topics in Cognitive Science, n/a(n/a). 2021. doi: https://doi.org/10.. 1 12525. URL hatp::///onnneeliii Lnnribrrry......mmmmoomooooaarre wiley. www.ly.cco//eeee homm/doi/ant htt///do//bb////////// http///...111toss.555555
[3] David Rother, Thomas Weisswange, and Jan Peters. Disentangling interaction using maximu- mentropy reinforcement learning multi-agent systems. In European Conference on Artificial Intelligence, 2023.
[4] DJ Strouse, Kevin McKee, Matt Botvinick, Edward Hughes, and Richard Everett. Collaborating with humans without human data. Advances in Neural Information Processing Systems, 34: 14502-14515, 2021.
[5] Ming Tan. Multi-agent reinforcement learning: Independent cooperative agents. In Proceed- ings of the tenth international conference on machine learning, pages 330-337, 1993.
[6] Jakob Foerster, Gregory Farquhar, Triantafyllos Afouras, Nantas Nardelli, and Shimon Whiteson. Counterfactual multi-agent policy gradients. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[7] Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In International Conference on Machine Learning, pages 4295-4300 PMLR, 2018
[8] Chao Yu, Akash Velu, Eugene Vinitsky, Jiaxuan Gao, Yu Wang, Alexandre Bayen, and Yi Wu. The surprising effectiveness of PPO in cooperative multi-agent games. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022. URL https: /opeereview www.nnt/oorumiiiddWaooeettee
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The abstract gives accurate overview and contributions of the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations in the Limitations and Discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We outline training and algorithmic details in the main paper and appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
We provide access to the codebase.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/ Saaiidde/CodeeSuiisssisii
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https: / //nips. ctlipscc//puu cc/pub/lpppc htc//guiee//oaa/ Subiie/// seibbmmmsssooooooooooooooooooooo
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
We provide training and hyperparameter details in the paper and appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide standard deviation for the results.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
20

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We provide the information in the appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer:
[Yes]
Justification:
We follow the Code Of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We have separate Societal Impact section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
21

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: All trained agents are in simulated cooperative environments.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: We properly credit the creators of environments used in this paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, papore thcode. com/ com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
22

---

Answer: [Yes]
Justification:
We provide all assets used in this work along with the code publicly.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
23


Reviews:
[{"review_id": "ZvEYp5jotZ", "reviewer": "Reviewer_sQwt", "summary": "The submission positions itself within the problem of ad-hoc teamwork: learning to cooperate with teammates previously unseen during training. Indeed, an important aspect of ad-hoc teamwork is the rule of \"no prior coordination\". Previous work focused on developing a rich enough set of training partners to allow good test performance. Various diversity metrics were used, such as cross-play minimization. However, these approaches lead to self-sabotaging behaviours in training partners, where they develop secret handshakes (i.e. initial sequences of actions) that identify partner types, and if the other partner fails the handshake protocol, the agent can refuse to cooperate and sabotage. The authors correctly identify that methods that resolve this handshake problem actually lead to a loss of diversity in the training set. Thus a big question asked in the paper is: how to have _meaningful diversity_ in the training set without having the handshake problem.\n\nThey propose to measure the diversity of a population as the entropy of a _function of the induced trajectory distribution under the population._ The choice of function allows one to decide what kind of diversity the designer cares about. They propose to measure the _overfitness_ (a.k.a. handshake problem) by measuring if the members of the population can in fact cooperate with a generalist oracle, an oracle agent that is assumed to not have learned handshakes. Finally the specialization is measured by the negative entropy of a specific policy. They empirically demonstrate that overfitness and/or under-specialization are both bad for training. \n\nTheir proposed solution for training set generation is simple: take a population generated by cross-play minimization, and distill it to a more specialized population by making the population members cooperate with each other more efficiently, which is achieved via reinforcement learning within population. They empirically show that this reduces the overfitness of the population generated by cross-play minimization, and leads to better training partners, without creating the loss of specialization problem compared to previous work.", "questions": "At this point, I do not have any questions.", "limitations": "Limitations discussion was left entirely to the appendix, which I do not like. However, in principle, they are discussed.", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 4, "strengths": "**Originality**\n- The proposed method for generating training partners for ad-hoc teamwork is novel. \n- The loss of specialization, diversity, and overfitness is partially novel. Especially, perhaps the diversity metric is not novel per se as similar metrics were proposed for behavioural diversity, but the authors don't claim novelty here either. The loss of specialization definition feels like folk knowledge, yet I also have not seen this defined in this way before. \n- Overall: the paper is sufficiently novel, and the proposed algorithm is original.\n\n**Quality**\n- The experimental evaluation of the proposed method is relatively extensive, the results are presented nicely, and the overall quality of the work appears high. \n\n**Clarity**\n- The presentation is clear, although a bit dense. See weaknesses.\n\n**Significance**\n- The paper has notable significance for cooperative AI and ad-hoc teamwork.", "weaknesses": "- Personally, I am not a big fan of the Overcooked for studying ad-hoc teamwork. I believe it is too constrained. It would be interesting to see how this method performs/compares in a more open-ended task with more degrees of freedom. It might actually be easier to achieve diversity then, but harder to achieve specialisation. \n\n- Table 2 with all the abbreviations and colours look extremely busy. I believe in some parts of the paper the amount of coloured text and abbreviations make it actually harder to parse things rather than easier. This is probably also a personal constraint on my side. \n\n- Not a big deal per se but the related works section is incredibly sparse. It is literally 16 lines. I understand that the authors wanted to squeeze in a lot of content to the paper, but I would advise you to extend this a little bit. Perhaps not in the way of adding more citations but in the way of explaining the landscape a bit more. Although, you could also argue with this and say the stuff that is super relevant are already discussed in detail in the introduction, and left-overs are quickly explained in the Related Works section. \n\n- The limitations and also a potential discussion on the future work is left to the appendix. Now I do not like this. Instead of a Conclusion section regurgitating the paper I just read already, I would rather see a discussion on what kind of future work this opens up, and what are the limitations, **in the main paper.**", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the overall positive sentiment towards the paper and their thoughtful feedbacks.\n\n_We have included weaknesses mentioned by the reviewer in the general response. We hope that our responses will resolve the questions and concerns raised by the reviewer. We are happy to further discuss and clarify if the reviewer feel their comments are not addressed._", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H1fvO3RszG", "reviewer": "Reviewer_Vvdc", "summary": "This work studies partner diversity in the context of training a generalist agent.\nThe authors observe that XP-min approaches, while capable of producing behavioral diversity in its teammates, generates “handshaking`` behaviors—a kind of overfitting.\nWhile MP-reg aims to correct for this overfitting, the authors hypothesize this generates a “loss of specialization” (LOS) problem. \nThe authors empirically support this hypothesis in a controlled experiment, concluding  “unspecialized or overfit partners are not good training partners”. \nThis study leverages mathematically grounded definitinos of diversity, specialization, and overfitness (each of which requires domain knowledge and/or a trained specialist agent).\nThe authors propose “SpecTRL” and “SpecTRL DAgger”, both of which operate on a pre-trained XP-min (and/or mutual information (MI)) based population, but aim to “reduce overfitness while maintaining the diversity and specialization” which already exists in the population.\nThe main idea of SpecTRL is for the distilled partners to learn to cooperate with the specialization of the XP-min agents, but not their sabotaging behaviors.\nSpecTRL DAgger introduces supervision, which cain aid the distilled agent in learning to utilize complex handshakes that may not be discovered through random exploration.\nThe experimental results demonstrate that MP-reg / MI increase diversity, but lose specialization—in line with the LOS hypothesis. \nOn the other hand, the SpecTRL-based approaches successfully reduces overfitting while preserving the specialization in the XP-min population.\nNotably, while SpecTRL’s distillation phase reduces overfitting, repeated distillation does not further reduce it.", "questions": "* I think I have some intuitive understanding for why distillation instead of regularization during training may help prevent the overfitting issue while maintaining specialization, but could the authors elaborate on this aspect? A key component of the method appears to be this distillation component, but from what I see there’s not so much discussion about the motivation/intuition/inspiration for this.", "limitations": "The authors appear to adequately address all major limitations in Appendix G: \n\n* The proposed measures are just one proposition—there may be other sets that better quantify the quality of training populations.\n* Most notably, domain knowledge is necessary for the behavior characteristic function (f), and for training the oracle specialists. \n* Evaluated on single domain (expressed as weakness above).", "rating": 7, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "* Work appears to be novel and grounded in relevant literature.\n* The paper is well-written and very well-formatted; easy to read.\n* The results are significant, and will likely be of use to the MARL community.", "weaknesses": "* The limitations section is currently in the Appendix, but as this works’ analyses contains notable limitations, the authors may consider putting this in the main body.\n* The results are only validated on the Overcooked environment. While the results on this environment are thorough and promising, it is difficult to be confident in the generality of the findings without validation on more domains.’\n* Table 2 is a bit hard to read; it’s nice for seeing the raw data, but it’s a lot of numbers to try to make sense of. The visual aspects of the Table are nice, but plots might have made these easier to interpret (and the raw data could have been placed in the Appendix).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the overall positive sentiment towards the paper and their thoughtful feedbacks.\n\nHere, we address the question raised by the reviewer\n> I think I have some intuitive understanding for why distillation instead of regularization during training may help prevent the overfitting issue while maintaining specialization, but could the authors elaborate on this aspect? A key component of the method appears to be this distillation component, but from what I see there’s not so much discussion about the motivation/intuition/inspiration for this.\n\nOur aim for the distilled population is for them to learn diverse and specialized behaviors of a source XP-min population while removing sabotaging behavior (i.e., overfitness). The reason we think reward maximization (RL) objective would achieve this is that the distilled partners are incentivized to nudge the source partners to perform cooperative behaviors (which gives high return) and away from their sabotaging behaviors (which gives low return). Therefore, the distilled partners would not learn the sabotaging behaviors. Additionally, when the source partners cooperate, they do so in specialized ways as they have already learned specialized behaviors with XP-min. This means that the distilled partners will also learn these specialized behaviors as well. We’ll include more of this intuition in section 5 in the camera ready version.\n\n\n_We have also included weaknesses mentioned by the reviewer in the general response. We hope that our responses will resolve the questions and concerns raised by the reviewer. We are happy to further discuss and clarify if the reviewer feel their comments are not addressed._", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6CvtDqKFkS", "reviewer": "Reviewer_G3wa", "summary": "The paper argues that while diversity among training partners is essential for developing robust generalist cooperative agents, specialization also plays a crucial role. The authors introduce a method for quantifying both diversity and specialization using mutual information. They highlight the limitations of the cross-play minimization (XP-min) technique, which generates diverse but overfitted partners. To address this issue, the authors propose reinforcement learning and supervised learning methods to extract beneficial behaviors while reducing overfitting. Empirical results demonstrate that these methods lead to more robust generalist agents.", "questions": "* Can you elaborate on the computational requirements for implementing SpecTRL and SpecTRL DAgger in different environments?\n* What strategies can be employed to identify or create an oracle generalist in environments where one is not readily available?\n* I noticed that the hyperparameter selection process is detailed in the appendix. However, I am still concerned: is the algorithm sensitive to hyperparameters, and can minor changes in hyperparameters cause significant performance degradation?", "limitations": "The authors discuss the limitations in detail in the appendix. The discussion is reasonable, and it is impractical to completely address them in the current version of the paper.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "* The paper introduces a novel method to quantify partner diversity and specialization using mutual information.\n* It effectively identifies the issue of overfitting in partners generated by the XP-min technique.\n* The proposed methods are empirically validated, showing improvement in training robust generalist agents.\n* The paper provides a thorough analysis of how diversity and specialization impact the robustness of generalist agents.", "weaknesses": "* The experiments are conducted within a specific cooperative environment (multi-recipe Overcooked), which may limit the generalizability of the results.\n* The overfitness measurement relies on an oracle generalist, which may not always be available or practical in real-world scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the overall positive sentiment towards the paper and their thoughtful feedbacks.\n\nHere, we address the questions raised by the reviewer\n\n> Can you elaborate on the computational requirements for implementing SpecTRL and SpecTRL DAgger in different environments?\n\nThe computation cost of the distillation process of SpecTRL and SpecTRL DAgger is similar to that of self-play. One can approximate the training time of SpecTRL by referencing the training time of self-play in the same environment. Although the computation cost scales linearly with the number of the source partners, one could parallelize the distillation process (e.g., one distillation pair per CPU core). For reference, in the multi-recipe Overcooked, distilling 8 partners in parallel takes 12 hours.\n\n> What strategies can be employed to identify or create an oracle generalist in environments where one is not readily available?\n\nWe believe that an efficient way to generate oracles is via reward shaping alongside usual self-play training, which we also employed in this work. This approach allows us to exert domain knowledge for the design of the oracles without requiring us to program the oracles’ behaviors.\n\n> I noticed that the hyperparameter selection process is detailed in the appendix. However, I am still concerned: is the algorithm sensitive to hyperparameters, and can minor changes in hyperparameters cause significant performance degradation?\n\nOur proposed distillation method only introduces one additional hyperparameter, $\\lambda_{dagger}$, for the SpecTRL DAgger variant. We do agree that this hyperparameter could cause performance degradation if $\\lambda_{dagger}$ is too big. We suggest that one should start using SpecTRL with the DAgger component (i.e. $\\lambda_{dagger}=0$). Then, if the performance improvement is unsatisfactory, start introducing the DAgger component with a small coefficient (e.g., 0.01) and incrementally increase the coefficient value (e.g., 0.1 or 0.2).\n\n_We have also included weaknesses mentioned by the reviewer in the general response. We hope that our responses will resolve the questions and concerns raised by the reviewer. We are happy to further discuss and clarify if the reviewer feel their comments are not addressed._", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
# General response to all reviewers

We thank all the reviewers for the overall positive sentiment towards the paper and their thoughtful feedbacks. Here, we address common concerns among the reviewers:

- __Only evaluating with multi-recipe Overcooked (reviewer sQwt, Vvdc, and G3wa)__

We agree with the reviewers that evaluation in different domains would be beneficial. While we focus on the multi-recipe Overcooked environment due to the scarcity of sufficiently complex cooperative settings designed for ad-hoc teamwork, future research could explore the generalizability of our findings to other domains as such environments become available since the algorithm is domain agnostic and should be readily applied to new domains. We believe that, given the current research landscape, multi-recipe Overcooked—a  complex coordination task that covers various coordination challenges—provides a robust and suitable benchmark for this investigation. 

- __Table 2 is hard to read (reviewer sQwt, Vvdc)__

We will improve legibility of the table in the camera-ready version.

- __Limitations and discussions are not in the main text and limited related work (reviewer sQwt, Vvdc)__

We’ll include the limitations and discussions sections (which are now located in the appendices) in the additional page given for the camera-ready version. We’ll also provide an extended related work section.


_We hope that our responses will resolve the questions and concerns raised by the reviewers. We are happy to further discuss and clarify if any reviewers feel their comments are not addressed._

Meta-review: The paper considers the ad-hoc teamwork problem and proposes a method that distills the diversity and specialization of the partner population generated by the cross-play minimization technique. The proposed method is also empirically better than mixed-play regularization methods. 

All reviews are positive and acknowledge this paper's contribution to the field of ad hoc teamwork, though all reviewers are concerned that the experiments were conducted in only one environment (multi-recipe Overcooked). 

The authors should incorporate the promised modifications into the final version,
Decision: Accept (poster)