LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
Authors: Lihe Li
Lei Yuan
Pengsen Liu
Tao Jiang
Yang Yu
Venue: ICML 2025 poster (2025)
Keywords: Multi-agent Reinforcement Learning
Large Language Models
Multi-agent Coordination
Primary area: reinforcement_learning->multiagent
TL;DR: We utilize LLMs to propose novel coordination behaviors described in natural language, and then transform them into teammate policies, enhancing teammate diversity and interpretability.

Abstract: Training with diverse teammates is the key for learning generalizable agents. Typical approaches aim to generate diverse teammates by utilizing techniques like randomization, designing regularization terms, or reducing policy compatibility, etc. However, such teammates lack semantic information, resulting in inefficient teammate generation and poor adaptability of the agents. To tackle these challenges, we propose Semantically Diverse Teammate Generation (SemDiv), a novel framework leveraging the capabilities of large language models (LLMs) to discover and learn diverse coordination behaviors at the semantic level. In each iteration, SemDiv first generates a novel coordination behavior described in natural language, then translates it into a reward function to train a teammate policy. Once the policy is verified to be meaningful, novel, and aligned with the behavior, the agents train a policy for coordination. Through this iterative process, SemDiv efficiently generates a diverse set of semantically grounded teammates, enabling agents to develop specialized policies, and select the most suitable ones through language-based reasoning to adapt to unseen teammates. Experiments show that SemDiv generates teammates covering a wide range of coordination behaviors, including those unreachable by baseline methods. Evaluation across four MARL environments, each with five unseen representative teammates, demonstrates SemDiv's superior coordination and adaptability. Our code is available at https://github.com/lilh76/SemDiv.

LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
Lihe Li 2 Lei Yuan 123
Pengsen Liu" 12
Tao Jiang
123 Yang Yu 123
Abstract
1. Introduction
Recently, cooperative multi-agent reinforcement learning (MARL) has gained significant attention (Oroojlooy & Ha- jinezhad, 2023), demonstrating promising applications in various fields such as autonomous driving (Zhang et al., 2024c), domain calibration (Jiang et al., 2024), and fi- nancial trading (Huang et al., 2024). Classic MARL ap- proaches (Lowe et al., 2017; Rashid et al., 2018; Wang et al., 2021; Yu al., 2022) primarily focus on training group of agents to cooperatively complete specific tasks and evaluate their performance in the same setting. However, in open multi-agent environments (Yuan et al., 2023b), agents are often required to team up with unseen teammates exhibiting diverse coordination behaviors. For instance, autonomous driving agents frequently encounter human drivers with a wide range of driving behaviors. such scenarios, agents trained using conventional MARL techniques may strug- gle to coordinate effectively, they tend to overfit to the behaviors of their training teammates.
Training with diverse teammates is the key for learning generalizable agents. Typical approaches aim to generate diverse teammates by utilizing techniques like randomization, designing regu- larization terms, or reducing policy compatibil- ity, etc. However, such teammates lack seman- tic information, resulting in inefficient teammate generation and poor adaptability of the agents. To tackle these challenges. we propose Semanti- cally Diverse Teammate Generation (SEMDIVV), novel framework leveraging the capabilities of large language models (LLMs) to discover and learn diverse coordination behaviors at the seman- tic level. In each iteration, SEMDIV first gener- ates a novel coordination behavior described in natural language, then translates it into a reward function train teammate policy. Once the policy is verified to be meaningful, novel, and aligned with the behavior, the agents train pol- icy for coordination. Through this iterative pro- cess, SEMDIV efficiently generates a diverse set of semantically grounded teammates, enabling agents to develop specialized policies, and select the most suitable ones through language-based reasoning to adapt to unseen teammates. Ex- periments show that SEMDIV generates team- mates covering wide range of coordination be- haviors, including those unreachable by baseline methods. Evaluation across four MARL environ- ments, each with five unseen representative team- mates, demonstrates SEMDIV's superior coordi- nation and adaptability. Our code is available https: https:///githuie com hoom/iihh666 1eli1hh7640100 /
Training with diverse teammates is the key for learning gen- eralizable MARL agents. To generate diverse teammates, recent research in areas such as ad-hoc teamwork (Mirsky et al., 2022) and zero-shot coordination (Treutlein al., 2021) has emerged. FCP (Strouse al., 2021) trains team- mates using different random seeds, while TrajeDi (Lupu et al., 2021) and MEP (Zhao et al., 2023) introduce diver- sity regularization terms for teammates. Other methods like LIPO (Charakorn et al., 2023), Macop (Yuan et al., 2023a), BRDiv (Rahman et al., 2023), and L-BRDiv (Rah- man et al., 2024) induce diversity by reducing compatibility among teammates or between teammates and agents. While achieving some progress, these approaches primarily focus policy-level diversity, generating teammates that lack semantic information and are not grounded into specific coordination behaviors. This limitation results in two sig- nificant challenges. First, the exploration of the teammate policy space is inefficient, as teammates are driven to opti- mize for differences at the policy-level rather than actively discovering novel coordination behaviors at the semantic- level. Second, agents are unable to utilize semantic informa- tion, and limited to rril-and-erroo interactions for teammate adaptation, hindering their deployment in costly tasks.
'National Key Laboratory for Novel Software Technology, Nanjing University "School of Artificial Intelligence, Nanjing University "Polixir Tochnologies. Correspondence to: Lei Yuan <yuanl@lamda, lamda.jj..cdu.cn.. Yang Yu <yuy Syuu@@jj..du..n>>.
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

LLM--ssisttd Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
Training
"Male passes and Johuson tcores alone
Testing
Johason
Heiil
SEMDIV Agent
Level
Language
Tuuning
Kooppr
spand
have learned
"Lef Tuwing making one pass
Level
Coordinate using
Teamnaat
policy ?
poliyy space
Learn
"We prefer to make two passes before shooting
prefer that we makec pass before shooting,
SEMDIV Agent
Figure 1. An overview of the training and testing process of SEMDIV. Left: During training, SEMDIV proposes novel coordination behaviors natural language and transform them into teammate policies for agent learning. Right: During testing. SEMDIV takes as imput the description of the unseen teammates and selects the optimal learned policy for coordination.
To tackle these challenges, we propose Semantically Di- verse Teammate Generation (SEMDIV), a novel framework leveraging the capabilities of large language models (LLMs) to discover and learn diverse coordination behaviors at the semantic level, illustrated Figure 1. SEMDIV employs an iterative process: in each iteration, it first generates novel coordination behavior described in natural language, then translates it into reward function (Xie al., 2024; Ma et al., 2024a) to train a teammate policy. Once the policy is verified to be capable of completing the task, distinct from previous teammates, and aligned with the behavior, the agents with multi-head architecture (Kessler et al., 2022; Yuan al., 2024) train a new policy head for coordina- tion. Through this process, SEMDIV efficiently generates diverse set of semantically grounded teammates, enabling agents to develop specialized policies, and select the most suitable ones through language-basee reasoning to adapt to unseen teammates with specific coordination behaviors.
2. Problem Formulation
In this work, focus cooperative MARL tasks where agents need to coordinate with unseen and uncontrollable teammates. This problem can be formulated M = (N = Nag U. S, A, P,S, O, R. ?) by extending the Dec-POMDP framework (Oliehoek & Amato, 2016). Here, N is the set of all agents, divided into controllable agents Nog = {l,..., Uag) and uncontrollable teammates Nmm { 1...., aag + pum) S is the of global states, A A= Aug Au IIJEN, A I.E AA is the joint action space. P function, f is the set of observations, O N ft the observation function, Rt S A x S function, and step I,, agent i E A receives observation of = O(e i) Ω and outputs an action of E A' with policy n² (-lo") The joint action a, (at, ai Rag (ii) leads the next state S1+1 ~ P(-|st, a.) and a team reward R(s;, a,, St+1). The objective of the controllable agents is to find joint policy Th (-10")) = (-I0%)==II:: =IIEEN, T (:lo") that maximizes the expected return with unknown teammates th = IIKENN T², i.e., Egii [J """]] = Eq"" [E,, EE, 7" R((s,, ae st+1)]].
tuple
A Pr(S) is the transition
is the reward [0, 1) is the discount factor. At each time
conduct experiments across four MARL environments, including Level-Based Foraging (LBF) (Papoudakis al., 2021), Predator- Prey (PP) (Lowe et al., 2017), StarCraft Multi-Agent Challenge-v2 (SMACv2) (Ellis et al., 2023), and Google Research Football (GRF) (Kurach et al., 2020). SEMDIV successfully generates teammates with novel co- ordination behaviors unreachable by policy-level baselines, for example, multiple passes in GRF Teaming up with five unseen teammates with distinct and representative coordina- tion behaviors each of the four environments, SEMDIV's agents outperform the best baseline by 19% for task success rate and 39% for the success rate of satisfying the teammates preferred coordination behaviors. These results highlight the capability of SEMDIV to train adaptive agents with strong coordination ability in open multi- environments.
As we aim study teammate generation and agents coordi- nation at the semantic-level, we consider scenarios in which the group of teammates' Th provides a natural language description prior to the execution phase. This description outlines their preferred coordination behaviors, such as a specific plan to complete the task, or the occurrence of a particular coordination event, etc. The agents can leverage this natural language description b to adapt their individual policies n. J,N,,, thereby aligning their actions with the co-
'For simplicity, denote group of teammates "a team- mate" hereafter when no ambiguity arises.
2

---

LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
(x) Generating coordination behavior
(b) Truining aligned teummate policy
Training agents
(1)
Teammate pollcy
Continual learning
Task description
LLM behavior generator
LLM reward generator
ea. "Cissagp will swotter Schanie
Instruction
Vh-1
t/
Policy hends
Previous behaviors
Feedback
MARL Training
"Le foleus costrar boll...
(Seniiy) to sythor mann
(Prignam outpptt
Feature extractor
by-s "Pars our new and poory... ...
(Billutioon resurss
Similarity information
Avlarvius and are vimilor with each other...
Coordination behavior 6n two passes before shooting...
Reward Function Def reward1(self:: -> float
Agents
Figure 2. The overall workflow of SEMDIV. (a) Generating coordination behavior. SEMDIV iteratively generates of semantically diverse coordination behaviors, enabling efficient exploration of the teammate policy space. (b) Training aligned teammate policy. For each coordination behavior described in natural language, a tcammate policy is trained to align with that behavior. (c) Training agents. Agents continually trained with these teammates, developing strong coordination ability.
ordination preferences of π|",, ultimately enhancing overall team coordination and task performance.
they need to complete. The instruction inst is a simple sentence like "come up with a possible and concrete coor- dination behavior". When > to ensure novelty and diversity in each iteration, the prompt also includes previ- ous behaviors B = {bm E .b.....,bb.11 Im True), with explicit instructions in inst for the LLM to avoid replicating these behaviors while proposing a new one. Fur- thermore, to ensure meaningful diversity in the generated teammates, SEMDIV incorporates feedback mechanism refine the behavior generation process. Specifically, when pair of policies T π m'mmm Pn-1 are similar with each other, this information info simm i is fed back into the LLM prompt. For example, in a navigation task, different behav- iors such as "move to point A" and "move to coordinate (3, 4)" might produce similar policies if point close to (3,4), 4). By identifying such redundancies, process elabo- rated later, the LLM gains deeper understanding of the coordination task. This grounding feedback enables SEM- DIV to iteratively generate coordination sehavior-policy pairs that are diverse at both semantic and policy levels, enhancing exploration of the policy space. The full prompts for the LLM behavior generator are in Appendix F.2.
3. Method
This section introduces SEMDIV (Figure 2), a novel frame- work that leverages LLMs to efficiently generate semanti- cally diverse teammates, and train agents with strong coordi- nation ability. SEMDIV begins with the iterative generation of semantically diverse coordination behaviors, enabling efficient exploration of the teammate policy space (Sec- tion 3). For each coordination behavior described in natu- ral language,& teammate policy is trained to align with that behavior (Section 3.2). Simultancously, agents are continu- ally trained with these teammates, enabling them to develop strong coordination ability and adapt efficiently to unseen teammates during execution (Section 3.3).
3.1. Iterative Generation Semantically Diverse Coordination Behaviors
To derive semantically diverse teammates in cooperative MARL task, SEMDIV first leverages an LLM to iteratively generate a diverse set of plausible coordination behaviors described in natural language.
Next, the LLM behavior generator utilizes the prompt p [desc, inst, B, info imm, along with its internal knowledge, to output new concrete behavior b, in nat- ural language. This behavior is then used to generate a corresponding policy Tr If TT demonstrates the intended behavior b,,, is different from previous policies in Pn--1 and completes the task, I, is set to True. Otherwise, 7, is set to False. Then, P, Pn- U (bn, т!!!! {,)) This iterative process continues until sufficient number of valid team- mates are generated, fostering the development of agents with strong coordination capabilities.
Concretely, let Pa-1 = (bm, -m (m)) m=) denote the set iterations,
of teammates generated the previous where each tuple (bm, Tm Im) consists of a behavior bmx its corresponding policy =mtm and boolean value T.. in- dicating whether the teammate is valid (4m = True) or not (Im Fa False). In the thth iteration, the LLM behavior generator takes a task description desc and an instruction inst as prompts. The description desc includes the basic information about the environment, the agents, and the task
3

---

LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
3.2. Grounded Generation of Each Single Teammate
puted as the sum of two components: the task-specific reward rt R(st, at, 8t+1) and the generated reward Pl" = Rm axs St+1). For the weighting factors, À = 1, X2 decays from I to 0 over the course of training. This decay ensures that mm learns to complete the task.
This section describes how SEMDIV generates a teammate policy that aligns with a specified coordination behavior and completes the MARL task, while ensuring that the teammate policy is distinct from previously generated ones.
Policy Verification After training T SEMDIV veri- fies its validity. First, it evaluates ( π (m) for multiple episodes to compute returns for re and (.) checking issues (ii) failure to complete the task, and (iii) non-functional re- wards. For issue (iv), SEMDIV extracts the main information in these episodes, transforms it into natural language, and uses an LLM to confirm that nM demonstrates the intended coordination behavior bm For issue (V), we assume a joint agent policy T that can effectively coordinate with all pre- vious teammates -1 {TT]TT PP- 1 True), ) which will be elaborated in the next section. To confirm that Th is distinct from m- we follow (Charakorn et al., 2023) and check whether the following condition holds:
Prompts to Reward Functions Within each iteration, given a coordination behavior 8m. SEMDIV uses an LLM to generate a corresponding reward function Aim= : S x A x > R as an executable program. Similar to the behavior generator, the LLM reward generator takes the task descrip- tion, an instruction, and feedback information as prompts. The task description must include basic callable attributes and APIS to ground the reward function in the task envi- ronment. For instance, in a 3D navigation task, attributes like agent
np. ap.nde .nwarra [ (3,) ) and APIs like distance calculation functions should be provided. The instruction is a sentence like "write a reward function that formats 'def reward(self) float' and aligns with the coordination behavior }bm. However, with only the task description and instruction, the generated reward may not be able to train a valid teammate for several issues: (i) The reward function is not executable, c.g., it calls an unde- fined attribute. (ii) The teammate fails to complete the task after training with this reward function. (iii) The return of the reward function remains nearly constant during training. indicating that it's non-functional. (iv) The teammate does not demonstrate the intended coordination behavior bm (v) The teammate is similar to previously generated ones.
J J( n !!!)
J( TI mmm))
> E,
(2)
J [T π
tm E 11m-1. under configurations W1 = 1, A2 = 0 for all / and \₁ = 0, A2 1, where e > 0 is a predefined threshold. If this condition is satisfied, πh is confirmed to be distinct, as T cannot effectively coordinate with it. Otherwise, similarity information is recorded and provided as feedback to the LLM behavior generator, as described in Section 3.1. This verification process ensures the quality and diversity of each generated teammate. The prompts used for behavior- policy alignment verification are detailed in Appendix F.4.
To address these issues, SEMDIV incorporates the above critical grounding feedback into subsequent prompts it- eratively refine the reward function. This iterative process continues until either valid teammate policy T! is learned or the maximum number of attempts is reached. A valid policy is one that satisfies all verification criteria (described below), at which point the tuple (bm, Th, True) is added to Pm--1 If the maximum number of attempts is reached, .m (bm: π n = nul null, Fa se) is added instead. The prompts for this LLM reward generator are in Appendix F.3.
3.3. Continual Learning and Execution of the Coordinating Agents
The goal of SEMDIV is to derive a joint agent policy T that can effectively coordinate with both self-generated and unseen teammates based on natural language descriptions of their coordination behaviors. As the coordination be- haviors of different teammates may vary significantly or even conflict with each other, it can be challenging to train a single policy network that coordinates effectively with all teammates. Additionally, when training with a newly generated teammate, the agent's policy may lose the ability to coordinate with previous ones due to network parameter updates, i.e., catastrophic forgetting.
Reward Functions to Policies Given an executable re- ward function R,,. SEMDIV incorporates it into the environ- ment code and leverages an off-the-shelf cooperative MARL algorithm to train the teammate policy -hm. The training objective is to maximize the self-play return defined as:
To address these challenges, SEMDIV adopts a multi-head network architecture (Kessler et al., 2022; Yuan et al., 2024) and empowers the agents with continual learning ability.
J( m: - Esy,a
E (Arm++
(Aire + S2fM))
For each individual agent n'ENG, the policy network is de- composed into a feature extractor for and multiple policy heads (har Jy=1, }/= where n [ ! represents the number of valid teammates generated up to the th it- eration. For simplicity, we ignore invalid teammates and
where * is the complementary policy of mi which con- trols agents Nug. It outputs actions (a) ..., a, ). which are combined with the actions (a, Putl ...., aj Pua Pun out- put by πt tm to form the joint action a. Rewards are com- m
4

---

LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
assume all teammates in Pn are valid (mm True) in this part. For new generated teammate n+11 trained by reward pm+t1 to demonstrate behavior bn+1, SEMDIV first instantiates a new policy head hgg.n+s for the agent's ' co- ordination with this new teammate. The joint agent policy - = ILNN, , n² = ILEN,, fai o higinii 12 is then trained to coordinate with 1 by maximizing the objective:
reasoning, avoiding the need for trial-and-error interactions and significantly improving efficiency. The prompts for this LLM are provided in Appendix F.5.
4. Experiments
In this section, we conduct a series of experiments to ad- dress the following questions: (1) Can SEMDIV effectively coordinate with unseen teammates who provide descriptions of their coordination behaviors (Section 4.2)? (2) How does SEMDIV operate in detail during a single run (Section 4.3)? (3) Can baselines achieve the performance of SEMDIV by increasing the population size (Section 4.4)?
J(T T m++1 = EE, ar Σn (re + )er
(3)
where A2 is the same decaying factor with the one used in Equation (I). Different checkpoints of x are utilized for sampling to improve generalization. During training, the policy heads { } remain fixed, and gradients only propagate through for and the new head bgg.nts. Since the feature extractors faz are already well-trained to capture the common features of the task, 7 can quickly adapt to new leammates. However, TheH may lose the coordinate ability with previous teammates if Jo updates dramatically, i.e., catastrophic forgetting. So, SEMDIV applies a regulariza- tion term to constrain the update, forming the final objective for training the joint agent policy:
4.1. Environments, Teammates, and Baselines
We evaluate SEMDIV and baseline methods across four classic multi-agent coordination environments. The first is Level-Based Foraging (LBF) (Papoudakis et al., 2021), a grid- world scenario where agents coordinate to collect food items together. Next, we introduce a modified version of the Predator-Prey (PP) (Lowe et al., 2017) environment, incorporating two prey types to enhance complexity. We then conduct experiments using the StarCraft Multi-Agent Challenge-v2 (SMACv2) (Ellis et al., 2023), which tasks agents with controlling StarCraft units to defeat enemies controlled by the game's built-in AI. SMACv2 improves upon SMAC (Samvelyan et al., 2019) by introducing fea- tures like randomized start positions, making it significantly more challenging. Finally, we test in Google Research Football (GRF) (Kurach et al., 2020), where agents con- trol football players aiming to score through diverse tactics. Detailed introduction are provided in Appendix D.1.
max U(T" m=11 a Wagl
Σ iEN1
1100- -o'-- 0'ppp (4)
di -0 s.Nr,
where J is the objective defined in Equation (3), a is hyperparameter, ø' is the snapshot of parameters o² after training with the last teammate ,. and || |Ip represents the Ip norm. This learning framework effectively balances the need to adapt to new tcammates while preserving the ability to coordinate with previous ones. It has excellent scalability as the number of diverse teammates increases during training. Once the training process is complete, SEM- DIV produces joint agent policy T with set of policy heads [han.s }. each tailored to coordinate with a class of teammates exhibiting a specific coordination behavior by. It is worth noting that, the agents are equipped with contin- ual learning ability to adapt to future teammates that may appear after this training process, showcasing potential for online real-world applications.
In each environment, we train five teammates exhibiting distinct and representative coordination behaviors. For ex- ample, in GRE, we train teammates that prefer scoring after completing one or two passes. These teammates, along with their behavior descriptions, remain entirely unknown to the tested methods during training, ensuring an unbiased performance evaluation. To assess whether agents can effec- tively coordinate with these teammates to complete tasks, we measure the task success rates, denoted as R1. Addi- tionally, we evaluate the success rate of agents in satisfying the teammates' preferred coordination behaviors, denoted as R2. Detailed introduction of the testing teammates are illustrated in Appendix D.2.
During the execution phase, the agents need to coordinate with an unseen teammate π with coordination behavior b deseribed in natural language. SEMDIV utilizes an LLM to select the optimal policy head for the agents before rollout. This LLM selector takes the task description, learned behav- iors {b, Lj True), behavior b, and an instruction as prompts. The instruction is a sentence like "select the policy that can best coordinate with the teummate". Then, the LLM outputs the index k of the selected head hyg... Finally, each individual agent uses x² = far o h to effectively coordinate with teammate Tm This approach enables the agents to adapt to the teammate through language-basse based
Next, we present the implementation details of SEMDIV and the baselines for comparison. our experiments, we employ GPT-4o as the LLM For MARL algorithms, we utilize MAPPO (Yu et al., 2022) for GRF and VDN (Sune- hag et al., 2018) for other environments. We first compare
We use the opt-4c-2024-00 https://plat.ooms openai com/ docs/guides: /pt..
model via APIs
5

---

LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
Table 1. Coordination performance (mean + std) with unseen teammates across four environments. "R1" and "R2" represent the success rates of task completion and agents satisfying the teammates preferred coordination behaviors, respectively. The best result in each column, excluding performance upper bounds of SEMDIV (denoted in gray), is highlighted in bold.
LBF
PP
SMACv2
GRF
Average
Methods
R1
R2
R1
R2
R1
R2
R1
R2
R1
R2
Oracle
1.00
1.00
0.91
0.90
0.94
0.93
0.95
0.95
0.95
0.95
SEMDIV
0.90 10.05
0.90 10.05
0.72 10.03
0.54 10.10
0.65 :0.02
0.64 10.02
0.67 10.08
0.62 10.07
0.74
0.68
SEMDIV-Dist
0.45 10.14
0.45 10.14
0.51 10.03
0.28 1:005
0.24 1:008
0.23 2.008
0.47 10020
0.37 10.16
0.42
0.33
SEMDIV-RI
0.91 #0.04
0..1
#0.04
0.76 #0.01
0.53 #0.04
0.70
#0.00
0.69 #0.01
0.88 #0006
0.62 #0.08
0.81
0.69
SEMDIV-R2
0.91 20.04
0.91
#0.04
0.74 #0.01
0.58
=:0 06
0.70 =: 00
0.69 =0.01
0.78 20.08
0.73 20.05
0.78
0.73
Macop-RI
0.82 10.10
0.81 10.11
0.58 10.02
0.23 1:000
0.48 1:003
0.45 10.03
0.59
15
0.44 10.01
0.62
0.48
Macop-R2
0.82 10.10
0.81 #0.11
0.54 #0.01
0.25 1:000
0.47 1:00 03
0.45 #0.03
0.56 10.15
0.45 #0.03
0.60
0.49
SEMDIV-PBT
0.64 20.02
0.64 #0.02
0.70 =0.01
0.31 #0.01
0.61 20:01
0.61 #0.01
0.57 10.30
0.39 12.12
0.63
0.49
Macop-PBT
0.67 10.00
(166) 10.02
0.72 10.03
0.33 10003
0.56 10.04
0.54 10.03
0.49 10.21
0.35 10.10
0.60
0.46
FCP
0.46 #0.22
0.43 20.20
0.57 #0. 23
0.21 #0.15
0.40 #0.05
0.37 #0.08
0.50 #0.25
0.36 #0.12
0.48
0.34
MEP
0.57 #0.08
0.56 +0.08
0.70 #0.01
0.31 #0.01
0.55 =0.04
0.47 #0.02
0.50 #0.26
0.35 #0.14
0.58
0.42
LIPO
0.54 +0.00
0.51 #0.02
0.69
10.00 02
0.31 10.01
0.45 10:10
0.38 #0.06
0.51 20.25
0.37 10.12
0.55
0.39
LLM-Agent
0.88 #0.05
0.88 #0.05
0.71 #0.00
0.53 20.08
0.35 #0.10
0.35 #0.10
0.14 #0.09
0.12 #0.09
0.52
0.47
SEMDIV with classic two-stage population- based training (PBT) methods that induce diversity at the policy level, in- cluding FCP (Strouse al., 2021), MEP (Zhao et al., 2023), and LIPO (Charakorn et al., 2023). These methods train population of diverse teammates using different techniques in the first stage, and use them to train agents in the sec- ond stage. Then, we compare SEMDIV with Macop (Yuan et al., 2023a), which employs an iterative process similar to SEMDIV but generates new teammates by minimizing compatibility with agents. For a fair comparison, we derive total of 6 teammates and extract their three checkpoints: the initial middle, and final stages of training (Strouse et al., 2021). This results in 3 checkpoints per teammate and a total of 18 teammate policies for agent training across all meth- ods. To analyze the quality of the generated teammates and the impact of the multi-head architecture, we use the team- mates of SEMDIV and Macop as the first-stage teammates in PBT methods, denoted as (SEMDIV, Macop)-PDD To in- vestigate the head selection module, we include (SEMDIV, Macop)-R1 and -22, which report the results of the heads with the highest RI or R2 values, serving as upper bounds. Additionally, we introduce SEMDIV-Dist. an ablation of SEMDIV that selects heads based on the distance between embeddings of behavior descriptions, computed using T5- XL model (Chung et al., 2024). Since SEMDIV combines the strengths of MARL and LLMs, we also include base- line LLM-Agent that uses LLM only, assess the necessity of MARL. All methods are evaluated over three random seeds. Finally, we report the self-play performance of test- ing teammates as upper bounds (Oracle). Further details for SEMDIV and the baselines are in Appendix B and C.
4.2. Competitive Results
In this section, we present the overall results of SEMDIV, its ablations, and the baseline methods when coordinat- ing with unseen teammates across four environments. As shown in Table 1, the classic method FCP demonstrates poor performance, due to its limited ability to generate sufficiently diverse teammates. In contrast, methods that incorporate additional diversity objectives, such as MEP and LIPO, show improved performance, highlighting the importance of fostering distinct coordination behaviors that cannot be captured by simply training with varied seeds. However, all these two-stage PBT methods exhibit limited coordination ability. When we replace the first-stage team- mates with those generated by SEMDIV or Macop (*-PBT), performance improves significantly, suggesting that the two- stage framework struggles to generate sufficiently diverse teammates without considering the agents. Among these PBT methods, SEMDIV-PBT achieves the best results (see the third block of the table), demonstrating that SEMDIV generates teammates with superior quality and diversity.
Further analysis reveals that single policy network is insuf- ficient to effectively adapt to all distinct teammates, i.c., the multi-modalityy issue. The multi-head versions of SEMDIV and Macop (second table block) outperform their PBT coun- terparts, indicating that multi-head architecture can address this issue. Next, SEMDIV consistently outperforms all base- lines, demonstrating the effectiveness of its semantically diverse teammate generation. In the multi-head settings, SEMDIV leverages an LLM to understand the behaviors and coordination tasks, thus selecting matched policy heads.
6

---

LLM--ssisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
"Execute pass before taking shot the goul."
SEMDIV Termmatz
Johnson
Joharon scutes alome
0.6
60
0.5
50
Turing
pias
0
30
0.2
RI tormmate
20
KI of agere
10
Johnsin dribbllss attracts defenie
R2 of agont
two passes
0.0
0.2
0.1
0.6
0.8
1.0 le?
13
t=
Step
(a)
(b)
(c)
Figure 3. A case study in the GRF environment. (a) Learning curves of the teammate and the agent in the first iteration of SEMDIV.
(b) An episode where the first generated teammate successfully scores goal and demonstrates the desired coordination behavior. (c)
Trajectories visualization of the 12 teammates generated hy SEMDIV and FCP.
It achieves results comparable to the upper bounds of -R1 and -R2, and outperforms the best baseline Macop by 19% for RI and 39% for R2. In contrast, SEMDIV-Don -ist selects heads based on embedding distances between behavior de- scriptions, and shows significant performance degradation, indicating that language embedding similarity alone is in- sufficient to address the complex task of head selection. Although SEMDIV still falls short of the Oracle baseline, can bridge the gap by generating more teammates or incorporating additional diversity objectives.
def revardl (swlf) :loal::
Check if the score event happens at this step iE self f.score:
8 Cleck if Lhaze is exectly
paza in the histosyy
lentzelf-pose history)
Cheek the paes is between the two players ... (Codes cnitted for ciarity)
herge rexard reinforce the destred beravior
100.0
Detauit return, no exera rewerd in other casee return 0.0
Figure 4, Python code example for reward calculation.
Additionally, while LLM-Agent performs comparably to SEMDIV in simpler tasks such as LBF and PP, it experi- ences severe performance degradation in more complex environments, highlighting the necessity of incorporating task-specific reinforcement learning for successful multi- agent coordination. More experimental results, including performance of each testing teammate, are in Appendix E.
tion is then incorporated into the reward wrapper class. Sub- sequently, SEMDIV applics the MAPPO (Yu et al., 2022) algorithm train the teammates to maximize both the task reward and the generated reward, as defined in Equation (1). The training results are shown in Figure 3(a). Upon com- pleting training, SEMDIV verifies the validity of the learned teammate policy. First, as shown in the learning curves, at the early stage of training, the teammate occasionally scores goals without completing the desired passing behav- ior, leading to discrepancy between the blue and green curves. As training goes, the teammate successfully learns to score while maximizing the generated reward. Second, trajectory data is extracted and translated into natural lan- guage, producing a summary: "In this episode, Johnson passed to Turing, and finally successfully scored a goal. The player who scored the goal is Turing. ... " Based on this summary, an LLM confirms that the policy aligns with the intended coordination behavior, Key steps of this episode are visualized in Figure 3(b). Third, the similarity check is skipped this is the first teammate. This coordination behavior and its corresponding teammate policy are thus validated as suitable for training the agent.
4.3. Case Study
To illustrate the functionality of SEMDIV in detail, we present a case study that demonstrates the teammate genera- tion process, agent training, and evaluation with an unseen teammate during a single run in the GRF environment.
At the beginning, the LLM behavior generator takes the de- signed prompt as input, and outputs possible coordination behavior: execute one pass before taking shot the goal. Based on this behavior and the context of the football game, the LLM reward generator outputs the corresponding reward function in Python, as shown in the example in Figure 4.
The generated function correctly utilizes the provided envi- ronment attributes encourage the teammate learn the specified passing tactic. The inclusion of well-doccmented comments enhances the reward's interpretability. This func-
Next, SEMDIV creates a new policy head for the agent, and trains it to coordinate with this teammate, as defined in
7

---

LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
Equation (4). For this initial teammate, the regularization coefficient is set to O. The agent efficiently learns to score goals with the teammate while executing the intended passing tactic, resulting in rapidly rising and overlapping learning curves shown in red and orange. This process is repeated iteratively until the agent is trained with six distinct valid teammates.
1.0
SEMDIV
SEXDIV-POT
PCP
0.8
0.6
0.4
3
6
24
48
# teammates trained with agents (LBF)
0.7
To assess the impact of the semantic-level exploration tech- nique on enhancing diversity among teammate policies, we visualize the generated trajectories. Specifically, we col- lect 100 trajectories for each of the six valid teammates, totaling 600 trajectories. For comparison, we also gather an equivalent dataset from six teammates generated dur- ing run using FCP (Strouse et al., 2021). From these trajectories, we extract those that result goal, convert them into vector representations, and apply t-SNE (Van der Maaten & Hinton, 2008) for visualization. As shown in Figure 3(c), the projection of SEMDIV exhibits broader and more dispersed coverage compared to FCP (highlighted in circles). This confirms that semantic-level exploration significantly enhances the coverage of the teammate policy space, ultimately enhancing the agent's coordination.
0.6
0.5
0.4
24
48
# teammates trained with agents (GRF)
Figure 5. Coordination performance with testing teammates when agents train with various numbers of generated teammates.
generating semantically diverse teammates not only enables more efficient exploration of the teammate policy space but also facilitates the discovery of coordination behaviors that policy-level exploration alone cannot cover. For instance, in the GRF enviromment, we observe that FCP and other baselines fail to discover complex tactics that pass multiple times. Furthermore, with its multi-heat architecture, SEM- DIV scales more effectively with the number of teammates, achieving significantly better performance than SEMDIV- PBT. This highlights the importance of specialized design that allows for rapid adaptation to unseen teammates.
Finally, the agent is evaluated with an unseen teammate. For example, teammate joins the team as Turing, the player at the center. Our agent controls the other player, Johnson, and needs to coordinate with Turing. Before the game begins, Turing describes his/her desired coordination behavior: "I prefer to score myself." The LLM head selector takes the task description, Turing's 's desired behavior, and behaviors the agent have learned, as inputs. It inferences that "This policy (the one described fulfillll Turing's 's desire to score, as it allows him to set up for a shot after receiving a pass.", and selects the optimal head. Equipped with the selected head, the team achieves an 88% scoring rate with the teammate, with all goals scored by Turing. This case study highlights the effectiveness of SEMDIV generating diverse teammate policies, enabling efficient coordination even with unseen teammates.
5. Related Work
In open multi-agent environments, the important factors of the environment or the multi-agent system may change un- expectedly (Yuan et al., 2023b). To handle the change of teammates, recent research in areas such as ad-hoc team- work (Mirsky et al., 2022; Wang et al., 2024a) and zero-shot coordination (Treutlein et al., 2021) has emerged. This line of work includes training paradigm design (Hu et al., 2020; Strouse et al., 2021), diverse teammate generation (Lupu et al., 2021; Zhao et al., 2023; Charakorn et al., 2023; Yuan al., 2023a; Rahman et al., 2023; 2024; Cui et al., 2023; Sarkar al., 2023), investigation of human bias (Yu et al., 2023a; Hu & Sadigh, 2023), goal deduction (Zhang et al., 2024d), and policy co-evolution for heterogeneous settings (Xue et al., 2024). Researchers also develop bench- marks (Wang et al., 2024b) to evaluate these methods. This paper further delves into this line of work utilizing the power of LLMs enhance teammates' semantic diversity.
4.4. The Impact of the Number of Teammates
One of the key factors affecting performance is the number of teammates with whom the agents train. To investigate its impact, we run SEMDIV, its variant SEMDIV-PBT, and the baseline FCP with different numbers of training teammates, and assess the agents' performance with the testing team- mates. As shown in Figure 5, when training with only one teammate, these methods degenerate to the same setting, showing almost identical performance. As the number of teammates increases, JEMDIV-PB'T outperforms FCP with the same number of training teammates, achieving compa- rable or even superior results to FCP with a significantly larger number of 48 teammates. This demonstrates that
LLMs have recently gained significant attention in multi- agent tasks due to their advanced capabilities in natural language processing and planning (Guo et al., 2024). One line of work utilize LLMS for language agents communi- cation (Park et al., 2023; Guan et al., 2024: Zhang et al., 2024b; Li et al., 2023a; Du et al., 2024; Wang et al., 2024c).
8

---

LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
Some other works utilize LLMs as multi-agent task planners, which can be classified into several key areas, including MARL subgoal generation (Li et al., 2023b), multi-agent path finding (Chen et al., 2024a), and multi-robo task plan- ning (Liu et al., 2024b; Chang et al., 2024). Despite these advancements, LLMs still face challenges handling low- level coordination in multi-agent settings. Rather than di- rectly deploying LLMs as coordinating agents, we leverage their capabilities to generate diverse teammates and adapt- ing policies, thereby combining the strengths of LLMs with MARL. We discuss more related work in Appendix A.
guage model-enhanced reinforcement learning: Concept, taxonomy, and methods. IEEE Transactions on Neural Networks and Learning Systems, pp. 1-21, 2024.
Carta, T., Oudeyer, P.-Y., Sigaud, O., and sylvain lamprier. EAGER: Asking and answering questions for automatic reward shaping in language-guided RL. In NeurIPS, 2022.
Chang, M., Chhablani, G., Clegg, A., Cote, M. D., Desai, R., Hlavac, M., Karashchuk, V., Krantz, J., Mottaghi, R., Parashar, P., Patki, S., Prasad, I., Puig, X., Rai, A., Ramrakhya, R., Tran, D., Truong, J., Turner, M., Un- dersander, F.,, and Yang, T.-Y. Partnr: A benchmark for planning and reasoning in embodied multi-agent tasks. preprint arXiv:2411.00088 2024.
6. Final Remarks
We propose novel framework of LLM-assisted Semanti- cally Diverse Teammate Generation (SEMDIV) for efficient multi-agent coordination. The framework utilizes LLMs to discover diverse coordination behaviors described in natural language, facilitating the training of teammate policies align- ing with these behaviors. Agents train with these teammates in a continual learning process, developing policies tailored to the coordination behaviors and enabling rapid adapta- tion testing teammates. Empirical results across various environments and with unseen teammates provide strong evidence of SEMDIV's effectiveness. Looking ahead, as more advanced MARL techniques and LLMs emerge with enhanced performance, SEMDIV has the potential to further improve agent generalization in complex real-world coordi- nation scenarios, such as embodied multi-agent tasks (Feng et al., 2025) for real-world applications.
Charakorn R., Manoonpong, P., and Dilokthanakul, N. Gen- erating diverse cooperative agents by learning incompati- ble policies. In ICLR, 2023.
Chen, W., Koenig, S., and Dilkina, B. Why solving multi- agent path finding with large language model has not succeeded yet. preprint arXiv:2401. 00360000...0.... 2024a.
Chen, X.-H., Wang, Z., Du, Y., Jiang, S., Fang, M., Yu, Y., and Wang. J. Policy learning from tutorial books via understanding, rehearsing and introspecting. In NeurlPS, 2024b.
Cho, K., van Merrienboer, B., Gülçehre, Ç., Bahdanau, D., Bougares, F., Schwenk, H., and Bengio, Y. Learning phrase representations using RNN encoder-decoder for statistical machine translation. In ACL, pp. 1724-1734, 2014.
Acknowledgements
This work was supported by the NSFC (62495093, U24A20324) and Jiangsu Science Foundation (BK20241199, BK20243039). We thank Tencent AI Arena for their support, and the anonymous reviewers for their support on improving the paper.
Choi, W., Kim, W. K., Kim, S., and Woo, H. Efficient policy adaptation with contrastive prompt ensemble for embodied agents. In NeurlPS, pp. 55442-55453 2023.
Chung, H. W., Hou, L., Longpre, S., Zoph, B., Tay, Y.,
Impact Statement
Fedus, W., Li, Y., Wang, X., Dehghani, M., Brahma, S., Webson, A., Gu, S., Dai, Z., Suzgun, M., Chen, X., Chowdhery, A., Castro-Ros, A.. Pellat, M., Robinson, K., Valter, D., Narang, S., Mishra, G., Yu, A., Zhao, V., Huang, Y., Dai, A.. Yu, H., Petrov, S., Chi, E. H., Dean, J., Devlin, J., Roberts, A., Zhou, D., Le, Q. V., and Wei, J. Scaling instruction-finetuue
The goal of the work presented in this paper is to advance the development of cooperative multi-agent reinforcement learning. The proposed framework is intended to enhance the generalization of coordinating agents, providing an ef- fective approach for future research on open multi-agent systems. Furthermore, the work presented does not raise any additional ethical concerns, and thus no special discus- sion on ethical issues is required.
language models. Journal of Machine Learning Research, 25(70): 1-3,, 2024.
Cui, B., Lupu, A., Sokota, S., Hu, H., Wu, D.J., and Foerster, J. N. Adversarial diversity in hanabi. In ICLR, 2023.
References
DeepSeek-AL. DeepSeek-R1. Incentivizing reasoning ca- pability in Ilms via reinforcement learning. preprint arXiv:250I. 12948, 2025.
Cao, Y., Zhao, H., Cheng, Y., Shu, T., Chen, Y., Liu, G., Liang, G., Zhao, J., Yan, and Li, Y. Survey on large lan-

---

LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
Dorri, A., Kanhere, S. S., and Jurdak, R. Multi-agent sys- tems: A survey. IEEE Access, 6:28573-28593, 2018.
M., Bousquet, O., and Gelly, S. Google research football:
A novel reinforcement learning environment. In AAAI, pp. 4501-4510, 2020.
Du, Y., Watkins, O., Wang, Z., Colas, C.. Damelll T., Abheel,
P., Gupta, A., and Andreas, J. Guiding pretraining in reinforcement learning with large language models. In ICML, pp. 8657-8677, 2023.
Kwon, M., Xie, S. M., Bullard, K., and Sadigh, D. Reward design with language models. In ICLR, 2023.
Li, G., Hammoud, H. A. A. K., Itani, H., Khizbullin, D., and Ghanem, B. Camel: Communicative agents for "mind" exploration of large language model society. In NeurlPS, 2023a.
Du, Y., Li, S., Torralba, A., Tenenbaum, J. B., and Mordatch, I. Improving factuality and reasoning in language models through multiagent debate. In ICML, pp. 11733-11763, 2024.
Li, W., Qiao, D., Wang, B., Wang, X., Jin, B., and Zha, H. Semantically aligned task decomposition in multi- agent reinforcement learning, preprint arXiv:2305.10865. 2023b.
Ellis, B., Cook, J., Moalla, S., Samvelyan, M., Sun, M., Mahajan, A., Foerster, J. N., and Whiteson, S. SMACv2: An improved benchmark for cooperative multi-agent re- inforcement learning. In NeurlPS Datasets and Bench- marks Track, 2023.
Lin, J., Du, Y., Watkins, O., Hafner, D., Abbeel, P., Klein, D., and Dragan, A. Learning to model the world with language. arXiv preprint vrXiv:2308.013990 2024.
Feng, 7., Xue, R., Yuan, L., Yu, Y., Ding, N., Liu, M., Gao, B., Sun, J., and Wang, G. Multi-agent embod- ied ai: Advances and future directions. arXiv preprint 250::2500000000 2025.
Liu, J., Zhong, Y., Hu, S., Fu, H., FU, Q. Chang, X., and Yang. Y. Maximum entropy heterogencous-agenn rein- forcement learning. In ICLR, 2024a.
Guan, C., Zhang, L., Fan, C., Li, Y.C.C Chen, F., Li, L., Tian, Y., Yuan, I.., and Yu, Y. Efficient human-Al coordination via preparatory language-basee -ased convention. In ICLR Workshop on Large Language Model Agents, 2024.
Liu, K., Tang, Z., Wang, D., Wang, Z., Zhao, B., and Li, X. Coherent: Collaboration of heterogeneous multi- robot system with large language models. preprint vrXiv:249.155166 2024b.
Guo, T., Chen, X., Wang, Y., Chang, R., Pei, S., Chawla, N. V., Wiest, O., and Zhang, X. Large language model based multi-agents: A survey of progress and challenges. In IJCAI, pp. 8048-801 08888057,222 2024.
Lowe, R., Wu, Y., Tamar, A., Abbeel, J. H. P., and Mor- datch, Multi-agent actor-critic for mixed cooperative- competitive environments. In NIPS, pp. 6379-6390, 2017.
Hu, and Sadigh, D. Language instructed reinforcement learning for human-ai coordination. In /CML, 2023.
Lupu, A., Cui, B., Hu, H., and Foerster, J. Trajectory diver- sity for zero-shot coordination. In ICML, pp. 2021.
Hu, H., Lerer, A., Peysakhovich, A., and Foerster, J. "other- play" for zero-shot coordination. In ICML, pp. 4399- 4410, 2020.
Ma, Y. J., Liang, W., Wang, G., Huang, D.-A., Bastani, O., Jayaraman, D., Zhu, Y., Fan, L., and Anandkumar, A. Eureka: Human-level reward design via coding large language models. In ICLR, 2024a.
Huang, Y., Zhou, C., Cui, K., and Lu, X. A multi-agent re- inforcement learning framework for optimizing financial trading strategies based on timesnet. Expert Systems with Applications, 237:1 21502, 2024.
Ma, Y. Liang, W., Wang, H., Zhu, Y., Fan, L., Bastani, O., and Jayaraman, D. Dreureka: Language model guided sim-to-real transfer. In Robotics: Science and Systems, 2024b.
Jiang, T., Yuan, L., Li, L.. Guan, C., Zhang, Z., and Yu, Y. Multi-agent domain calibration with a handful of offline data. In NeurIPS, 2024.
Kessler, S., Parker-Holdes J., Ball, P., Zohren, S., and Roberts, J. J. Same state, different task: Continual rein- forcement learning without interference. In Proceedings of the AAAI Conference on Artificial Intelligence, pp. 7143-7151,2033 2022.
Mirsky, R., Carlucho, L., Rahman, A., Fosong, E., Macke, W., Sridharan, M., Stone, P., and Albrecht, V. A survey of ad hoc teamwork: Definitions, methods, and open problems. preprint arXiv:2202.10450 2022.
Kurach, K., Raichuk, A., Stanczyk, P., Zajac, M., Bachem, O., Espeholt, I.., Riquelme, C., Vincent, D., Michalski,
Olichock, A. and Amato, C. A Concise Introduction to Decentralized POMDPs. Springer, 2016.
10

---

LLM--ssisttd Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
OpenAl. Learning reason with https://opent com index/ learning torrooo--------o---------o with 11ms/,
Spiegel, B. A., Yang, Z., Jurayi, W., Bachmann, B., Tellex, S., and Konidaris, G. Informing reinforcement learn- ing agents by grounding language to markov decision processes. In RLC Workshop on Training Agents with Foundation Models, 2024.
llms.
2024.
Accessed: 2024-09-19.
Oroojlooy, A. and Hajinezhad, D. A review of coopera- tive multi-agent deep reinforcement learning. Applied Intelligence, 53(11):13677-1333) 2023.
Strouse, D., McKee, K., Botvinick, M., Hughes, E., and Everett, R. Collaborating with humans without human data. In NeurlPS, pp. 14502-14515, 2021.
Paischer. F., Adler. T., Patil, V., Bitto-Nemling, A., Holzleit- ner, M., Lehner, S., Eghbal-Zadch. H., and Hochreiter, S. History compression via language models reinforce- ment learning. In ICML, pp. 17156-17185, 2022.
Sunehag, P., Lever, G., Gruslys, A., Czarnecki, W. M., Zam- baldi, V., Jaderberg, M., Lanctot, M., Sonnerat, N., Leibo, J. Z., Tuyls, K., et al. Value-decomposition networks for cooperative multi-agent learning based on team reward. AAMAS, pp. 2085-2087, 2018.
Pang. J.-C., Yang, X., Yang, S.-H., Chen, X.-H., and Yu, Y.
Natural language instruction-followin with task-related language development and translation. In NeurIPS, pp. 9248-9278, 2023.
Treutlein, J., Dennis, M., Oesterheld, C., and Foerster, J. A new formalism, method and open issues for zero-shot coordination. In ICML, pp. 10413-10423, 2021.
Pang. J.-C., Yang, S.-H., Li, K., Zhang. J., Chen, X.-H., Tang, N., and Yu, Y. Knowledgeable agents by offline re- inforcement learning from large language model rollouts. In NewrlPS, 2024.
Van der Maaten, L. and Hinton, G. Visualizing data using t-sne. The Journal of Machine Learning Research, 9(11), 2008.
Wang. C., Rahman, A., Durugkar, I., Liebman, E., and Stone, P. N-agent ad hoc teamwork. In NeurIPS, pp. 111832-111862 2024a.
Papoudakis, G., Christianos, F., Schäfer, L., and Albrecht, S. V. Benchmarking multi-agent deep reinforcement learning algorithms in cooperative tasks. In NeurIPS Datasets and Benchmarks Track, 2021.
Wang. J., Ren, Z., Liu, T., Yu, Y., and Zhang, C. QPLEX: duplex dueling multi-agent q-learning, In ICLR, 2021.
Park, J. S., O'Brien,, J. C., Cai, C. J., Morris, M. R., Liang, P., and Bernstein, M. S. Generative agents: Interactive simulacra of human behavior. In UIST, pp. 2:1-2:22, 2023.
Wang, X., Wang, S., Liang, X., Zhao, D., Huang, J., Xu, X., Dai, B., and Miao, Q. Deep reinforcement learning: survey. IEEE Transactions on Neural Networks and Learning Systems, 2022.
Rahman, A., Fosong, E., Carlucho, I., and Albrecht, V. Generating teammates for training robust ad hoc team- work agents via best-response diversity. Transactions on Machine Learning Research, 2023, 2023.
Wang, X., Zhang, S., Zhang, W., Dong, W., Chen, J., Wen, Y., and Zhang, W. Zsc-eval: An evaluation toolkit and benchmark for multi-agent zero-shot coordination. In NeurIPS Datasets and Benchmarks Truck, 2024b.
Rahman, M., Cui, J., and Stone, P. Minimum coverage sets for training robust ad hoc teamwork agents. In AAAI, pp. 17523-17530, 2024.
Wang. Z., Mao, S., Wu, W., Ge, T., Wei, F., and Ji, H. Un- leashing the emergent cognitive synergy in large language models: A task-solving agent through multi- multi-persona self- collaboration. In NAACL, pp. 257-279, 2024c.
Rashid, T., Samvelyan, M., Schroeder, C., Farquhar, G., Foerster, J., and Whiteson, S. Qmix: Monotonic value function factorisation for deep multi-agent reinforcement learning. In ICML, pp. 4295-4304 2018.
Wen, M., Kuba, J. G., Lin, R., Zhang, W., Wen, Y., Wang, J., and Yang, Y. Multi-agent reinforcement learning is a sequence modeling problem. In NeurlPS, pp. 16509_ 16521, 2022.
Samvelyan, M., Rashid, T., de Witt, C. S., Farquhar, G., Nardelli, N., Rudner, T. G. J., Hung, C., Torr, P. H. S., Foerster, J. N., and Whiteson, S. The Starcraft multi-agent challenge. In AAMAS, pp. 2186-2188, 2019.
Wu, Y., Fan, Y., Liang, P. P., Azaria, A., Li, Y., and Mitchell, T. Read and reap the rewards: Learning to play atari with the help of instruction manuals, In ICLR Workshop on Reincarnating Reinforcement Learning, 2023. URL https: /ttp:///ppeereveew net/ forum? id=I GGT_GGUngVVN..
Sarkar, B., Shih, A., and Sadigh, D. Diverse conventions for human-ai collaboration. In NeurlPS, pp. 33115-23139, 2023.

---

LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
Xie, T., Zhao, S., Wu, C. H., Liu, Y., Luo, Q. Zhong, V., Yang, Y., and Yu, T. Text2reward: Reward shaping with language models for reinforcement learning. In ICLR, 2024.
reinforcement learning for autonomous driving: A survey, preprint arXiv:2408.0967.. 2024c.
Zhang, Z., Zhou, H.,, Imani, M., Lee, T., and Lan, T. Collab- orative ai teaming in unknown environments via active goal deduction. preprint arXiv:2403. 15341, 2024d.
Xue, K., Wang. Y. Yuan, L., Guan, C.,, Qian, C., and Yu, Y. Heterogeneous multi-agent zero-shot coordination coevolution. IEEE Transactions Evolutionary Compu- tation, 2024.
Zhao, R., Song, J., Haifeng, H., Gao, Y. Wu, Y., Sun, Z., and Wei, Maximum entropy population based training for zero-shot human-Al coordination. In AAAI, pp. 6145- 6153, 2023.
Yu, C., Velu, A., Vinitsky, E., Gao, J., Wang, Y., Bayen, A., and Wu, Y. The surprising effectiveness of PPO in cooperative multi-agent games. In NeurlPS, volume 35, pp. 2461 1-24624, 2022.
Yu, C., Gao, J., Liu, W., Xu, B., Tang, H., Yang, J., Wang. Y., and Wu, Y. Learning zero-shot cooperation with humans, assuming humans are biased. In ICLR, 2023a.
Yu, W., Gileadi, N., Fu, C., Kirmani, S., Lee, K., Arenas, G., Chiang, H. L.,, Frez, T., Hasenclever, I.., Humplik, J., Ichter, B., Xiao, T. Xu, P., Zeng, A., Zhang, T., Heess, N., Sadigh, D., Tan, J., Tassa, Y., and Xia, F. Language to rewards for robotic skill synthesis. In CoRL, pp. 374 2023b.
Yuan, L., Li, L., Zhang, Z., Chen, F., Zhang, T., Guan, C.,, Yu, Y., and Zhou, Z. Learning coordinate with any- one. In Proceedings of the Fifth International Conference on Distributed Artificial Intelligence, 2023a.
Yuan, L., Zhang, Z., Li, L., Guan, C., and Yu, Y. survey of progress cooperative multi-agent reinforcement lear- ing in open environment. SCIENCE CHINA Information Sciences, 2023b.
Yuan, L., Li, L., Zhang, Z., Zhang, F., Guan, C., and Yu, Y. Multi-agent continual coordination via progressive task contextualization. IEEE Transactions Neural Networks and Learning Systems, 2024.
Zhang, A., Nguyen, K., Tuyls, J., Lin, A., and Narasimhan, K. Language-gun guidedd world models: A model-based ap- proach to ai control. arXiv preprint arXiv:2402.01699, 2024a.
Zhang, H., Du, W., Shan, J., Zhou, Q., Du, Y., Tenenbaum, B., Shu, T., and Gan, C. Building cooperative embodied agents modularly with large language models. In ICLR, 2024b.
Zhang, K., Yang, 7.,, and Başar, Multi-agent reinforce- ment learning: A selective overview of theories and al- gorithms. Handbook of Reinforcement Learning and Control, pp. 321-384, 2021.
Zhang, R., Hou, J., Walter, F., Gu, S., Guan, J., Röhrbein, F,, Du, Y., Cai, Chen, G., and Knoll, Multi-agent
12

---

LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
A. More Related Work
Cooperative Multi-Agent Reinforcement Learning (MARL) Many real-world problems, particularly those that are large-scale and complex, are inherently suited to be modeled as multi-agent systems (MASs) rather than single-agent systems due their efficiency and practicality in addressing intricate challenges (Dorri et al., 2018). Multi-agent reinforcement learning (MARL) (Zhang et al., 2021) has emerged as powerful framework for tackling these problems, leveraging the problem-solving capabilities of deep reinforcement learning (Wang et al., 2022). When agents within a MAS share common objectives, the problem falls under the category of cooperative MARL (Oroojlooy & Hajinezhad, 2023), which has demonstrated significant success across diverse domains such as autonomous driving (Zhang et al., 2024c), domain calibration (Jiang et al., 2024), and financial trading (Huang et al., 2024). Recent advancements in MARL have introduced a variety of approaches to improve agent coordination. These include policy- based methods such as MADDPG (Lowe et al., 2017) and MAPPO (Yu al., 2022), value-based techniques like VDN (Sunehag et al., 2018) and QMIX (Rashid et al., 2018), as well as innovative approaches leveraging architectures such as the transformer (Wen et al., 2022). These methods have demonstrated exceptional coordination capabilities in diverse tasks, including SMAC (Samvelyan et al., 2019) and GRF (Kurach et al., 2020). In this paper, our method focuses on enhancing the generalization abilities of coordinating agents, aiming to improve their adaptability and performance across a wider range of potential teammates.
Large Language Models (LLMs) for RL The integration of large language models (LLMs) into reinforcement learning (RL) has emerged promising research direction (Cao et al., 2024), leveraging the rich semantic understanding and generalization capabilities of LLMs to enhance decision-making processes. Recent studies have explored the use of LLMs for tasks such as processing and translating task information (Paischer et al., 2022; Choi et al., 2023; Pang et al., 2023; Spiegel et al., 2024), to reduce the burden of network updates. Another line of work utilizes L.MSs as reward generator (Carta et al., 2022; Kwon et al., 2023; Wu et al., 2023; Yu et al., 2023b; Du et al., 2023) to guide RL algorithms. Specifically, some approaches (Xie et al., 2024; Ma et al., 2024a;b) explicitly generate executable codes as reward functions. LLMs are also utilized as world models (Pang et al., 2024; Chen et al., 2024b; Lin et al., 2024; Zhang et al., 2024a) as they are trained with rich real-world context, enhancing the sample efficiency of RL. In our work, we mainly utilize LLMs propose coordination behaviors described in natural language, reward generation, and eehavior-trajecter alignment verification.
B. Implementation Details of SEMDIV
In this section, we present the implementation details of SEMDIV. The cpt- model is utilized as the LLM. For MARL algorithms, we employ VDN (Sunehag et al., 2018) for the LBF, PP, and SMACv2 environments, and MAPPO (Yu et al., 2022) for GRF Specifically, our VDN implementation is based on the PyMARL codebase (Samvelyan et al., 2019)?. We adopt parameter sharing in the agent network architecture. The feature extractor f% is designed as 3-layer MLP followed by a GRU (Cho et al., 2014), while the policy head hgs is 3-layer MLP. Both the MLP and GRU have a hidden dimension of 64. The policy head processes the feature extractor's output to generate Q-values for all actions, which are subsequently aggregated by summing individual agents' Q-values to compute the joint Q-value. The architecture for teammate networks mirrors this design, differing only in having a single policy head. For MAPPO, we build upon the HARL codebase (Liu et al., 2024a)*. Unlike VDN, parameter sharing is not applied by default settings. For the actor networks, the final two-layer MLP serves as the policy head, and the remaining components form the feature extractor. The critic networks are left unmodified. A single run of SEMDIV incurs cost of approximately $0.10 for OpenAl APIs and $300 for the full project.
We use the default hyperparameter settings of PyMARL and HARL, e.g., the learning rates of the algorithms. The selection of the special hyperparameters introduced in this paper, e.g., the training steps for each teammate, is listed in Table 2.
C. Implementation Details of Baselines
We first compare SEMDIV with classic two-stage population- based training (PBT) methods, which train population of teammates using different techniques in the first stage, and use them to train agents in the second stage. FCP (Strouse et al., 2021) first trains a population of teammate policies using different random seeds independently. Then, it trains the agents by pairing them with three checkpoints of each teammate: the initial, middle, and final stages of training. In our implementation.
hhtpps ://s wttpb..cm//wwhirr////mmom 1pymar hhtpps: httpps:://tti //gi thwb..cc.o.. Boom/PKO-MARI/HA
13

---

LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
Table 2. Hyperparameters the experiments.
Hyperparameter
Value
Training steps for teammate
10 (LBF), 5 x 10 (PP), 10 (SMACv2), 102 (GRF)
Number of leammates trained with agents
Training steps for agents with one teammate Threshold for teammate performance verification Maximum attempls for generating lcammate policy Threshold e for teammate novelty verification in Equation (2)
3 x 10 (LBF), 10 (PP), 10 (SMACV2), 10 (GRF)
0.3 (LBF, PP), 0.5 (SMACv2, GRF)
0.2
Coefficient for regularizing feature extractors in Equation (4)
500
JOLESO
Turing
(a)
Figure 6. Environments used in this paper. (a) Level-based Foraging (I.FF) (Papoudakis et al., 2021). (b) Predator-Prey (PP) (Lowe et al.,
2017). (c) StarCraft Multi-Agent Challenge-v2 (SMACv2) (Ellis al., 2023). (d) Google Research Football (GRF) (Kurach al., 2020).
we set the population size 6, and train the teammates and agents until convergence. Based on FCP, MEP (Zhao et al., 2023) applies entropy term (- 8t)) when training the population, where 7 (a, se) = Ei=1 n² (uz 82). LIPO (Charakom al., 2023) replaces this term with Jur 0.001 across all environments. The rest implementation of MEP and LIPO remains the same as FCP. During execution, these methods directly deploy the only agent policy for coordination, without explicit adaptation process. While two-stage methods generate teammates before training agents, another baseline Macop (Yuan et al., 2023a) adopts an agents-centric paradigm, where it alternatively generates new teammates and trains the multi-head agents, inducing diversity by reducing the compatibility [(Tag: mm) between the teammates and the current agents. To select the policy heads for execution, Macop must collect multiple episodes to gather adequate information, hindering its deployment in costly tasks. In our experiments, we report Macop's results of the heads that maximize the RI or R2 values. Since SEMDIV combines the strengths of MARL and LLMs, we also include a baseline LLM-Agent that directly uses an LLM as the policy, to assess the necessity of MARL. The prompts for this LLM are provided in Appendix F.6.
Eizy [ Tim) We set the weights for these optimization terms as
D. Experiment Details
In this section, we provide more details about the experiments, including the environments and the unseen testing teammates.
D.I. Environments
We use four classic cooperative MARL environments with diverse coordination behaviors, as shown in Figure 6.
Level-based Foraging (LBF) (Papoudakis et al., 2021) is a discrete game where agents of varying levels navigate grid to collect foods with corresponding levels. Each agent moves one cell at a time one the four cardinal directions: (up, left, down, right}. Agents are rewarded with when they are positioned one cell away from a food item and the sum of their levels matches or exceeds the food's level. In this work,
6 6 grid-world setup with four level-2 foods
located at (0, 0). (0,5), 5), (5, 0), and (5, 5). Two level-1 agents are randomly spawned at cells {(2, 2), (2, 3), (3, 2), (3. 3)}. An episode terminates when agents collect one food or after nine steps. Coordination is essential as agents must observe their teammate's s preferences and collaborate to collect the foods.
Predator-Prey (PP) is a widely- benchmark from the Multiagent Particle Environment (MPE) (Lowe et al., 2017), where predators and prey are represented as circles on a plane. Agents controlling predators can accelerate one of four
14

---

LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
directions {up, left, down, right) to pursue prey, which employ a heuristic policy evade the nearest predator. We extend this benchmark to include five prey: two stags, which require both predators simultaneously capture them, and three rabbits, which can be captured by single predator. Capturing stag rewards the agents with 1, while capturing rabbit rewards 0.5. However, if only one predator attempts to capture a stag, the team is penalized with reward of -0.01. An episode terminates when predators catch one stag or two rabbits, or after twenty steps. Effective coordination is required for agents to adapt to their teammate's strategies and successfully hunt the prey.
StarCraft Multi-Agent Challenge-v2 (SMACv2) (Ellis al., 2023) an extended version of SMAC (Samvelyan et al., 2019). In this environment, ally units (agents) must defeat enemy units controlled by the game's built-in AL. Agents receive positive rewards for dealing damage, eliminating enemies, and winning battles, while incurring negative rewards for receiving damage, losing units, or being defeated. SMACv2 introduces randomized start positions, increasing the difficulty and variability of scenarios. In our experiments, two ally marine units face four enemy marine units. The predefined surroundes start position requires agents to move cohesively, focus fire on individual enemies, and adapt to their teammate's 's combat strategies to win battles. Agents are deemed successful if they eliminate at least one enemy.
Google Research Football (GRF) (Kurach et al., 2020) a physies-based 3D football simulator that closely replicates the rules and dynamies of real-world football. Agents can perform actions such as passing, defending, and shooting. We design scenario where two agents control players, Johnson and Turing, attempting to score from the edge of the penalty box. Johnson starts with the ball on the wing, while Turing positions centrally, facing the goalkeeper (Meitner). The team receives a reward of for scoring a goal and small rewards for getting closer to the goal. An episode terminates when goal is scored, the goalkeeper gains possession of the ball, the ball goes out of bounds, or after 100 steps. Effective collaboration between Johnson and Turing is required to win the game.
D.2. Testing Teammates
We evaluate the generalization capabilities of different methods by training five manually designed teammates with distinct and representative coordination behaviors for each environment. These teammates, along with their behavior descriptions, remain entirely unknown to the tested methods during training, ensuring an unbiased performance evaluation. In LBF, we train four teammates that specialize in collecting one specific food, with descriptions such as "I prefer to collect food A/B/C/D", ", and one teammate that "collects the food closest our average position".. ". In PP, we train five teammates with preferences for capturing specific prey, described as "I prefer to catch {prey)". 'hhse include teammates that prioritize stag 1, stag 2, rabbit 1&2, rabbit 1&3, and rabbit 2&3. In SMACv2, we design five teammates similar to those in LBF, where the "foods" are replaced by "enemies". In GRE, we train teammates with behaviors such as letting Johnson or Turing score, or scoring after passing 0, I, or times. Given the slight heterogeneity between the two players, we evaluate scenarios where the teammate controls either Turing or Johnson and report the average results. The pronoun in the description changes depending on the player controlled by the teammate. For example, when the teammate trained to let Turing score controls Turing, the description is / prefer to score myself." instead of prefer let Taring score.". This imposes a higher requirement on the methods' ability to understand teammate behaviors.
E. More Experiment Results
We present more experiment results, including the coordination performance with each unseen teammate, the impact of ambiguity in the natural language descriptions of teammates' behaviors, and the impact of the quality of the LLMs.
E. Results with Each Testing Teammate
In Section 4.2, we evaluate the agents from different methods in each environment by testing them with five unseen teammates and computing the average R1 and R2 values. To minimize randomness, we repeat the experiments across three random seeds and report the mean + standard deviation of the average R1 and R2 values, summarized in Table 1. Here, we present the performance with each individual testing teammate in Table 3-7. As shown in these tables, certain baselines exhibit unstable performance, coordinating effectively with some teammates but failing with others. For instance, FCP achieves RI and R2 values exceeding 0.7 with Teammates 1 and 5, yet falls below 0.5 with the remaining teammates. This suggests that the baselines struggle to capture specific coordination hehaviors, leading to agents that overfit to teammates with limited diversity. In contrast, SEMDIV consistently delivers the best overall results across all methods, demonstrating its robustness and ability to adapt effectively to diverse teammates.
15

---

LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
Table 3. Coordination performance (mean + std) with five unseen teammates in the LBF environment. "R1" and "R2" represent the success rates of task completion and agents satisfying the teammates preferred coordination behaviors, respectively. The best result in each column, excluding performance upper bounds of SEMDIV (denoted in gray), is highlighted in bold.
Teammate
I
Teammate 2
Teammate
Teammate 4
Teammate 5
Average
Methods
R1
R2
RI
R2
RI
R2
RI
R2
R1
R2
RI
R2
Oracle
1.00
1,00
1.00
1.00
1.00
1.00
1,00
1.00
1,00
1.00
1.00
1.00
SEMDIV
0.91 20.07
0.91
#0.07
0.90 -0.08
0.90 10.03
0.95 =0.04
0.95 20.04
0.87 -.00
0.87 =0.02
0.89 #0.16
0.89 #0.15
0.90
0.90
SUMDIV-45
0.29 20.41
0.29
40.41
0.07
20.00
0.07 10.00
0.52 =0:30
0.52 20.50
0.15 -0.36
0.15 #0.35
0.89 20.15
0.89
10.15
0.15
0.15
SEMDIV-RI
0.92 o7
0.92
10.07
0.93
06
0.93
0.95
20.04
0.95 10.01
0.57
0.87 10.02
0.94 10.08
0.93 10.00
0.91
0.91
SEMDIV-R2
0.92 20.07
0.92
#0.07
0.93
20.06
0.93 #0.06
0.95 =0.04
0.95 20.04
0.87 =0.02
0.87 +0.02
0. 0.94 a0.0a
0.93 +0.00
0.91
0.91
Macop-R1
0.97 =0.02
0.97
:0.02
0.81
20.00
0.81 #0.00
0.75 =0.31
0,75 $0.51
0.87 -0.12
0.87
::.12
0. 0.70 20.12
0.65 $0.15
0.82
0.81
Macop-R2 R2
0.97 =0.02
0.97
=0.02
0.81 =0.00
0.81 =0.00
0.75 =0.31
0.75 +0.31
0.87 -0.12
0.87
=:012
0.70 =0.12
0.65 #0.15
0.82
0.81
SEMDIV-PBT
0.69
=0 20
0.69 20.25
0.51
=0 13
0.51 20.13
0.33 =0
0.33 20.07
0.65 20:26
0.65 20025
1.00 20.00
1.00 =0.00
0.64
0.64
Macop-PBT
0.68 20.22
0.68 10.22
0.51 20.08
0.51 1:008
0.47 20.18
0.47 10.15
0.57 =0.22
0.57 1:.22
0.83 20.08
0.77 1:014
0.61
0.60
FCP
0.74
:0:00 28
0.74 10.28
0.46 20.00
0.46 1:0.3
0.37 20:36
0.37 10036
0.47 -0:31
0.47 10031
0.79 20.92
0.70 10.22
0.57
0.55
MEP
0.57
0.57
0.27
0.27 20.35
0.75
0.75 10:17
0.41
0.41 20:15
0.85 20.15
0.79 10:15
0.57
0.56
LIPO
0.55 20:31
0,55 20.31
0:13 20.54
0.13 20.54
0.19 20:27
0.49 #:027
0.17 20:33
0.17 10:35
0.75 20.14
0.63 10.14
0.51
0.51
LLM-Agent
0.83 20.10
0.83 #0.10
0.82 =0.02
0.82 #0.02
0.88 =0.06
0.88 #:006
0.92 =0.06
0.92 #0.08
0.95 a0.04
0.95 #0.04
0.88
0.88
Table Coordination performance (mean + std) with five unseen teammates in the PP environment. "R1" and "R2" represent the success rates of task completion and agents satisfying the teammates preferred coordination behaviors, respectively. The best result in each column, excluding performance upper bounds of SEMDIV (denoted in gray); is highlighted in bold.
Teammate
Teammate 2
Teammate 3
Teammate 4
Teammate 5
Average
Methods
R1
R2
RI
R2
RI
R2
RI
R2
R1
R2
RI
R2
Oracle
0.93
0.96
0.73
0.76
0.93
0.86
0.96
0.92
0.99
0,98
0.91
0.90
SEMDIV
0.71
0.74 20.12
0.47
0.41 20.30
0.85 20:31
0.49 20:29
0.81
0.54 20.01
0.77
0.51
0.72
0.54
SEMDIV-45
0.63
20 10
0.68 20.16
0.01 20.01
0.00 20.00
0.61 20.00
0.17 20.us
0.63 20.11
0.23 20.25
0.66
20:10
0.33 20.20
0.51
0.28
SEMDIV-RI
0.73 20.45
0.70 20.07
0.50 20:20
0.41 10.50
0.91 20:02
0.52 $0.26
0.85 20.04
0.36 20.18
0.52 20.c0
0.65 10.05
0.76
0.53
SEMDIV-R2
0.71 20:3 10
0.74
12
0.50
20 30
0.41 10:30
0.86 20:00
0.56 20.25
0.81 20.06
0.54
10000
0.80 20.04
0.67 10.05
0.74
0.58
Macop-R1
0.28 20.04
0.00
#0.00
0.32
=0.02
0.00 +0.00
0.75 =0.06
0.15 #0.05
0.82 =0.00
0.62 :0.05
0.78 #0.07
0.39 #0.00
0.58
0.23
Macop-R2
0.11 20.01
0,03 10.01
0.32 20.02
0,00 10.00
0.72 =0.00
0.21 10.08
0.82 =0.00
0.62 :0.08
0. 0.74 20.09
0.10 10.08
0.541
0.25
SEMDIv-PBT
0.43 oz
0.00
10.00
0.47 20
0.01
1:0002
0.94 00:00
0.88 20003
0.89
00:01
0.37 20.02
0.77 20:04
0.31 10.00
0.70
0.31
Macop-PBT PBT
0.45
0.00
0.46 =0
0.00 20.00
0.89
0.53
0.92
0.59 20.13
0.88 20.04
0.52
0.72
0.33
FCP
0.32 =0.22
0.00 =0.00
0.33 =0.20
0.00 #0.00
0.74 =0.21
0.30 #0.24
0.79
=0.21
0.52 20.30
0.69 e0.25
0.21 #0.15
0.57
0.21
MEP
0.45 20.00
0.00 10.00
0.47 20.00
0,00 10000
0.95 20.01
0.91 1:002
0.88
20.04
0.31 1.000
0.77 20.02
0.32 10.05
0.70
0.31
LIPO
0.43 20.00
0.00 20.00
0.46 g0.0t
0.00
20.00
0.96 20.02
0.92 20.04
0.87 20.00
0.25 20.05
0.76 ao.0t
0.41 20.02
0.69
0.31
A.eeMt
0.63 =0.15
0.85 #0.18
0.82 =0.00
0.87 ::012
0.58 =0.00
0.23 +0.00
0.73 =0.04
0.45 20.07
0.57 a0.13
0.25 #0.04
0.71
0.53
Table 5. Coordination performance (mean + std) with five unseen teammates in the SMACv2 environment. "R1" and "R2" represent the success rates of task completion and agents satisfying the teammates preferred coordination behaviors, respectively. The best result in each column, excluding performance upper bounds of SEMDIV (denoted in gray), is highlighted in bold.
Teammate
1
Teammate
Teammate
Teammate
Teammate 5
Average
Methods
RI
R2
R2
RI
RI
R2
R1
R2
RI
R2
Oraele
1.00
1.00
0.96
0.96
0.98
0.92
0.82
0.82
0.96
0.96
0.94
0.93
SEMDIV
0.88 =0.10
0.88 =0.10
0.47 =0.07
0.47 #0.07
0.66 2024
0.66 #0.24
0.58 =0:20
0.59 #0.20
0.65 =0.11
0.59
=0.13
0.65
0.64
SEMDIV-55
0.45 20.30
0.44
20.31
0.27
20 27
0.27 10.27
0.02
20:00
0.02 20.02
0.44
20:44
0.44
g0.41
0.00
20.00
0.00
10.00
0.24
0.23
SEMDIV-RI
0.88 20 no
0.88 10.10
0.63
20
0.63 20:33
0.69
20.07
0.61 20.11
0.65 20:11
0.65
20.11
0.66
20:24
0.66 10.04
0.70
0.69
SEMDIV-R2 R2
0.88 20.00
0.88 10.10
0.63
20.33
0.63 10:35
0.69
00.00
0.61 20.11
0.65 20.11
0.65
10.11
0.66
20.44
0.66
10.24
0.70
0.69
Macop-R1
0.81 20 14
0.81 20.14
0.61
0,58 10.16
0.70
0.63 10:11
0,22 20.06
0.22
10.06
0.04
20.02
0.01
10.02
0.48
0.45
Macop-R2
0.81 20 14
0.81 20.14
0.61 go
0.58 10.16
0.70 14
0.63 20.11
0.22 20.00 06
0.22 20006
0.03 a0.02
0.02 20.02
0.47
0.45
SEMDIV-PRT
0.74 20.22
0.74 20.22
0.35 =0.
0.35 20.23
0.75 =0.01
0.75 #0.01
0.80 =0.20
0.80
=0.30
0.42
a0.22
0.42 #0.22
0.61
0.61
Macop-PBT
0.71 =0.19
0.71 =0.19
0.67
=0.22
0.67 =n.22
0.60 =0.00
0.55 #0.06
0.65 =0.28
0.65 #0.25
0.17 a0.1a
0.13 +0.15
0.56
0.54
FCP
=0 19
0.81 20.19
0.15
00.00
20.05
0.63
20.17
0.63 20.17
0.21
20.15
0.21
20.15
0.19 a0.11
0.15 20.14
0.40
0.37
MEP
0.60 20.22
0.69 20.22
0.20
=0
0.20 20.12
0.81 =0.08
0.80 =0.05
0.51 =0.26
0.51 #0.25
0.51
20.14
0.16 #0.23
0.55
0.47
LIPO
0.75 o7
0.75 10.07
0.20 oz
0.20 20.07
0.53 20:00
0.43 10:15
0.37 20.95
0.37 20.25
0.39 20:17
0.15 10.90
0.45
0.38
LLM-Agent
0.30 20.0
0.30 10.04
0.83 00.44
0.83 10.14
0.37 20.17
0.37 20.17
0.13 20.05
0.13 20.05
0.10 20.14
0.10 10.14
0.35
0.35
16

---

LLM-Assisttd Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
Table Coordination performance (mean + std) with five unseen teammates in the GRF environment, where the unseen teammate coontrols player Turing. "R1" and "R2" represent the success rates of task completion and agents satisfying the teammates preferred coordination behaviors, respectively. The best result in each column, excluding performance upper bounds of SEMDIV (denoted in gray), is highlighted in bold.
Teammate 1
Teammate 2
Teammate
Teammate 4
Teammate 5
Average
Methods
RI
R2
RI
R2
RI
R2
R1
R2
R1
R2
RI
R2
Oracle
0.93
0.93
0.92
0.92
0.92
0.92
0.98
0.98
1.00
1.00
0.95
0.95
SEMDIV
0.85 10.17
0.85 20.17
0.77 10
0.77 10.15
0.62 20.44
0.62 :0.44
0.48 =0.08
0.48 20.08
0.38 20.38
0.35 :0.41
0.62
0.61
SEMDIV-45
0.53 =0.37
0.53
0.36
=0.38
0.06 #0.08
0.50
=0.41
0.29 #0.41
0.51
0.51 #0.05
0.08
=0.05
0.04
0.36
0.29
SEMDIV-RI
0.85 20
0.85 10.17
0.92
0.59 10.02
0.81
0.62 20:01
0.93 20.00
0.93 10006
0.90 20.00
0.31 10.43
0.88
0.66
SEMDIV-R2
0.85 17
0.85 20.17
0.88 20.00
0.88 40000
0.81 20:17
0.02 20.44
0.93 20.06
0.93 2.000
0.66 a0:37
0.35 20.43
0.83
0.73
Macop-RI
0.01 20.01
0.01 10.01
0.90 20.06
0.90
:0.00
0.10 00.07
0.10 10.07
0.96 00000
0.96
:0.00
0.21 20.04
0.15 10.11
0.41
0.42
Macop-R2
0.01 20:0
0.01
20001
0.90 20.06
0.90 1,0006
0.00 20:00
0.00 2:000
0.96 -0.00
0.96 100.0
0.19 20006
0.19 20.06
0.41
0.41
SEMDIV
0.02 20.00
0.02
20.03
0.49 =0
0.49 #0.25
0.01 =0:01
0.00 =:000
0.81 =0.10
0.81 #0.15
0.05 20.02
0.05 =0.02
0.28
0.27
Macop-PBT
0.00 =0.00
0.00 #0.00
0.32 =0.00
0.32 +0000
0.00 =0:00
0.00 #0.00
0.93 =0.06
0.93 #0.00
0.05 #0.01
0.01 +0.02
0.26
0.25
FCP
0.07 =0.07
0.07 #0.07
0.24 =0.01
0.24 #0.01
0.01 =0.01
0.00 #0.00
0,80
0.80 #0.04
0.14 #0.10
0.12 #0.12
0.25
0.25
MEP
0:20
0.20 20.24
0.24
=0 10
0.24 #0.19
0.05
0.00 2:000
0.61
0.61 #0.37
0.12
0.02
0.24
0.21
LIPO
0.02 20.00
0.01 #0.01
0.29 20.07
0.29 +0.07
0.00 =0.00
0.00 #0000
0.91 =0.02
0.91 40.02
0.12 a0.00
0.07 +0.00
0.27
0.26
LLM-Agent
0.00 20.00
0.00 #0.00
0.13 20.06
0.13 #0.05
0.03 00.06
0.00 $0.00
0.13 20.06
0.00 #0.00
0.00 20.00
0.00 $0.00
0.06
0.03
Table 7. Coordination performance (mean + std) with five unseen teammates in the GRF environment, where the unseen teammate coontrols player Johnson. "R1" and "R2" represent the success rates of task completion and agents satisfying the teammates preferred coordination behaviors, respectively. The hest result in each column, excluding performance upper bounds of SEMDIV (denoted in gray), is highlighted in bold.
Methods
Teammate 1
Teammate 2
Teammate
Teammate
Teammate 5
Average
R1
R2
R1
R2
RI
R2
R1
R2
R1
R2
RI
R2
Oracle
0.93
0.93
0.92
0.92
0.92
0.92
0.98
0.98
1,00
1.00
0.95
0.95
SEMDIV
0.78 20.00
0.29 #0.41
0.79
=0 12
0.79
#0.12
0.90
=0.00
0.90 =0.00
0.65 =0.10
0.65 #0.10
0.52 20.13
0.51 #0.15
0.73
0.63
SEMDIV-45
0.32
0.16
#0.29
0.61 =0.38
0.61 #0.38
0.87
=0.00
0.87 #0.08
0.53
0.53 #0.06
0.59
#0.25
0.10 #0.14
0.58
0.45
SUMDIV-RI
0.85 20:06
0.30
#0.41
0.90
30.00
0.90 40.05
0.93
00.01
0.93 $0.01
0.81 20.00
0.81 10.04
0.91 $0.00
0.00 $0.00
0.89
0.59
SEMDIV-R2
0.49 =0.30 02
0.49
#0.32
0.90
=0.00
0.90 20.03
0.93 =0.01
0.93 20.01
0.81 =0.00
0.81 #0.04
0.58 a0.04
0.58 #0.04
0.74
0.74
Macop-R1
0.62 =0.00
0.01
#0.01
0.72
=0.01
0.72 #0.01
0.85 =0.02
0.85 #0002
0.77 =0.02
0.77 #0.02
0.76 a0.01
0.01 #0.01
0.74
0.47
Macop-R2
0.46 2: 10
0.05
20.03
0.72
2. 01
0.72 a0.0
0.85
=0 oz
0.85 20.02
0.77 =0 .z
0.77 20.02
0.76 go.ot
0.04 20.04
0.71
0.49
SEMDIV-PBT
0.92 :0.01
0.00
20.00
0.87 20.06
0.87 10.06
0.85
20:01
0.85 20.01
0.83 =0.00
0.83 20.05
0.87 20.05
0.01 10.01
0.87
0.51
Macop-PBT
0.61 20.04
0.01
#0.01
0.68 =0.06
0.66 =0.00
0.81 =0.06
0.80 #0.07
0.74 =0.01
0.71 #0.05
0.74 #0.02
0.01 #0.01
0.72
0.44
FCP
0.72 10.13
0,00
10.00
0.55 20.26
0.55 :0.26
0.89 00.01
0.89 :0001
0.91 00.0
0.91 10.07
0.64 20.17
0.00 10.00
0.71
0.17
MEP
0.63
0.02
0.79 00.00
0.79 #0.03
0.90
-0.01
0.90
0.70
0.70
#0.15
0.78 #0.05
0.00 +0.00
0.76
0.48
LIPO
0.60 #0.32
0.00 20.00
0.72 10.12
0.72 #0.12
0.86 =0.02
0.86 #0.02
0. 50 =0.06
0.80
20.06
0.75 a0.07
0.01 +0.01
0.75
0.48
A.eeMt
0.07
0.07 20000
0.10
0.10 10.08
0.80
0.80 20.00
0.10 20.06
0.00
20.00
0.07
20.09
0.07
0.23
0.21
Min
Median
Maxi
Min
Median
Max
Thammats
Tsammate
Teammate
Teammare
5-2
Tramiite
Thamnate
Sammata
Tsamuue
(b)
(c)
Figure 7. Experiments on the impact of ambiguity in teammates' coordination behaviors and the quality of LLMs. (a)(b) The minimum, median, and maximum RI values of each teammate's 's 10 different behavior descriptions in LBF and SMACx2. (c) The performance of SEMDIV when using different LLMs in PP and GRF
17

---

LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
E.2. The Impact of Ambiguity in Teammates' Coordination Behaviors
In previous experiments, each testing leammate was described using single, unambiguous statement for clarity, this section, we investigate the impact of introducing ambiguity into these descriptions and evaluate the robustness of SEMDIV. Specifically, for each testing teammate, it to generate 9 alternative phrasings of the original description. This process yields total of 10 descriptions for each teammate. We then evaluate SEMDIV using all 10 descriptions for each teammate, calculating the minimum, median, and maximum RI values across these variations in the LBF and SMACv2 environments. As shown in Figure ((a)(b), the performance SEMDIV remains consistent despite the introduced ambiguity, demonstrating the robustness of its language-based reasoning process for head selection.
imput its original description and the task information into LLM, prompting
E.3. The Impact of the Quality LLMs
LLMs play a critical role the design of SEMDIV. To assess their impact, we replace the 4pt--4-2024-00 with gpt-40-mir. and conduct experiments the PP and GRF environments. As illustrated in Figure 7(c), the use of GPT-4o-mini results in modest performance decline, demonstrating that even smaller LLM can effectively support SEMDIV 's functionality. With the ongoing development of more advanced LLMs offering enhanced capabilities (OpenAI, 2024; DeepSeek-A,, 2025), SEMDIV holds the potential for further performance improvements.
model
F. Prompt Engineering
In this section, we provide the prompts for LLMs used in this paper.
F1. Task Information
We first provide the prompts about task information across all four environments, as they are frequently reused in prompts for different purposes in SEMDIV.
LBF:
You are expert in cooperative multi-agent reinforcement learning (MARL) and code generation. We going to train team of two players in the Level-Based Foraging (LBF) game. The game is 2D square grid-world with two agents, and four foods (denoted as food "A", "B", "C", and "D") are scattered in four different corners. Each player controls an agent. They need to choose a same food and move towards it, and be at adjacent grids of it together to collect the food. When agents successfully collect the first food, like food "B", they get reward and the game ends. Here's a part of the original code:
class ForagingEnn (Srv)
self. accats_positiion
anarravi (2, Ip.ndarrayI(G.11
selt. attfffodd__ootttee (taf: np. axrary ((D, 91), "a": rp ap.ayray 711, *ers ap. ap.arcey (?,, 01), rD": ap.array 7112
stz record the tooo ("A + been collected yet.
colttcool
*g" *g* =n) collected by trd tean, and neans no rood has
othor attributes functions
def soent_tood_disttnceeee (selt, sount_idk: sts, Ido: sto):
agent_ros selt. sgents_ppssition [agent_idx] food_pos wwif._tods_pasittteeee food_id]] distance cp. lig.iinag,... meraaie..ceeegee www.nccctnntaoooe post
return distance
(sclf))
cusles
reward
it process collectings: if sgents scccesstully collect cne food, zoward fooiLoul)
Jond, acenta ba adfacent orids together collect a_adt_players atae ramber cow, food_col) if aaad) self.collstadd_tod feod revard break
fosda_
f when soents successtully collect food, they get rovard 1 and the game ands. [me_l..rrrent_ate sellf._nxseep stae_steps) revard aulf.addil ionai_
zocuzn now state, zoward, done, and other step info
18

---

LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
PP:
You are an expert in cooperative multi-agent reinforcement learning (MARL) and code generation. We are going to train a team of two players in the Predator-Prey Prey (PP) game. The game a 2D world with two predators and five prey (two stags $1 S2 and three rabbits RI R2 R3). Each player controls a predator. They need to choose the prey to catch (like SI or R2+R3), then chase the chosen prey catch them. Stags require two predators to catch the same time. If only one predator is them, both players will be punished. Rabbits only require one predator catch them. When players successfully catch a stag, they get reward 1. When players successfully catch rabbit, they get reward 0.5.
Here's a part of the original code:
Game:
self sell-predatorg position ("lls up- rp-nnerrayl (,),, *2h rpindarreyl (2111) instialization boch op rendon, Forn
4 +0.1, 2)
uit.preyy (781f; meareyyl na2hs sp.ndarrayl G,lly "Rin; ap.ndarrayl (6)), 1R271 rp rp.ndazray tnibialization 191*s 0.1, (-ha 0.1, (9.8, 0,61, 1827t 1-0.8, 0,81, 10.,
up.nderrayl
-1..
selt selt..aught_ppeyy.
e) f record the pray daught the team, like (7517) or (1R27, "Ra"), and en erpty set meane no pray has been caught yet-
entity_diissanen discsnccceesiit catityl sto, entityz stx) float:
recurn the distance between the input entities, 2ike *I" and 12t, *2" ano 1817, "RI" and 1827, etc.
def oct_prey_____ levol iself, pray stx) +> iat:
for *51* get_nem prwdator nearby (selt, prey
"52*,
181* and "R2" and "83"
int:
seturn the number of predators aear / catching the pray (dictance 0.25), can do or or def (scltt)
that
poritions
roward
for prey in pelt.prre__postee keys
srey_level gell..ge__rry__lre level (prev)
aun_predator_nee self.
continue
elif 0 aur_predator__eerbyy prcy_levcl:
her_prodscaa__naaryy proy_lavel:
rewaed + pray_levei
cellt ace (prey]
revard
SMACv2:
You are expert in cooperative multi-agent reinforcement learning (MARL) and code generation. We are going to train a team of two players in the Starcraft Multi-Agent Challenge (SMAC) game, which involves unit micromanagement tasks. this game, ally units need to beat enemy units controlled by the built-in AI. Specifically, each player controlls a marine agent ("1" and "2") to beat four enemy marines ("A", "B", "C", and "D"). The two marine agents are spawned the center of the field, and four enemies are scattered four different corners. Agents need choose a same enemy,
towards it, and fire it together to kill it. When agents successfully kill the first enemy, like enemy "B", they get reward about 10 and the game ends. If both agents are killed, they lose.
Here's 's part of the original code:
aolata pecati_oosition ("2*; no. (2,)) *2": ap ap.ndarrayl
cnmmicc__oortiion ("A": ap..ndrrayl G2 "gft ndartay 12,1], "c": (darray((2,)),, *D": ap..:pp.nnddr.. nd..ndrrryt1,,1] self.oak
these
caloulated
cencerr))
initiai positions: agents near (9., 0.1. *2" lower left, "g" upper left, "c* upper right, "p" botton right Zor ageats and ensates that aze killed, their postions will be set to [0., 0.1 the enany ("h*
cele_killed_eemm
/ *ga kiiled
+ Allled yet.
othez attributes and funetions
agent_ids: idsny_idat agent_pos awents_peeittaeeeeeee_
enomy_pos self, anemics, idd]
distance linaly (agentt distance
def step (swl() revard = D.C
that obange
the atdributes,
original
revard addit Lonai_rwaadd other codes
19

---

LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
GRF:
You are an expert in cooperative multi-agent reinforcement learning (MARL) and code generation. We are going to train team two football players (Turing and Johnson) in the Google Research Football (GRF) game, They try to score from the edge of the box, Johnson on side with the ball, Turing the center and facing the goalkeeper (Meitner). Our team gets reward when scoring goal. An episode ends when our team scores a goal, Meitner owns the ball, or the ball is out of bounds.
Here's 8 a part of the original code:
#4 Locset ian Inroeeetionn
The cloaes Co the oppooent fa goal, the lazgez the x-ocordinatv. The y-Courciinate of the left half of the field dar y-coordinate of right hale
swlf.bel1 poollllosiiion ng.ndacray (allerray(.)))))
ball'a (ir y! coordinate, (0.7, 0.281
beginning Turing's 's is y) cosrdinater 10.7, 0.0) at the begimiin
self. Sol.-Turing.f buciig_poaiiinn
sp..ndascaylP..
Jolonnnn position
nndrraar (0,31 Johnson'e (8, coordinste, 10.7, -0.3) che begloning
6.01 the beginning se1f. M.l.Betteer_eee p Jowez and zight cozneza of the geal about (2.0, -0.04) (2.0, 0.04)
op.ndarray 02,31 Meitnert
mordinaate,
Goordinated
#4 2. Criticee hoel gene-leve? informat
swIF-paxe tes.s_historr List Lizt to vtoze the history of pasaea as tuples, with the fleat elament the player who made the exarple, [1"Noenson", "Turing"i, (*Turing", "Johnson") score bool True if the tean scores goel at this scep and Palse otherwize
pase and second exenent as
player
swlf-suore_ suoce____Tuutii ing : bool Trow 2f Turing scuces a yual at Liii atep and Palae utheruise
gelt.score. Johnneo bool True it Jonneon scores step(self))
at this and Faise othervise
athiez cudles that change above actributes
revard = 0.0
revard seldiadditoons. rexard
otheer codee
F.2. Behavior Generator
We provide the prompt for the L.M behavior generator.
(Task information)
Human player teams may have specific cooperation preferences to play the game. They have their own addi- donal-rewarr shown in the code. A new player outside team needs to learn and adapt these preferences to cooperate well after joining the team.
Here are some behavior examples:
Example 1: (A previous valid coordination behavior]
up with another possible cooperation preference. The preference should be deterministic and concrete. It should be as simple as possible. Avoid conditional terms like if, unless, when, etc. Avoid sequential behaviors like "first X, then Y". It should be easily implemented in python codes using the provided code snippet. It should not conflict with the original task objective.
Based the information above, think step by step
Finally, output the preference in the format: "Human players may prefer to (preference)".
F.3. Reward Generator
We provide the prompt for the L.M reward generator.
TTask information)
Now we want to train team with this specific cooperation behavior: {A natural language coordination behavior)
According to this cooperation preference, write an operational and executable reward function that formats as "def additional rewarddsee) - float" and returns the "reward float" only.
1. Please think step by step and tell us what this code means. The code function must align with the cooperation
20

---

LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
preference. 3. It can be dense reward that guides the team to learn the cooperation preference. 4. Short and simple code is better.
We have tried some reward function code before, but they are not good enough:
Attempt 1: {Information of the previous attempt)
Based on these information, You may consider change or rewrite the function.
F.4. Alignment Verification
We provide the prompt for the alignment examination between behaviors and policies.
{Takk information}
We tried to train team with this specific cooperation behavior: {A natural language coordination behavior)
After training the team with this reward function, we ran it for multiple episodes: (episode information]
Based on the information above, please review the running behavior of the team aligns with the desired behavior or not. Think step by step, and tell us your answer. Make sure your output contains a string "::1::" if your answer is "Yes" and contains a string "::0::" your answer is "No".
F5. Policy Selector
We provide the prompt for the head selection process before testing with an unseen teammate in GRF as an example.
You are an expert in football. We are going to build team of two football players (Turing and Johnson, no other teammates). They need score from the edge of the box. When the game starts, Johnson is on the left side controlling the ball, Turing is at the center and facing the goalkeeper.
Johnson was trained under the same situation, but with different teammates other than Turing to achieve the following cooperation preferences, and learned corresponding policies:
1: flearned behavior !).... 6: (learned behavior 6}
Now, Turing says that: "[testing teammates' coordination behavior)" Based the information above, please carefully analyze the game, the ball, the policies, etc. Think step by step to select the policy (I~6) for Johnson that can best coordinate with Turing and satisfy his preferences. Output your answer in the format "[n]". For example, if your answer is policy 3, output "[3]".
F66 The LLM-Agent Baseline
Here provide the prompts for the LLM-Agent baseline in all four environments.
LBF:
You are an expert in the Level-Bases Foraging (LBF) game. We are going to build team of two players and 2 in the LBF game. They need to collect food together. There are four foods (A, B, C, D) in the field. foods.position A [0. 0), [0, 7], C= [7, 0], D= [7, 7] Before the game starts, player 2 said that: "I prefer to collect food A." Now, player is at [4. 3.J. player 2 is at 4.J. Based the information above, select an action (0~4) for player that best coordinate with player? 2 and satisfy his preferences Output your answer in the format "[n]". For example, your answer is action 3, output "[3]".
Actions: (no-op), 1 (x coordinate minus one), 2 (x coordinate plus one), (y coordinate minus one), 4 (y coordinate plus one).
Output:
21

---

LLM-Assisted Semantically Diverse Teammate Generation for Efficient Multi-agent Coordination
PP:
You are an expert in the predator-prey (PP) game, We are going to build a team of two players and 2 controlling two predators in the PP game. They need to chase and catch the prey. There are five prey including two stags (SI, S2) and three rabbits (R1 R2, R3). Stags require two predators to catch at the same time. If only one predator is near them, both players will be punished. Rabbits only require one predator to catch them. Before the game starts, player 2 said that: "I prefer to catch stag S1." Now, player at (-1.00, 0.00), RI is at (0.80, 0.60), R2 is at (-0.80, 0.60), R3 is at (0.00, -1.00). Based on the information above, select an action (0~-4) for player that can best coordinate with player 2 and satisfy his preferences. Output your answer in the format "[n]". . For example, if your answer is action 3, output "[3]".
(0.06, 0.02), player is at (-0.09, -0.02), S1 (1.00, 0.00), S2 is
Actions: O (no-op). (accelerate towards +x direction), 2 (accelerate towards -x direction). 3 (accelerate towards +y direction), 4 (accelerate towards -y direction).
Output:
SMACv2:
You are an expert in the Starcraft Multi-Agent Challenge (SMAC) game. We are going to build team of two players and 2 controlling two marines in the SMAC game. They need beat enemy units controlled by the built-in AI. There are four enemy marines (A, B, C, and D) scattered in four different corners. Initial positions: agents near the map center [16., 16.], enemy A at the lower left corner, B at the upper left corner, C at the upper right corner, D at the bottom right corner. Agents need to choose a same enemy, move towards it, and fire at it together to kill it. When agents successfully kill the first enemy, like enemy B. they win. If both agents are killed, they lose. Before the game starts, player 2 said that: "I prefer to kill enemy C." Now, player 1 is at [16.95, 22.19), with health value 1.00 (1 is full health, is dead). Player 2 at [17.15, 21.50]. with distance 0.72 and health value 0.60. Enemy A is out of sight or dead, so cannot be observed and attacked. Enemy B is out sight or dead, so cannot be observed and attacked. Enemy C is out of sight or dead, so cannot be observed and attacked. Enemy be observed and attacked. Based the information above, please select action (1~9) for player 1 that best coordinate with player 2 and satisfy his preferences. Output your answer in the format "[n]". For example, if your answer is action 3, output "[3]".
out of sight or dead, so cannot
Actions:
(stop the current action), 2 (move north). (move south), 4 (move east), 5 (move west), (shoot enemy A), (shoot enemy B), 8 (shoot enemy C), 9 (shoot enemy D).
Available actions: [1, 4, 5, 8).
Output:
GRF:
You are expert in football. A team of two football players (Turing and Johnson) need score from the edge of the box. When the game starts, Johnson is on the side controlling the ball, Turing is the center and facing the goalkeeper Meitner. Before the game, Turing said that: "I prefer Johnson to score." Now, the ball is at [0.70 -0.28) with direction [0.00 -0.00J, Turing is at [0.70 0. Johnson is at [0.70 -0.29], Meitner is at [0.99 -0.02]. The center point of the goal is at [1.0, 0.0]. For [x, y] coordinates, -x direction on the left, +x direction is on the right, -y direction is on the top, +y direction is on the bottom. The pass history of our team []. Based on the information above, please select an action (1~18) for Johnson that can best coordinate with Turing and satisfy his preferences. Output your answer in the format "[n]". For example, your answer is action 3, output "[3]".
(run to the left). 2 (run to the top-left), 3 (run to the top), 4 (run the top-right), 5 (run to the right), 6 (run to the bottom-right), 7 (run to the bottom), 8 (run to the bottom-left), 9 (perform long pass). 10 (perform a high pass), 11 (perform a short pass), 12 (perform a shot). 13 (start sprinting). 14 (reset current movement direction), 15 (stop sprinting), 16 (perform a slide), 17 (start dribbling), 18 (stop dribbling).
Actions:
Output:
22


Reviews:
[{"review_id": "8lWcHKHWMl", "reviewer": "Reviewer_EFqS", "summary": "The paper proposes a novel partner generation method, which iteratively generates new partner through generated reward functions via LLM queries. The input to the LLM contains semantic information of the behavior, allowing the method to include semantic information for generating the partners in addition to policy-level validity and diversity check. The proposed method also leverages the LLM for selecting the right best response policy given a behavior description of the test partner. The method outperforms various baselines across diverse environments.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the valuable comments! Extra experimental results can be found in this [link](https://telling-floor-898.notion.site/1c7c2fed721a80b9ba7ef7fa2b3bffed).\n\n**Q1: The motivation for unseen teammate communicating behaviors before testing.**\n\nA: Communication and intention sharing are the basic and core techniques in multi-agent systems [1,2], mitigating partial observability. Following these techniques, our approach leverages one-time natural language communication to boost coordination efficiency. For situations where communication is entirely infeasible, SemDiv can still infer teammate behaviors from interactions using techniques in [3,4]. We note this as a future direction to extend our framework to zero-comm environments.\n\n**Q2: Compare more baselines to support the claims in Section 4.4.**\n\nA: As shown in Table 1 in the link, even with extra policy-level diversity techniques and larger population sizes, policy-level baselines still achieve limited performance, further indicating the necessity and efficiency of semantic-level diversity.\n\n**Q3: Compare baselines with Policy Verification.**\n\nA: Even with Policy Verification, population-based baselines show limited improvement (Table 2). The Policy Verification process: (1) Train 12 teammates, verifying all can complete the task. (2) Select the 6 most diverse policies:\n\n- MEP: Farthest from population mean.\n- LIPO: Iteratively pick worst-performing partners.\n\nNo behavior verification is needed as baselines lack semantic info. Results confirm LLM-assisted diversity drives SemDiv's success, not just Policy Verification.\n\n**Q4: Pass rate of Policy Verification in SemDiv.**\n\nA: We report the results of Policy Verification (Table 3), including (1) the number of occurrences of different issues averaged over 3 random seeds and (2) total pass rate. SemDiv efficiently generates diverse teammates while verifying their quality. We will add these results in the paper.\n\n**Q5: Similarity information and agent behavior text representation.**\n\nA: We simply represent info_sim as: “Not-novel example: {behavior}. This behavior is the same as {similar_behavior}“. For behavior texts, we write light-weight functions to extract information from trajectory data and fill in text templates, aligning with previous works [5,6]. For example, in SMACv2, info = “Agents killed enemy {enemy_id} …”. We will add more details of text representation in the paper.\n\n**Q6: Whether SemDiv’s improvement is stemmed from the instability of baselines.**\n\nA: SemDiv’s improvement is stemmed from better semantic-level diversity rather than exploting the instability of baselines. We report the performance of all generated teammates in Table 4. The teammate stability and quality of baselines is similar to that of SemDiv.\n\n**Q7: Substantiate previous methods‘ inefficient exploration of the teammate policy space.**\n\nEfficiently exploring diverse high-quality teammate policies is a fundamental challenge in MARL for open environments [7]. Early methods primarily imposed regularization at the action level, requiring exhaustive traversal of the joint policy space, which becomes computationally prohibitive in complex scenarios. Recent works like [4,8] have demonstrated the limitations of classic policy-level methods like FCP and LIPO. In contrast, our approach abstracts the joint policy space into a higher-level semantic space, where a single semantic behavior can correspond to a wide range of policies. This abstraction explicitly reduces exploration difficulty by decoupling diversity from low-level action redundancy. Empirical results validate that SemDiv achieves more efficient and scalable policy exploration compared to prior works, representing a significant technical advancement in this direction.\n\n**Q8: Mention related work in Eq. 2.**\n\nA: We apologize for not properly citing the work [9] when introducing Eq. 2 and explaining its relevance for policy similarity verification. We will correct this by adding the citation at Eq. 2 and providing a clearer discussion.\n\nReferences\n\n[1] Albrecht et al. Multi-Agent Reinforcement Learning: Foundations and Modern Approaches. MIT Press, 2024.\n\n[2] Zhu et al. A survey of multi-agent deep reinforcement learning with communication. Auton. Agents Multi Agent Syst. 38(1): 4 (2024).\n\n[3] Yuan et al. Multi-agent Continual Coordination via Progressive Task Contextualization. TNNLS 2024.\n\n[4] Yuan et al. Learning to Coordinate with Anyone. DAI 2023.\n\n[5] Xie et al. Text2Reward: Reward Shaping with Language Models for Reinforcement Learning. ICLR 2024.\n\n[6] Ma et al. Eureka: Human-Level Reward Design via Coding Large Language Models. ICLR 2024.\n\n[7] Yuan et al. A Survey of Progress on Cooperative Multi-agent Reinforcement Learning in Open Environment. 2023.\n\n[8] Rahman et al. Minimum Coverage Sets for Training Robust Ad Hoc Teamwork Agents. AAAI 2024.\n\n[9] Charakorn et al. Generating diverse cooperative agents by learning incompatible policies. ICLR 2023.", "claims_and_evidence": "> This demonstrates that generating semantically diverse teammates not only enables more efficient exploration of the teammate policy space but also facilitates the discovery of coordination behaviors that policy-level exploration alone cannot cover.\n\nThe paper claims that \"policy-level exploration\" cannot discover certain behaviors. But in the experiment, it only uses FCP---which does not even explicitly generate diverse partners---as the only reference for this claim. It is not fair to claim this without testing more sophisticated \"policy-level exploration\" methods.", "methods_and_evaluation_criteria": "The method and evaluation are reasonable. Though the paper would be more convincing with an ablation study on the Policy Verification is included (see Comments or Suggestions)", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experiments and analyses are reasonable with statistical confidence.", "relation_to_broader_scientific_literature": "The paper could be impactful as it introduces the use of LLMs for generating diverse partners. Prior work focuses on generating diverse partners at the policy-level. This approach incorporates semantic information to generate more diverse partners.", "essential_references_not_discussed": "The paper should mention the work that Eq. 2 is based on ([1]) and why using Eq. 2 could possibly check for policy similarity in the first place. In its current form, the paper simply mentions that it uses Eq. 2  to verify diversity without giving any intuition. The paper already has the citation in various places but not at Eq. 2.\n\n[1] Charakorn et al. \"Generating diverse cooperative agents by learning incompatible policies.\" The Eleventh International Conference on Learning Representations. 2023.", "other_strengths_and_weaknesses": "Strengths\n- The idea of using LLM for partner generation is novel.\n- The evaluation includes many SOTA baselines.\n\nWeaknesses\n- The claim in Section 4.4 is not fair (see Claims and Evidence).\n- It is not clear whether the use of the LLM is the main contributor to the improvement over the baseline (see Suggestions).\n- Some information about the pipeline is not provided fully, e.g., how `info_sim` or agent behaviors are represented for the LLM.", "other_comments_or_suggestions": "### Comments\n> First, the exploration of the teammate policy space is inefficient, as teammates are driven to optimize for differences at the policy-level rather than actively discovering novel coordination behaviors at the semantic-level.\n\nTo strengthen this statement, the paper should substantiate why \"the exploration of the teammate policy space is inefficient\". Is there any existing paper that confirm this statement?\n\n> As we aim to study teammates generation and agents coordination at the semantic-level, we consider scenarios in which the group of teammates $\\pi^\\text{tm}$ provides a natural language description $b$ prior to the execution phase.\n\nThe reason given for the use of prior communication is not sound. One could also communicate without semantic-level coordination. Coordinating at the semantic-level could also omit communication.\n\n### Suggestions\n- The baselines do not use the \"Policy Verification\". Since it is very likely that with high regularization, XP-min methods tend to generate incapable policies, just like non-functional reward functions generated by the LLM. The paper would benefit by a baseline that uses the Policy Verification as well. This would confirm that the semantic information is indeed useful for partner generation, not the Policy Verification part. In its current form, there could be an alternate explanation that the Policy Verification does the heavy lifting of the improvement.\n\n- Related to Policy Verification, I wonder how much percentage of the generated reward functions are invalid? How much percentage of them leads to executable reward functions but does not pass the Policy Verification? What is the average number of repetitions before generating a valid policy?\n\nI'm willing to increase the score if all these concerns are addressed properly.", "questions_for_authors": "- How is the `info_sim` represented?\n\n- How are agent behaviors represented in text before feeding into the LLM (for the alignment step)? I suppose different environments would have different behavior representations/descriptions.\n\n- Is the better performance stemmed from the fact that XP-min based methods are unstable/inefficient? My understanding is that the proposed method still generates incompatible policies but through generated reward functions (with the \"diversity check\" which checks the compatibility of newly generated policy) instead directly optimizing the \"diversity check\", which is based on the XP-min methods.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JmWOuMMQGF", "reviewer": "Reviewer_XbFR", "summary": "This paper proposes a teammate generation method called SemDiv, which uses LLMs to learn diverse coordination behaviors a the “semantic” level. SemDiv generates novel teammates by iterating the following steps: (1) generating natural language description of a novel coordination behavior, (2) translating it into a shaping reward function for training a teammate policy, (3) verifying whether training using such a reward function generates a meaningfully different policy, (4) if verified, train a new policy head to coordinate with the newly generated teammate, using a continual learning objective to avoid forgetting how to coordinate with previously seen teammates. At test time with an unknown teammate and a behavior description of that teammate, SemDiv uses an LLM to select the best response policy head. \n\nThe method is evaluated on 4 MARL environments: LBF, Predator-Prey, SMAC-v2, and Google Research Football (GRF), with 5 unseen teammates per task. Emprical results show that SemDiv outperforms various teammate generation baselines and LLM baselines in terms of generalization.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the valuable comments!\n\n**Q1: Difference between “policy level” and “semantic level”, and why optimizing policy level differences is bad.**\n\nA: Policy-level methods mainly optimize policy-space diversity without explicitly modeling the corresponding semantics. In this approach, the search space grows exponentially with the number of agents, leading to inefficient exploration. In contrast, our semantic-level method explicitly models human-interpretable coordination behaviors through language-guided generation. By clustering similar policies into semantically meaningful behaviors, our approach significantly reduces the complexity of the search space.\n\nPrior works [1,2] have provided analysis and conducted experiments to demonstrate that traditional policy-level methods (e.g., FCP and LIPO) struggle to efficiently explore teammate policy spaces. We further validate this claim through comprehensive performance evaluations and trajectory analysis, highlighting the advantages of semantic-level optimization over policy-level approaches.\n\n**Q2: More results on t-SNE visualization, cross-play matrices, etc.**\n\nA: We report these important results in this [link](https://telling-floor-898.notion.site/1c7c2fed721a80b9ba7ef7fa2b3bffed), and will add them in the paper.\n\n**Q3: Requirement of natural language behavior descriptions from unknown teammates.**\n\nA: Please refer to Q1 for **reviewer EFqS**.\n\n**Q4: The performance of Macop.**\n\nA: We would like to clarify that we have actually included the performance of Macop in Table 1, though presented as its *upper bound* versions that directly report the results of the best policy heads. However, there is still a gap between Macop-{R1, R2} and SemDiv, proving the impact of SemDiv’s semantically diverse teammates.\n\n**Q5: [Clarification Question] How SemDiv selects policy heads and the SemDiv-Dist baseline.**\n\nA: During training, all methods are unaware of testing teammates' behavior descriptions. SemDiv agents only select heads for the corresponding training teammates for policy verification. During testing, SemDiv agents select heads according to the testing teammates’ behavior descriptions. The SemDiv-Dist baseline also requires these descriptions during testing to select heads. Instead of using LLMs for reasoning, it selects the head that minimizes Distance(embed($b$), embed($b_{test}$)), where $b$ is the behavior learned by the head, $b_{test}$ is the behavior of the testing teammate. To improve SemDiv-Dist, we can finetune the language model that computes the embeddings with task-specific data, enhancing its understanding of the task.\n\n**Q6: The reward shaping problem.**\n\nA: In our work, we empirically set a fixed step limit for inner-loop training, a practical trade-off that may limit the \"deepness\" of learnable behaviors. While this suffices for simpler tasks, we fully acknowledge the challenge for more complex behaviors. Future work will adopt Neural Architecture Search (NAS) [3] to automate hyperparameter setting, enabling efficient training of deeper behaviors under shaped rewards. We will clarify this limitation and solution in the paper.\n\nWe apologize for these unclear expressions, and will improve them in the paper.\n\n- Number of testing teammates and controlled agents: We have modified the Abstract and Introduction to explicitly state: “Evaluation with five unseen representative teammates per environment”, and we will add an explanation in the Problem Formulation that we focus on scenarios where N=2.\n- Related work of Section 3.3: Please refer to the reply for **reviewer Q2F9**.\n- More detailed figure captions:\n    - Figure 1: An overview of the training and testing process of SemDiv. Left: During training, SemDiv proposes novel coordination behaviors in natural language and transform them into teammate policies for agent learning. Right: During testing, SemDiv takes as input the description of the unseen teammates and selects the optimal learned policy for coordination.\n    - Figure 2: The overall workflow of SemDiv. (a) Generating coordination behavior. SemDiv iteratively generates of semantically diverse coordination behaviors, enabling efficient exploration of the teammate policy space. (b) Training aligned teammate policy. For each coordination behavior described in natural language, a teammate policy is trained to align with that behavior. (c) Training agents. Agents are continually trained with these teammates, developing strong coordination ability.\n- Upper bounds: {SemDiv, Macop}-R1, R2 are only upper bounds of {SemDiv, Macop}, which directly report the results of the best policy heads to investigate the impact of the head selection module. We again apologize for these unclear expressions.\n\nReferences\n\n[1] Yuan et al. Learning to Coordinate with Anyone. DAI 2023.\n\n[2] Rahman et al. Minimum Coverage Sets for Training Robust Ad Hoc Teamwork Agents. AAAI 2024.\n\n[3] Elsken et al. Neural Architecture Search: A Survey. JMLR 2019.", "claims_and_evidence": "Most claims are somewhat supported by evidence. There are some claims that are misleading or insufficiently supported, which I have pointed out below.\n\n- “First, the exploration of the teammate policy space is inefficient, as teammates are driven to optimize for differences at the policy-level rather than actively discovering novel coordination behaviors at the semantic level.” ([pdf](zotero://open-pdf/library/items/B38DLBJN?page=1))\n    \n    - The authors do not provide any theoretical justification or empirical demonstration of why optimizing policy level differences is bad. It’s not clear what the difference between “policy level” vs “semantic level” is.\n        \n- A key claim of this paper is that LLMs are better able to generate diverse teammates than non-LLM teammate generation baselines\n    \n    - I don’t think the paper presents enough evidence for this claim.\n        \n    - The case study figure where the strategies generated by SemDiv and FCP is good (Fig 3c), but not totally convincing, because (1) FCP is one of the weakest teammate generation baselines, and (2) T-SNE can produce different results when run with different seeds. The evidence would be much stronger if the authors do two things: first, re-generate Fig. 3c for the next-strongest non-SemDiv baseline for all tasks, and second, provide cross-play matrices for the populations generated by all teammate generation methods.\n        \n- Another key claim of this paper is that the proposed LLM-based teammate generation process generalizes better to unseen teammates, compared to non-LLM based methods.\n    \n    - Evidence provided in form of performance comparisons against basedline non-LLM methods (FCP, MEP, LIPO, Macop) and an LLM-agent only baseline, where manually designed unseen teammates are used to test SemDiv and baseline\n        \n    - However, given the external calls to LLMs and potentially computationally expensive verification sub-routines within SemDiv (e.g. testing if the reward function is valid), the authors should also report the wall-clock time required to train each method to obtain the results in Table 1.\n        \n- Misleading claim - the abstract states that SemDiv is evaluated on 20 unseen teammates and 4 tasks, making it unclear whether it was evaluated using 20 teammates per task, or divided evenly among the tasks (as turns out to be the case). The authors should reword that claim in the abstract (and anywhere else it appears in the paper) to avoid implying that their method was evaluated with more teammates than it actualy was.\n    \n- Misleading characterization the paper is couched in terms of N-agent teams, and in various places refers to the number of controlled agents as {1, …, n_ag} and the number of uncontrolled as {1, …, n_tm}. This implies that N>2. While I’m fine with describing the method in greater generality, the paper should clearly state somewhere that N=2 in all evaluation settings.\n    \n- Misleading characterization - SemDiv’s continual ego agent learning procedure and multi-head architecture directly comes from Macop (Yuan et al. 2023). This is fine, as this isn’t really the main contribution of the paper, but Section 3.3 (which describes the ego agent learning procedure) should clearly acknowledge this. and make it clear what is different in this work.", "methods_and_evaluation_criteria": "Overall, the proposed method makes sense, and the experiments evaluate the key claims of the method (improved generalization + improved teammate diversity). I have some concerns with the evidence provided for the claims, which I have described previously. Some additional concerns about the method are provided below.\n\n- The LLM is used to generate/verify several key points in the teammate generation procedure. i would like to see evaluations of each component in terms of metrics such as the number of attempts, success rate, and wall-clock time\n    \n    - Generating reward function programs\n        \n    - Verification of alignment between behaviors/policies\n        \n    - Policy selection process for testing with unknown teammate.\n        \n- Statistical significance measures: results are presented with mean +/- std deviations throughout the paper, which reflects the variance in the mean performance for each method. To provide information about the statistical significance of the results, the authors should provide statistical significance tests comparing the performance of SemDiv to the next best performing method, or computing 95% CI’s instead.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "See above.", "relation_to_broader_scientific_literature": "This method is a teammate generation method for ad hoc teamwork/zero-shot coordination. The authors propose using LLMs to generate novel teammate behaviors, which to my knowledge, has not been considered/addressed before for ad hoc teamwork.", "essential_references_not_discussed": "No", "other_strengths_and_weaknesses": "- Strengths:\n    \n    - Overall, the paper is well-written and well-situated in the field of ad hoc teamwork\n        \n    - The idea is novel and timely, and the overall framework is quite compelling.\n        \n    - The results convincingly show that SemDiv outperforms other common teammate generation baselines.\n        \n- Weaknesses\n    \n    - SemDiv requires natural language behavior descriptions from unknown teammates to allow an LLM to select the optimal head.\n        \n        - In my view, this is the largest weakness of the paper. SemDIV is presented as an end-to-end teammate generation + AHT method, but this is a pretty large limitation of the AHT component of the method. The policy selection problem makes up a large portion of the challenge for AHT methods (e.g. PLASTIC by Barret et al. 2017). Assuming access to a natural language description of the unknown teammate essentially assumes an oracle for the teammate modeling problem (i.e. generating a characterization of an unseen teammate, and relating it to the representation of known teammates).\n            \n        - However, I believe that the main contribution of the method is showing how to generate teammates using LLMs/natural language descriptions, which is sufficient on its own, even without any novelties in the ego agent learning process. The authors should explicitly acknowledge this limitation.\n            \n    - Multiple misleading claims are made.\n        \n\nBarrett, Samuel, Avi Rosenfeld, Sarit Kraus, and Peter Stone. 2017. “Making Friends on the Fly: Cooperating with New Teammates.” *Artificial Intelligence* 242 (January):132–71. [https://doi.org/10.1016/j.artint.2016.10.005](https://doi.org/10.1016/j.artint.2016.10.005).", "other_comments_or_suggestions": "- Details that need to be made clearer:\n    \n    - What percentage of the time does SemDiv generate valid vs invalid teammates?\n        \n    - Reading the methods section was confusing, because I thought that the paper addressed the scenario of N>=2. The authors should specify that the paper addresses N=2.\n        \n- Each figure caption should explain the main message that the figure is meant to show, to support readers skimming the paper based on figures alone\n    \n- There are a couple issues with Table 1:\n    \n    - Since {SemDiv, Macop}-R1, R2 are upper bounds, all four methods should be included at the top, perhaps in the oracle section, or perhaps in their own section. It’s confusing to have it presented together with SemDiv.\n        \n    - Shouldn’t Macop-R1, R2 be grayed out too, since it is also an upper bound and presumably excluded from consideration for computing the best result?", "questions_for_authors": "1. In Table 1, the difference between SemDiv and SemDiv-PBT is much larger than the gap between SemDiv-PBT and non-LLM teammate generation methods (e.g. Macop-PBT, FCP, MEP, LIPO), which makes me wonder if SemDiv is performing better than the non-LLM teammate generation methods because of the continual learning paradigm introduced by MACOP, rather than because of the generated teammates. Can you present the performance of MACOP (the original, non-PBT version) as well, in Table 1?\n    \n2. [Clarification Question] After reading the paper, my impression is that SemDiv requires behavior descriptions from unknown teammates. However, the experiments section (L242-245) states that the teammates’ behavior descriptions remain unknown to the tested methods during training. Does SemDiv select a policy head without the behavior description? If so, then what is the purpose of the Semdiv-Dist baseline?\n    \n3. Given that the SemDiv does require behavior descriptions:\n    \n    1. Can the authors provide a discussion of how reasonable this assumption is?\n        \n    2. Semdiv-Dist performs much worse than Semdiv, suggesting that the behavior descriptions do a lot of the ‘heavy lifting’ for enabling generalization to unseen teammates. Do the authors have any thoughts on how to improve this?\n        \n4. Reward shaping: the criteria outlined for whether the reward function is valid seems to require training a policy under the reward function  as an inner-loop step (Line 173 in the submitted PDF). Can the authors comment on how many timesteps are allocated to train the teammate under hte reward function? Some reward functions can only be optimized in a large number of steps -- how does this affect the “deepness” of the generated behaviors? Is it the case that only “easy/simple” behaviors can be generated?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bq00yA5wUN", "reviewer": "Reviewer_rgCD", "summary": "The paper introduces SEMDIV, a novel framework that uses LLMs to generate semantically diverse teammates for efficient multi-agent coordination. Unlike traditional methods that focus on policy-level diversity, SEMDIV iteratively generates natural language descriptions of coordination behaviors, translates them into reward functions, and trains teammates to embody these behaviors. This approach allows agents to learn and adapt through continual learning with a multi-head architecture, selecting the most suitable policy through language-based reasoning. Experiments across four environments show SEMDIV’s superior performance in coordinating with unseen teammates.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the valuable comments! Extra experimental results can be found in this [link](https://telling-floor-898.notion.site/1c7c2fed721a80b9ba7ef7fa2b3bffed).\n\n**Q1: Experiments with more agents and with continuous action space.**\n\nA: SemDiv is agnostic to team sizes and action spaces, still achieving the best performance compared with baselines, indicating its scalability and potential to solve more complex tasks. The results are in Table 8 in the link.\n\nFirst, we extend the original LBF task to a 3-agent setting, where it takes 3 agents to collect a food at the same step. During testing, one unseen agent joins the team. Training details (like MARL methods, prompts for LLMs, etc), and behaviors of testing teammates, are similar with the 2-agent experiments.\n\nNext, we include the Spread task based on the HARL codebase [1], in which 3 agents with continuous action space need to approach three different landmarks. For training, we select MAPPO similar with GRF. For LLM prompts, we slightly modify the ones used in PP since both of them are MPE tasks. During testing, 6 unseen teammates trained in teams with different approaching strategies (different agent-landmark pairs) are joined.\n\n**Q2: Testing with lazy agents.**\n\nA: We build lazy agents that take the noop action or move back and forth in the initial position, in all four environments used in the paper. We set their descriptions as “I am a lazy agent and I will do nothing” when testing SemDiv. The results are shown in Table 9.\n\n- LBF & SMACv2: All methods fail as ≥2 agents are needed for food/enemy tasks.\n- PP: Baselines match SemDiv by ignoring lazy teammates (reward for catching a rabbit = 0.5).\n- GRF: SemDiv excels by learning attack strategies without passing and selecting these policy heads.\n\n**Q3: Discuss several assumptions and limitations.**\n\nA: (1) Environment APIs, and language descriptions of episodes. Both assumptions are important considerations, which are well-established in prior LLM-assisted RL research [2,3,4]. However, we acknowledge that relaxing these assumptions is valuable future work. (2) Language descriptions of teammates' behavior during execution. Please refer to Q1 for **reviewer EFqS**. (3) Dependence on LLMs. Like prior work, our method relies on LLMs, which may hallucinate. Potential solutions include using more advanced models or human verification. (4) Generalization scope. Our evaluation focuses on close games with clear rewards. Extending to real-world tasks (e.g., embodied AI) introduces challenges like real-time perception and physical constraints. We will discuss these limitations explicitly in the paper.\n\n**Q4: How does SemDiv handle contradictory behaviors across teammates?**\n\nA: SemDiv uses a multi-head architecture (Sec. 3.3) to mitigate catastrophic forgetting in multi-agent settings [5,6]. Testing confirms the final agents successfully coordinate with all 6 learned teammates, including those with contradictory behaviors (Table 10).\n\nWe apologize for these unclear expressions, and will improve them in the paper.\n\n- Complementary policy: we aim to train a teammate policy $\\pi^{tm}$ (e.g., player A in a two-player game) to form a team and train with agent $\\pi^{ag}$ (e.g., the other player B). To achieve this, $\\pi^{tm}$ must first learn coordination by training alongside a complementary policy $\\tilde{\\pi}^{tm}$ (which also controls Player B) using MARL algorithms. This training phase with $\\tilde{\\pi}^{tm}$ ensures that $\\pi^{tm}$ develops specific coordination behaviors before interacting with $\\pi^{ag}$.\n- $\\lambda$ configurations in Eq. 2: The equation ensures the new teammate $\\pi^{tm}_m$ differs from previous ones $\\pi^{tm}_j$. When $\\lambda_1 = 1, \\lambda_2 = 0$, it verifies that agents trained with $\\pi^{tm}_j$ cannot achieve comparable original task rewards. When $\\lambda_1 = 0, \\lambda_2 = 1$, it checks the same for shaped rewards. Together, these configurations strictly guarantee the new teammate's distinctiveness.\n- Explanations to the figures and SemDiv-R1/R2: Please refer to the reply for **reviewer XbFR**.\n- Second paragraph of Section 4.1: The *five* teammates mentioned here are the ones for testing different methods, rather than the *six* teammates generated during training. We again apologize for the unclear expressions.\n\nReferences\n\n[1] Liu et al. Maximum entropy heterogeneous-agent reinforcement learning. ICLR 2024.\n\n[2] Xie et al. Text2Reward: Reward Shaping with Language Models for Reinforcement Learning. ICLR 2024.\n\n[3] Ma et al. Eureka: Human-Level Reward Design via Coding Large Language Models. ICLR 2024.\n\n[4] Ma et al. DrEureka: Language Model Guided Sim-To-Real Transfer. RSS 2024.\n\n[5] Yuan et al. Multi-agent Continual Coordination via Progressive Task Contextualization. TNNLS 2024.\n\n[6] Yuan et al. Learning to Coordinate with Anyone. DAI 2023.", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "Yes, this work focuses on the multi-agent domain with diverse teammates, and the proposed method and evaluation criteria are well-suited to this context.", "theoretical_claims": "There is not theoretical claim in this submission.", "experimental_designs_or_analyses": "This work conducts experiments in four multi-agent scenarios. However, several issues remain:\n- The number of agents in these scenarios is very limited — only two — which raises concerns about the scalability of the proposed method. It would be more comprehensive to evaluate additional tasks in SMACv2 and GRF with a greater number of agents.\n- All selected scenarios involve a discrete action space; it would be beneficial to assess performance in tasks with a continuous action space, such as MAMuJoCo.\n- The diversity of testing teammates appears to be limited. Including a wider variety of teammates, such as lazy agents, would provide a more thorough evaluation.", "relation_to_broader_scientific_literature": "The work integrates LLMs into the multi-agent RL, extending research on ad-hoc teamwork and zero-shot coordination. It thoughtfully builds on existing diversity-inducing methods while addressing their limitations on semantic information.", "essential_references_not_discussed": "no", "other_strengths_and_weaknesses": "Strengths\n--\n\n1. The paper is well-structured and the idea is well-motivated.\n2. The implementation details and prompt designs are clearly and thoroughly described.\n3. The experimental results are compelling and thoroughly explained.\n\nWeaknesses\n--\n\n1. The proposed method heavily relies on several assumptions about the environments and testing setups, including (i) accessible attributes and APIs of the environments, (ii) language descriptions of episodes, and (iii) language descriptions of teammates' behavior during execution, which may not hold in other scenarios.\n2. I find Section 3.2 a little bit confusing: (i) What does $\\tilde{\\pi}_m^{tm}$ refer to during teammate policy training? (ii) Why are both $\\lambda$ configurations necessary in Equation (2)?\n3. There are concerns regarding the experimental design, as noted above.\n4. The discussion of limitations is insufficient.", "other_comments_or_suggestions": "1. I am curious about the scaling effect of SEMDIV with a larger number of teammates.\n2. I am unclear about the meaning of SEMDIV-R1/R2. Does it refer to evaluating each heading and selecting the best one?\n3. minor: It would be helpful to add detailed explanations to the figures, such as Figures 1 and 2.\n4. Typos: Second paragraph of Section 4.1 — \"In each environment, we train **five** teammates exhibiting distinct and representative coordination behaviors.\" Should this be six teammates?", "questions_for_authors": "How does SEMDIV handle contradictory behaviors across teammates? Could conflicting policies cause catastrophic forgetting despite the regularization term?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FUPGv9MllA", "reviewer": "Reviewer_Q2F9", "summary": "This work proposes a novel algorithm called “SemDiv” which uses LLMs to generate semantically diverse behavior in MARL. Specifically, SemDiv uses an LLM to (1) generate a language description of a plausible, novel coordination behavior, (2) generate a reward function that incentivizes that coordination behavior, and (3) validate that trained agents actually follow that behavior. SemDiv also generates a common “coordinating agent” that can coordinate with unseen partners given a language description of their convention. Experiments in multiple standard MARL environments validate the performance of SemDiv agents when paired with unseen partners.\n\n## Update After Rebuttal\n\nThe author provided satisfactory answers to my questions, so I am comfortable keeping my review as a strong accept.", "questions": "", "limitations": "", "rating": 5, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate your recognition and the valuable comments! Extra experimental results can be found in this [link](https://telling-floor-898.notion.site/1c7c2fed721a80b9ba7ef7fa2b3bffed).\n\n**Q1: The performance of population-based baselines with the multi-head network architecture.**\n\nA: We train the agents using a multi-head architecture (denoted as {}-mh) with the generated teammates. For testing with unseen teammates, we evaluate the baselines by selecting the best-performing head among all learned heads (the same as the -R1 variants in the paper), which requires extensive interactions. As shown in Table 11 in the link. the multi-head architecture significantly improves performance in LBF but yields no gains in the more complex SMACv2 environment. In both cases, a substantial performance gap remains between the baselines and SemDiv, highlighting the necessity of semantic-level diverse teammate generation.\n\n**Q2: Discuss essential references.**\n\nA: Thanks for pointing out these essential related work. We will add the discussion in the paper.\n\n- The N-AHT framework [1] does align with our problem formulation, and we extend it by introducing natural language descriptions about testing teammates, making it more suitable for multi-agent systems with communication and intention sharing.\n- InstructRL [2] is a notable work that leverages LLMs to guide RL by using their decisions to regularize policy learning. This approach offers an alternative to reward-generation methods like those discussed in our paper. However, InstructRL requires querying the LLM at every step, which is computationally expensive and limits its applicability to relatively simple tasks. Due to these constraints, we did not include it as a baseline in our study.\n- ADVERSITY [3] and CoMeDi [4] propose novel techniques for cross-play based diversity, which can be categorized into the “policy-level” methods in our paper and introduced as strong baselines.\n\n**Q3: Human user studies and more complex games.**\n\nA: Both directions present valuable opportunities, particularly in bridging the gap between simulated coordination and human-AI teamwork, as well as extending the framework to richer strategic environments. We plan to investigate these challenges in future work.\n\n**Q4: Design choices for continual learning.**\n\nA: We adopt a continual learning paradigm instead of a two-stage framework (like FCP or MEP) because it allows us to leverage previously generated grounded behaviors as positive examples, guiding the LLMs toward progressively better behavior generation. In contrast, a two-stage approach, where behaviors are generated first and then used to train the teammate population, isolates the learning process of different teammates and may reduce diversity.\n\n**Q5: Design choices for agents’ RL instead of BC.**\n\nA: While behavior cloning (BC) from the complementary policy is a viable approach, we choose RL for two key reasons: 1. Surpassing complementary policy performance: BC merely imitates the complementary policy, capping performance at the quality of the training data. In contrast, RL enables exploration and optimization beyond demonstrations, potentially discovering superior coordination strategies. 2. Mitigating distribution shift: BC may struggle with OOD teammates due to its inability to recover from unfamiliar situations. Since our focus is on generalizing to unseen teammates, BC’s reliance on static datasets may lead to failure.\n\nWe apologize for unclear expressions and typos, and will correct them in the paper.\n\n- Relationship between Section 3.3 and Macop: For training, we remove the head merging technique of Macop, since the novelty of new teammates if confirmed in the policy verification process. For testing, we utilize language-based reasoning to select optimal heads, avoiding Macop’s need for trial-and-error interactions to improving efficiency. We will rephrase the first sentence of the second paragraph in Section 3.3: To address these challenges, SemDiv adopts a multi-head network architecture \\cite{owl,macpro} similar with Macop \\cite{macop} and empowers the agents with continual learning ability.\n- We report the 95% confidence intervals (CI) of SemDiv and the next best performing method (Macop-R1) in the link.\n- Testing teammate generation: We train these teammates using hand-designed reward functions and manually verify that they exhibit the desired behaviors. These reward functions are simple and sparse (returning either 1 or 0), allowing for direct derivation of the R2 values.\n\nReferences\n\n[1] Wang et al. N-Agent Ad Hoc Teamwork. NeurIPS 2024.\n\n[2] Hu et al. Language Instructed Reinforcement Learning for Human-AI Coordination. ICML 2023.\n\n[3] Cui et al. Adversarial Diversity in Hanabi. ICLR 2023.\n\n[4] Sarkar et al. Diverse Conventions for Human-AI Collaboration. NeurIPS 2023.", "claims_and_evidence": "The claims made in the paper are supported by clear and convincing evidence.", "methods_and_evaluation_criteria": "The methods and evaluation make sense for the problem at hand (though I have specific questions about alternative approaches in the “questions for authors” section of my review)", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "For completeness, it would be good to show the performance of FCP, MEP, and LIPO with the multi-head network architecture (i.e. keep stage 1 the same but replace stage 2 with the technique described in section 3.3)", "relation_to_broader_scientific_literature": "The key contribution of this work is the pipeline of (1) generating a language description of a coordination behavior, (2) converting this description into a policy, and (3) iterating to ensure novelty and diversity.\n- Other works have addressed the middle stage of the pipeline (“InstructRL” referenced in the next section of the review). However, the pipeline as a whole is still novel.", "essential_references_not_discussed": "- The problem formulation seems to follow the N-AHT from “N-Agent Ad Hoc Teamwork” in NeurIPS 2024\n- “Language Instructed Reinforcement Learning for Human-AI Coordination” in ICML 2023 provides an alternative way of regularizing a policy given a language instruction (InstructRL)\n- “Adversarial Diversity in Hanabi” in ICLR 2023 and “Diverse Conventions for Human-AI Collaboration” in NeurIPS 2023 are additional techniques for cross-play based diversity (ADVERSITY and CoMeDi)", "other_strengths_and_weaknesses": "I consider this to be a truly revolutionary paper for the MARL field. The presented pipeline for generating a diverse set of teammates is broadly applicable to many multi-agent settings and enables many real-world applications for more personalized AI co-agents.\n\nWeaknesses:\n- There aren't any human user studies. It is currently a bit unclear how directly the presented results transfer when working with humans.\n- This paper also lacks ablations over other potential options for generating single teammates given prompts, namely InstructRL.\n- Due to the dependence on LLMs, it is unclear whether the presented technique can learn strategies in complex games that require deeper understanding of game mechanics before planning coordination strategies.", "other_comments_or_suggestions": "- It was initially unclear to me whether section 3.3 was presented as a novel contribution of this work or identical to prior work. I think this needs to be clarified in the text (i.e. rephrase the first sentence of the second paragraph in 3.3), since I was initially wondering if this is different from what Macop does when reading the results section.\n- The “best results” in tables should also include results where the confidence intervals overlap. For instance, LLM-Agent seems to be within the margin of error for many of the results.\n- “Incorporate” -> “incorporating” on second column of line 320", "questions_for_authors": "- Why is a regularization term used to constrain updates in section 3.3 instead of using a more standard multi-task RL setup? I.e. first generate the diverse training population and then generate the coordinating agent by randomly sampling partners\n    - Furthermore, why isn’t behavior cloning used, considering that the complementary policy already exists from the training regime? The complementary policy should already be a strong coordinator, so conducting RL again from scratch seems unnecessary; a QDagger-like approach such as the one from CoMeDi may be helpful.\n- How are the testing teammates generated given the descriptions? Specifically, does it follow the same pipeline as your technique (skipping section 3.1 only), is it a scripted agent, or is it using a hand-designed reward function?\n- How are R2 values calculated exactly? The definition of “satisfying teammates’ preferred coordination behaviors” is vague.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: After discussion, reviewers have reached a consensus that the paper should be accepted, arriving at scores of [3,3,5,3]. 

Reviewer Q2F9 was the strongest advocate for the paper, and explained that: "Most existing generalizable techniques (based on trajectory/policy diversity or cross-play minimization) often result in arbitrary conventions, none of which are aligned to human intuitions. By introducing LLMs for convention formation, we now have more aligned conventions while still maintaining diversity (as verified by the cross-play alignment). This whole pipeline has the potential to significantly improve the field of human-AI collaboration". 

Reviewer rgCD was the most hesitant, and while they did not feel strongly that the paper should be rejected, they had questions about how broadly applicable the method could be, and hoped that the authors would "engage more deeply with the potential of leveraging LLMs or explore ways to address some of the constraints introduced by their use."

In summary, the reviewers agree that the proposed method is novel and promising, and that the paper provides strong empirical results. However, several questioned a critical drawback of the method, which is that it assumes access to a natural language description of the teammate at test time, which may be unrealistic for real-world settings.

In the rebuttal, the authors commented on this weakness, stating "Communication and intention sharing are the basic and core techniques in multi-agent systems [1,2], mitigating partial observability. Following these techniques, our approach leverages one-time natural language communication to boost coordination efficiency. For situations where communication is entirely infeasible, SemDiv can still infer teammate behaviors from interactions using techniques in [3,4]. We note this as a future direction to extend our framework to zero-comm environments."

As AC, I still have questions about the realism of this assumption, but agree that the method for generating training teammates is interesting and novel. Given this evidence, I am recommending accept.
Decision: Accept (poster)