Language Grounded Multi-agent Reinforcement Learning with Human-interpretable Communication
Authors: Huao Li
Hossein Nourkhiz Mahjoub
Behdad Chalaki
Vaishnav Tadiparthi
Kwonjoon Lee
Ehsan Moradi Pari
Charles Michael Lewis
Katia P. Sycara
Venue: NeurIPS 2024 poster (2024)
Keywords: Multi-Agent Reinforcement Learning
Emergent Communication
Ad-hoc Teamwork
Large Language Models
Primary area: human-AI_interaction
TL;DR: We propose a novel computational pipeline to ground MARL communication in human language using embodied LLM agents, enabling interpretable and generalizable communication in ad-hoc multi-agent teamwork.

Abstract: Multi-Agent Reinforcement Learning (MARL) methods have shown promise in enabling agents to learn a shared communication protocol from scratch and accomplish challenging team tasks. However, the learned language is usually not interpretable to humans or other agents not co-trained together, limiting its applicability in ad-hoc teamwork scenarios. In this work, we propose a novel computational pipeline that aligns the communication space between MARL agents with an embedding space of human natural language by grounding agent communications on synthetic data generated by embodied Large Language Models (LLMs) in interactive teamwork scenarios. Our results demonstrate that introducing language grounding not only maintains task performance but also accelerates the emergence of communication. Furthermore, the learned communication protocols exhibit zero-shot generalization capabilities in ad-hoc teamwork scenarios with unseen teammates and novel task states. This work presents a significant step toward enabling effective communication and collaboration between artificial agents and humans in real-world teamwork settings.

Language Grounded Multi-agent Reinforcement Learning with Human-interpretable Communication
Huao Li
Hossein Nourkhiz Mahjoub Honda Research Institute USA, Inc. ios
University of Pittsburgh hul520pitt. edu
com
Behdad Chalaki
Vaishnav Tadiparthi
Honda Rescarch Institute USA, Inc. behdad_chalakii Saaaaiihoonarrrree com
Honda Research Institute USA, Inc. vaishnav_ taatanawwtaiparthiitia Shonnhhhhha---a-a rii. com
Kwonjoon Lee Honda Research Institute USA, Inc. lee@honda-ri..c. com
Ehsan Moradi-Pari
Honda Research Institute USA, Inc. emoradi @wwnda-ri...... com
Michael Lewis University of Pittsburgh ml ml0sis.pa .ptst.iitteet........... edu
Katia Sycara
Carnegie Mellon University sycaraCandrew. cmu. edu
Abstract
Multi-Agent Reinforcement Learning (MARL) methods have shown promise in enabling agents to learn a shared communication protocol from scratch and accom- plish challenging team tasks. However, the learned language is usually not inter- pretable to humans or other agents not co-trained together, limiting its applicability ad-hoc teamwork scenarios. In this work, propose novel computational pipeline that aligns the communication space between MARL agents with an em- bedding space of human natural language by grounding agent communications on synthetic data generated by embodied Large Language Models (I (LLMM) in- teractive teamwork scenarios. Our results demonstrate that introducing language grounding not only maintains task performance but also accelerates the emergence of communication. Furthermore, the learned communication protocols exhibit zero-shot generalization capabilities in ad-hoc teamwork scenarios with unseen teammates and novel task states. This work presents significant step toward enabling effective communication and collaboration between artificial agents and humans in real-world teamwork settings.
Introduction
Effective communication is erucial for multiple agents to collaborate and solve team tasks. Especially in ad-hoc tcamwork scenarios where agents do not coordinate priori, the ability to share information via communication the keystone for successful team coordination and good team performance [30]. Multi-Agent Reinforcement Learning (MARL) methods have shown promise in allowing agents to
learn shared communication protocol by maximizing the task reward 38 37 However, merely
Work done while interning at Honda Research Institute USA. Code available at htttp::/rommnller gi tttbb.oo/laaggrouuddd
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

maximizing the utility of communication in specific task goals might compromise task aask-agnostic objectives such as optimizing language complexity and informativeness 440 [6. making the learned communication protocols 1) hard to interpret for humans or other agents that are not co-trained together [T8 20 6 and 2) highly iata-inefffcient (9]. In addition, most previous works learn com- munication protocols with atomic symbols or combination of them leaving the relation between symbols unexplored [50]. Only a few researches attempt to learn a semantic space for zero-shot com- munication of unseen states 42 17 One of the popular directions for interpretable communication is to regulate the learning process with external knowledge from human languages Ell (22] 28] M..
However, this process is challenging due to the divergent characteristics between human and machine languages by nature [3] For example, most agent training methods (c.g., deep reinforcement learning) require huge amount of data that is impractical for human- in-the-loopp training even collecting from humans 28
The rise of Large Language Models (LLMs)) provides new opportunities in grounding agent commu- nication with human languages. Recent generative models fine-tuned with human instructions (e.g.. GPT-4, Llama 3) show reasonable capabilities in completing team tasks and communicating in human-like fashion via embodied interaction [36] 25). Essentially, LLMs encapsulate a highly trained model of human language patterns in teamwork, allowing them to generate descriptions and responses that are well-grounded in natural language. For the purpose of guiding multi-agent communication, they represent the most generally available reference based on vast corpus of human language data that would be infeasible to collect through other means. However, LLMs are known to suffer from a lack of grounding with the task environments (a.k.a. hallucinations), which prevents embodied agents from generating actionable plans [29] While attempts have been made to ground L.LMs with
reinforcement learning or interactive data collected from environments 47 4 39]. none of them involve teamwork nor communication among multiple embodied agents.
In this paper, we propose LangGround, a novel computational pipeline for artificial agents to learn ceman-interpretable communication for ad-hoc human-agent teamwork. Specifically, we use synthetic data generated by LLM agents in interactive teamwork scenarios to align the communication space between MARL agents with human natural language. Learning signals from both language grounding and environment reinforcement regulate the emergence of communication protocol to optimize both team performance and alignment with human language. We have also evaluated the learned communication protocol in ad-hoc teamwork scenarios with unseen teammates and novel task states. The aligned communication space enables translation between high-dimensional embeddings and natural language sentences, which facilitates ad-hoc teamwork. The proposed computational pipeline does not depend on specific MARL architecture or LLMs and should be generally compatible. We have sought to minimize the influence of prompt engineering to ensure the seamless applicability of our approach in diverse environments. To the best of our knowledge, this work is among the very first attempts training MARL agents with human- iettrriiterrretabbee natural language communication and evaluating them in ad hoc teamwork experiments.
Related Work
2.1 Multi-Agent Communication
Reinforcement learning has been used to coordinate the teamwork and communication among multiple agents in partially observable environments. In earlier works such as DIAL [12). CommNet (38), and IC3Net [37]] agents learn differentiable communication in an end-to-end fashion under the pressure of maximizing task reward. Other works use shared parameters [7 or a centralized controller (33] to stabilize the non-stationary lcarning process of multi-agent communication. More recently, representation lcarning methods such autoencoder (26) and contrastive Icarning [27) are used to ground an agent's 's communication on individual observations. However, Comm-MARL methods usually suffer from overfitting to specific interlocutors trained together [19]. The learned communication protocols can not be understood by unseen teammates in ad-hoc teams, let alone another human.
Another relevant trend of research is Emergent Communication (EC), where researchers focus more on simulating the development of natural (i.e., symbolic) language with artificial agents [19 15 The most common task scenarios used in the EC community are reference games or Lewis signaling games 224, in which a speaker must describe an object to listener, who must then recognize

---

it among a set of distractions [42. However, previous research has shown that learning EC in more complicated, scaled-up, and multi-round interactive task scenarios can be challenging or even infeasible 9 7. Even in situations where agents can learn to communicate, the learned protocols are usually either not human-interpretable
18 or semantically drifting from human language [23
2.2 Human-Interpretable Communication
To address the above-mentioned challenges, several recent works propose human-interpretall communication in RL. settings. Lazaridou al. leverage pre-trained task-specific language models to provide high-level guidance for natural language communication. A few other works align low- or learn discrete prototype communication But as pointed out in several studies [8, 49], the low mutual intelligibility between human language and neural agent communication makes the alignment process non-trivial. Our work is closest to (28], in which researchers alternate imitating human data via supervised learning and self-play to maximize reward a reference game. The differences that in (28) authors try train neural agents for reference games in an end-to-end fashion with backpropagation, while we train MARL agents in interactive team tasks. The exploration of this research direction is still very limited, as no previous work has ever evaluated natural language communication agents within interactive task environments and for ad-hoc human-agent teams.
level communication tokens with human language
in semantically meaningful space 42
2.3 Grooudde Reinforcement Learning
Reinforcement learning is known struggle with long-horizon problems with sparse reward sig- nals (31]. Natural language guidance has been used to provide auxiliary rewards to improve the data efficiency and learning robustness (44) Goyal et al. [13 use step-by-step natural language instructions provided by human annotators to construct auxiliary reward-learning modules, encouraging agents to learn from expert trajectories. Narasimhan al. [32) research the impact of language grounding on representation learning and transfer learning of RL agents in 2D game environment. Additional work has explored grounding RL with other formats of materials such as game manuals [4 46) and
human commands [48). However, none of those works has ever used the communication messages as the language ground nor guided the information-shariee
process in multi multi-agent teamwork.
The most relevant research to our proposed method is CICERO, which empirically evaluates the proposed Al agent in the complex natural language strategy game Diplomacy with human players (IT) CICERO has an RL module for strategic reasoning and a language model for generating messages. The two modules are trained separately conversation data collected from human players. The two modules in CICERO function independently. with the only connection being that the language model takes intention estimation from the planning module as input. While in our work, both action and communication are generated by individual RL agents that are trained end-to-end with combination of RL loss and supervised learning loss.
different datasets, namely self-play trajectories and
3 Preliminaries
We formulate the problem decentralized partially observable Markov Decision Process 351 which can be formally defined by the tuple (I A, C, T, Ω2 O, 7), where I is the finite set of agents, s E S is the global state space, = AięTAA; is the set of actions, and C = XiçTC; is the set of communication messages for each of n agents. T S x A + S is the transition function that maps the current state 8t into next state 81+1 given the joint agent action. In our partially observable environments, each agent receives a local observation o² E Q according to observation function O : S x C x I + Ω. Finally, R S A + F. is the reward function, while 7 E [o, 1) is the discount factor. At each time stamp t, each agent i takes an action of and sends out a communication message of after receiving the partial observation of task state 9; along with all messages sent by other agents from last time stamp C-11 Each agent then receives the individual reward r E R(st, oz). We consider fully cooperative settings which the reinforcement learning objective is to maximize the total expected return of all agents:
max
E
ΣΣ LET
( ~
(I)
T AXC
3

---

We borrow the definition of language learning from [28] We define a target language L* E L that we want the agents to learn, assuming C is the set of natural language and L* is the optimal communication language for achieving a good team performance in the specific task. Specifically, we consider a language agent observations to communication messages defined as L : f2 + C. In typical RL. settings, this can be thought as the mapping between input observation vectors and English descriptions of the observation. We consider a dataset D consisting of [D] (observation, action) pairs, which comes from expert trajectories generated by LLM embodied agents using the target language L*. The language learning objective is train agents speak language L* in order to collaborate with experts in ad-hoc teamwork. It is worth noting that we want the learned language to generalize to unseen examples that are not contained in D.
L to a set of communication messages C which are mapped from
To train agents that perform well on the team task and speak cuman-interpretable language, we need to solve multi-objective learning problem by combining the learning signals from both environment reward and supervised dataset D. The process of training an optimal communication-acn can be defined solving an optimization problem with two constraints 1) agents must learn to communicate effectively to maximize team performance and 2) agents must learn to use similar language as in the supervised dataset D.
policy
4 Language Grounded Multi-agent Communication
Action
Environment
LSTM
Observations
LLM agents self-play
Comm vectors
mean
MARL agent
Dataset
Comm vectors
Gate
Comm
Nel
Language Grounding
Translation Module
LSTM
During training
Human Interpretable Communication
Observations
During ad-hoc teamwork
Figure 1: Illustrations of our proposed computational pipeline LangGround. The framework consists of three modules: 1) collecting grounded communication from LLM agents, 2) aligning MARL communication with language grounds, 2) translating aligned communication vectors into natural language messages via cosine similarity matching.
In this section, we propose computational pipeline for learning sogguagg-groundee multi-agent communication (rongGround). The general framework LangGround is illustrated in Figure consisting of two parts: collecting grounded communication from LLM agents and aligning MARL communication with language grounds.
4.1 Grounded communication from LLM agents
We use embodied LLM agents to collect samples of target language Lxx, To allow LLM agents to interact with the task environments, a text interface is implemented following (25) to translate between abstract representations and English descriptions. Essentially, each of the LLM agents provided with general prompts about the team task and instructed collaborate with others to achieve the common goal. At each time stamp t, the LLM agent receives English descriptions of its own observation of the environment [(o]) which also includes communication messages from teammates in the last time stamp C.-1.The LLM agent is prompted output its next action and communication message which are then encoded into abstract Ai and C; and used to update the task environment.
Theoretically, we construct text environment in parallel with the actual task environment to ensure that embodied LLM agents are exposed to the equivalent information RL agents, albeit in different

---

format. The action and communication policy of LLM agents emerge from the backbone LLM, since the provided prompts do not include any explicit instructions on team coordination or communication strategy. In the results section we will confirm findings from previous literature that modern L.MSs (e.g,, GPT-4) is able perform reasonably well and communicate effectively in collaborative tasks. The expert trajectories generated by LLMs are used to construct the supervised dataset 'D which maps individual agent's (observation, action) pairs to natural language communication messages. D is used during the training of MARL to provide supervised learning signals to align the learned communication protocols with human language. More implementation details of LLM agents and data collection can be found in the Appendix.
4.2 Multi-agent Reinforcement Learning with aligned communication
The MARL with communication pipeline is similar to IC3Net 37 in which each agent has an independent controller model to learn how and when to communicate. During each time-step, input observation of is encoded and passed into each agent's individual LSTM. The hidden state of LSTM h4 is then passed to the probabilistic gating function to decide whether to pass message to other agents. A single-layer communication network transforms hū into communication vector ct. The mean communication vector of all agents is finally used by each agent's 's LSTM to produce the action ai.
To shape the learned communication protocol toward human language, we introduce an additional supervised learning loss during the training of MARL. Specifically, at each time step, sample reference communication from the dataset based on each agent's 's observation and action, D(o af). representing how human (LLM) would communicate in the same situation. We then calculate the cosine similarity between the agent communication vector c; and the word embedding of the reference communication с₁ To align the communication space learned by MARL with the sighddimensional embedding space of natural language, we construct the supervised loss function as follows:
Leup
ΣΣ iET
[1 cos(c, D(o a;))
(2)
at) = -(00
if ao) E D
otherwise
The construction of the communication message is shaped by two learning signals: 1) the reinforce- ment learning objective which determines useful information to share with other agents based on the policy loss gradient, and 2) the supervised learning objective which imitates the communication messages used by LLM agents in dataset D. The total loss function is formulated as follows:
= == + XLegup
The hyperparameter À is used to scale the supervised loss. Each agent's policy is optimized with backpropagation to minimize the joint loss.
5
Experiments
5.1
Environments
In this section, we evaluate our proposed method in two multi-agent collaborative tasks with varied setups and different characteristics. The first environment, Predator Prey [37], is widely used in Momm-MARL research as a benchmark. We include this environment to represent team tasks that require all agents to share their partial observations for the team form a complete picture of the task state. The second environment, Urban Search & Rescue (USAR) 25 34 presents a more demanding challenge due to the inclusion of heterogeneous team members and the temporal dependence between their behaviors. Here, agents must communicate not only their observations but also their intentions and requests to effectively coordinate. Illustrations of the two environments are shown in Figure T and more details provided in the Appendix.
5

---

Predatory Pray (visin==1)
Preditoory Pray (vision
LSAR
scCommn
VGHB
pratsComm
Ispadd Langih
80
40
20-
3.0
1.0
8.0
0.4 0.0 Thatling Gemoscartpa
5.0
Thunning Srmsssamp
Traniing Timrsttaaa
Figure 2: Learning curves of LangGround in comparison with baseline methods. The y-axis is task performance measured by the episode length until task completion, which is lower the better. The x-axis the number of training timestamps. Shaded areas are standard errors over three random seeds.
5.2 Experiment setups
We compare our proposed pipeline LangGround against previous methods, including IC3Net 37], autoencoded communication (aeComm) [26), Vector-Quantized Variational Information Bottleneck (VQ-VIB) [40], prototype communication (protoComm) [42]. and a control baseline of independent agents without communication (noComm). aeComm represents the state-of-the-ee communication methods that grounds communication by reconstructing encoded observations. has been shown outperform end-to-end RI. methods and inductive biased methods in independent, decentralized settings. VQ-VIB and prototype-based method are representative solutions for human- interpretable communication, which learn semantic space for discrete communication tokens and perform reasonably well in human-agent teams. Finally, IC3Net has similar architecture to our proposed pipeline representing an ablating baseline without language grounding.
multi-agent
All methods are implemented with the same centralized training decentralized execution (CTDE) architecture for fair comparison. Each agent has observation encoder, an LSTM for action policy, a single-layer fully-connecter neural network for transferring hidden states into communication messages, and gate function for selectively sharing messages. The parameters of the action policy and obs encoder are shared during training for a more stable learning process. We use REINFORCE [ to train both the gating function and policy network. The communication messages are continuous vectors of dimension 1) 256.
Results
As for the evaluation matrices, first consider if LangGround allows MARL agents complete collaborative tasks successfully (i.e., task utility) and converge to a shared communication protocol quickly (i.e., data-efficiency). in comparison with other mate-of-the-ar methods the baselines. Then we analyze the properties of aligned communication space such as human interpretability. topographic similarity, and zero-shot generalizability, show how close the learned language is to the target human natural language. Finally, we evaluate the ad-hoc teamwork performance in which MARL agents must communicate and collaborate with unseen LLM teammates via natural language.
6.1 Task performance
In Figure 2
we compare the task performance of multi-agent teams using different communication methods by plotting out the average episode length during training over 3 random seeds with standard errors.
In Predator Prey vision 1 (i.e., PPe1), our method LangGround achieves a similar final perfor- mance with IC3Net and aeComm, outperforming other baselines. However, the improvement not outstanding due to the simplicity of the task environment. In Predator Prey vision = 0 (i.e., pP-o), the predator's s vision range is limited to their own location making effective information sharing more important in solving this search task. As shown in the middle figure, LangGround has

---

Agent Observation
Communication embedding space
Agent Observation
Pay
30
-35
Converging prey location
Moving (0:
3]
-100
100
Natural Language Message
Natural Language Message
Figure 3: Learned communication embedding space. Communication vectors between agents in PPv0 are visualized with t-SNE and clustered with DBSCAN. Two semantically meaningful clusters are identified as examples, each corresponding to specific agent observation. We also present the most similar reference message from dataset 'D to illustrate the alignment between the agent communication space and the human language embedding space.
comparable final performance with aeComm and outperforms other baselines. Finally, in the most challenging USAR environment, LangGround outperforms all baselines in solving the task in fewer steps with the same amount of training time-steps. In addition, language grounding also stabilizes the communication learning process such that the variance of LangGround is much smaller than other methods.
To test the performance of proposed method in scaled environments, we ran additional experiments in Predator Prey with larger map size (i.e., PPv1 (10 by 10)). The learning curves of LangGround and baselines are presented in Fig 4 shown the figure, our method outperforms ablating baselines without language grounding (i.e., IC3Net) or without communication (i.e., noComm). This result demonstrates the benefit of introducing LangGround in stabilizing the learning process of emergent communication of MARL agents in scaled environments.
In summary, LangGround enables multi-agent teams to achieve on-par performance in comparison with SOTA multi-agent communication methods. Introducing language grounds as an auxiliary learning objective does not compromise the task utility of learned communication protocols while providing interpretability.
6.2 Aligned communication space
In addition to task utility, also interested in other properties of the learned communication space, such as human interpretability, topographic similarity, and zero-shot generalizability.
6.2.1 Semantically meaningful space
To evaluate whether the learned communication space is semantically meaningful, we visualize the learned communication embedding space by clustering message vectors sent by agents over 100 evaluation episodes in pp.o. following [26). The high dimensional vectors are reduced to two dimension space via t-SNE 43 and clustered with DBSCAN [TO]. As shown Figure 3 agent communication messages can be clustered into several classifications with explicit meanings associated with agent observation from the environment. For example, the pink cluster on the right side corresponds to the situation where the agent locates in coordinates (0, 3) without vision of prey, can look up from dataset D for the reference message that has most similar word embedding with agent communication vectors in the pink cluster. The reference message (i.e., "moving down from (0, 3)") accurately refers to the agent observation, indicating learned communication space is semantically meaningful and highly aligned with natural language embedding space.

---

Grounding
Cos sim
Bleu score
100%
0.775
0.633
Env
A Cos sim
A Bleu score
75%
0.667
0.498
PPo1
0.82++002
0.52
50%
0.304
0,308
PPen
0.81=0.03 0.79=#012
0.45#0.12 12 0.42:t0.04
25%
0.188
0.224
USAR
Table 2: Zero-shot generalization evaluation results on ppv1 (10 by 10).
Table Similarity gain w/ LangGround
6.2.2 Human interpretability
Given the goal of aligning agent communication with human language, it is intuitive to evaluate the human interpretability of language- grounded agent communication. We use the offline dataset D as the reference to calculate the similarity between communication messages shared by LangGround agents and LLM agents in same situations. Given 100 evaluation episodes in ppen. we calculate 1) the cosine similarity between word embedding and agent communication vectors, and 2) BLEU score between natural language messages and reference messages in D with the most similar word embedding as agent communication vectors. The results are shown in Table [ demonstrating that LangGround achieves significant gains in both metries, cosine similarity and BLEU score, compared to baselines without alignment. These measures for other baseline methods would be equivalent to random chance because none of them are grounded with language during training, and hence we do not compute their performance on these metrics.
language-grounded
6.3 Ad-hoe Teamwork
The ultimate goal of our proposed pipeline is to facilitate ad-hoc teamwork between unseen agents without pre-coordination. Here, we propose two experiments to evaluate the zero-shot generalizability and ad-hoc collaboration capability of our trained agents.
6.3.1 Zero-shot generalizability
One prerequisite of ad-hoc teamwork is the ability to communicate about unseen states to their team- mates. We evaluate this capability by removing a subset of prey spawn locations from environment initialization of pp-o and training LangGround agents from scratch. In this condition, agents would neither exposed to nor receive any language grounding for those situations during training. During the evaluation, record the communication messages used by agents in those unscen situations and compare them with ground truth communication from dataset D. Results show that agents are still able complete tasks when the prey spawns in those 4 unseen locations. As shown in Table 3 the communication messages agents used to refer to those unseen locations are similar to natural language sentences generated by LLMs.
Precitoory Prey (10 By 101
Predatory Prey (10 by 101
Predatory
20
2315 g ournded
Suprested
0.3
0.7
M
0.2
0.2
1000 groundic
Mann
13% groumneed
Mas
25% grounded
Thairing Emmusarrpa
Training Ummasaarpa
Thaining Simunzarrr
Figure 4: Team performance of LangGround and baselines on Predator Prey with 10 by 10 map and vision range of 1.
Figure Team performance of LangGround agent with differ- ent levels of language grounding on pp.1 (10 by 10).
Figure 6: Communication align- ment of LangGrounnd agent with different levels of language grounding on ppv1 (10 by 10).

---

Table 3: sero-shot generalizability in pPou
Prey Loc
Cos sim
Bleu
Example message
(1,1)
0.81
0.41
Moving up to converge on prey location (1,0) for capture
(1,3)
0.81
0.27
Converging on prey location at (1,3)
(3.1)
0.82
0.51
Moving up toward prey location at (3.1)
(3,3)
0.78
0.72
Converging on prey location at (3,3)
Additionally, we train LangGround agents on Predator Prey (10 by 10) but only provide language grounding in a subset of states (i.e., 25%, 50%, 75%, 100%). The learning curves LangGround agents with different levels of language grounding in the figures, the more grounded states, the better the team performance, as well as the better the communication alignment. Table shows states during evaluation. The first column refers to percentage of grounding data used during training. The other two columns alignment measurements when agents encounter un on-grounded states during evaluation.
presented in Fig. S and Fig. 6 As shown
shows similar results of communication alignment in un-grounded
To summarize, these findings confirm the alignment between the agent communication space and the human language word embedding space zero-shot conditions. More importantly, we show that LangGround is not merely memorization of one-to-one mapping between observations and communications but also shapes the continuous communication space a semantically meaningful way. We could ground the agent communication space with the word embedding space of human language on a limited number of instances, and expect LangGround to output interpretable messages in un-grounded states via topographic similarity (discussed more in the Appendix) of the aligned space. In practice, this assumption depends on many factors such as the coverage of offline dataset, number of grounded states, dimension of the communication space, scale of the problem, etc. The above experiments illustrate the impact of language grounding percentage, and we leave further investigation to future work.
6.3.2 Ad-hoe teamwork between MARL and LLM agents
Finally, we evaluate the performance of our agents to work with unseen teammates in ad-hoc teamwork settings. Ad-hoe teamwork refers to situations where agents collaborate with unseen teammates without pre-coordination. In this work, we use embodied LLM agents to emulate human behaviors in human-agent ad-hoc teams. Teams with 2 MARL agents and LLM agent were evaluated on 8 episodes over 3 random seeds, resulting in a total of 24 evaluation episodes per condition. Ad-hoc teamwork performance is measured by the number of steps taken to complete the task; therefore, lower better. Means and standard deviations of each condition are reported Table
Table 4: Ad-hoc teamwork performance (lower better)
Team composition
PPv1
PPvo
USAR
LangGround
4.3 + 1.20
10.9 + 4.53
22.0 + 4.24
LLM
6.8 + 5.20
11.6 5.30
15.9 + 3.37
LangGround LLM
8.5 - 5.76
15.5 - 4.80
23.2 + 10.61
aeComm + LLM
10.3 + 6.46
17.5 = 4.60
20.3 + 9.07
noComm + LLM
10.6 + 5.73
20.0 + 0.00
32.4 + 13.47
We find that 1) homogeneous teams (i.c., LangGround and LLM) achieve better performance than ad-hoc teams because of their common understanding both action and communication. As those agents are either co- co-trained together or duplicates of the same network, they form stable strategy for team coordination and information sharing. Since ad-hoc teams (e.g.. LangGround + LLM) were not trained together nor speak the same language, their decreased performance is expected. 2) The ad-hoc team performance of LangGround agents is better than noComm and aeComm agents in at least two out of three evaluation scenarios. Because aeComm is not aligned with human language, it serves as a baseline with coordinated action policy and a random communication policy. The advantage of our method over acComm and noComm merely comes from effective information sharing via the

---

aligned language with unseen teammates. The empirical evidence presented in this section confirms the application of our method in ad-hoc teamwork.
7 Discussion
In this work, we developed a novel computational pipeline to enhance the capabilities of MARL agents to interact with unseen teammates in ad-hoc teamwork scenarios. Our approach aligns the communication space of MARL agents with an embedding space of human natural language by grounding agent communications synthetic data generated by embodied LLMs in interactive teamwork scenarios.
Through extensive evaluations of the learned communication protocols, we observed a trade-off between utility and informativeness. According to the Information Bottleneck principle [40] in- formativeness corresponds to how well a language can be understood in task-agnostic situations, while utility corresponds to the degree to which a language is optimized for solving a specific task. By introducing the additional supervised learning signal, our method pushes the trade-off toward informativeness compared to traditional comm-MARL. methods that merely optimize for utility. This partially explains the "inconsistent" patterns we observe in Figure 2 and Table 4 across different task scenarios. In relatively easy tasks such as predator-prey. the learned communication more optimized for informativeness, aligning better with human language and generalizing better in ad ad-hoc teamwork. In more challenging tasks such as USAR, the learned communication more shaped toward task utility, resulting faster convergence but less interpretability to unseen teammates.
Additionally, we found that introducing language grounding does not compromise task performance but even accelerates the emergence of communication, unlike the results reported in previous literature, where jointly optimizing communication reconstruction loss and RL loss leads to a drop in task performance [26] 27 The main reason for this contradiction is that our dataset D consists of expert trajectories from LLM embodied agents with a well-established grounding on the task. Therefore, the language grounding loss not only shapes the communication but also guides the action policy of MARL agents by rewarding behavior cloning and providing semantic representations of input observations.
worth noting that LangGround is an extremely flexible pipeline with most of the modules being interchangeable, such as the word embedding model, base MARL-comm model, message dataset source, and the translation module. This allows us allocate appropriate tasks LLMs and RL respectively, considering LLMs are known to have good linguistic capabilities (e.g., describing) while struggle with formal reasoning (e.g., planning). While we use embodied LLM agents to collect grounding communication datasets for LangGround as explained in Section These datasets can also come from rule-based agents or human participants, as long as they show effective communication in solving collaborative tasks. Because only communication messages are used, LLM agents' hallucinations and does not impact MARL's task performance directly. In more complicated scenarios in which communication is beyond describing observation and action, could still expect LLMs to generate reasonable outputs. Compared to alternative methods in embodied agents where LLMSs must make correct action planning every timestamp, it is more feasible to collect semantically meaningful messages from either LLMs or any other sources.
We believe our work would benefit the broader society for the following reasons. This research provides empirical evidence of linguistic principles during language evolution among neural agents, which might provide insights for broader research communities, including computational linguistics, cognitive science, and social psychology. The usage of embodied LLM agents as interactive simulacra of human team behaviors has broad impact since it has potential applications in social science and may deepen our understanding of modern LLMs. Most importantly, our proposed pipeline takes initial steps in enabling artificial agents to communicate and collaborate with humans via natural language, shedding light on the broad research direction of Human-centered AL.
As for future directions, we plan to evaluate our proposed pipeline in more complicated task environ- ments at scale and experiment with different selections of MARL algorithms, backbone LLMs, and word embeddings. Particularly, we plan to replace the use of a static dataset D by querying LLMS online during the training of MARL. This may allow us to capture complex information exchanged among team members in addition to individual observations, such as beliefs, intents, and requests.
10

---

Acknowledgments and Disclosure of Funding
We would like to thank Mycal Tucker, Seth Karten and the original authors of IC3Net for providing public access to their code base that were instrumental in this research. We also appreciate the helpful discussions with Lu Wen, Yikang Gui, and Vasanth Reddy that contributed to the development of this work.
References
[1] Akshat Agarwal, Swaminathan Gurumurthy, Vasu Sharma, Mike Lewis, and Katia Sycara. Community regularization of visually-grounded dialog. In Proceedings of the 18th International Conference Autonomous Agents and MultiAgent Systems, pages 1042-1050, 2019.
[2] Henry Brighton and Simon Kirby. Understanding linguistic evolution by visualizing the emergence topographic mappings. Artificial life. 12(2):229 242, 2006.
[3] Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojciech Zaremba. Openai gym, 2016.
[4] Thomas Carta, Clément Romac, Thomas Wolf, Sylvain Lamprier, Olivier Sigaud, and Pierre- Yves Oudeyer. Grounding large language models in interactive environments with online reinforcement learning. In International Conference on Machine Learning, pages 3676-3713. PMLR, 2023.
[5] Rahma Chaabouni, Eugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Anti-efficient encoding in emergent communication. Advances in Neural Information Processing Systems. 32, 2019.
[6] Rahma Chaabouni, Fugene Kharitonov, Emmanuel Dupoux, and Marco Baroni. Communicating artificial neural networks develop efficient color-naming systems. Proceedings of the National Academy of Sciences, 1 (18(12)e201656))) 6569118, 2021.
[7] Rahma Chaabouni. Florian Strub, Florent Altché, Eugene Tarassov, Corentin Tallec, Elnaz Davoodi, Kory Wallace Mathewson, Olivier Tieleman, Angeliki Lazaridou, and Bilal Piot. Emergent communication at scale. In International conference on learning representations. 2021.
[8] Roberto Dessi, Eleonora Gualdoni, Francesca Franzon, Gemma Boleda, and Marco Baroni. Communication breakdown: On the low mutual intelligibility between human and neural captioning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 8998-8007, 2022.
[9] Tom Eccles, Yoram Bachrach, Guy Lever, Angeliki Lazaridou, and Thore Graepel. Bi- ases for emergent communication in multi-agent reinforcement learning. Advances in neural information processing systems, 32, 2019.
[10] Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pages 226-231, 1996.
[11] Meta Fundamental AI Research Diplomacy Team (FAIR)+, Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 37866624)1067-10777 2022.
[12] Jakob Foerster, Ioannis Alexandros Assael, Nando De Freitas, and Shimon Whiteson. Learning communicate with deep multi-agent reinforcement learning. Advances in neural information processing systems, 29, 2016.
[13] Prasoon Goyal, Scott Niekum, and Raymond J Mooney. Using natural language for reward shaping in reinforcement learning. Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 2385-2391, 2019.

---

[14] Austin W Hanjie, Victor Y Zhong, and Karthik Narasimhan. Grounding language to entities and dynamics for generalization in reinforeement learning. In International Conference on Machine Learning, pages 4051-4062 PMIR, 2021.
[15] Seth Karten, Siva Kailas, Huao Li, and Katia Sycara. On the role of emergent communication for social learning in multi-agent reinforcement learning. arXiv preprint arXiv:2302.1422 14276 2023.
[16] Seth Karten and Katia Sycara. Intent-groutn -rrtundrudddd compositional communication through mutual information in multi-agent teams. In Workshop on Decision Making Multi-Agent Systems at International Conference on Intelligent Robots and Systems (IROS), 2022.
[17] Seth Karten, Myeal Tucker, Huao Li, Siva Kailas, Michael Lewis, and Katia Sycara. In- terpretable learned emergent communication for human-agent teams. IEEE Transactions on Cognitive and Developmental Systems, 2023.
[18) Satwik Kottur, José Moura, Stefan Lee, and Dhruv Batra. Natural language does not emerge 'naturally'in in multi-agent dialog. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2962 2962, 2017.
[19] Angeliki Lazaridou and Marco Baroni. Emergent multi-agent communication in the deep learning era. arXiv preprint arXiv:2006.02419 2020.
[20] Angeliki Lazaridou, Karl Moritz Hermann, Karl Tuyls, and Stephen Clark. Emergence of linguistic communication from referential games with symbolic and pixel input. In International Conference on Learning Representations, 2018.
[21] Angeliki Lazaridou, Alexander Peysakhovich, and Marco Baroni. Multi-agent cooperation and the emergence of (natural) language. arXiv preprint arXiv:1612.07182, 2016
[22] Angeliki Lazaridou, Anna Potapenko, and Olivier Tieleman. Multi-agent communication meets natural language: Synergies between functional and structural language learning. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics.
pages 7663-7674, 2020.
[23] Jason Lee, Kyunghyun Cho, and Douwe Kiela. Countering language drift via visual grounding. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing pages 4385 138554395,20 2019.
[24] David Lewis. Convention: A philosophical study, John Wiley & Sons, 2008
[25] Huao Li, Yu Chong, Simon Stepputtis, Joseph Campbell, Dana Hughes, Charles Lewis, and Katia Sycara. Theory of mind for multi-agent collaboration via large language models. In Proceedings the 2023 Conference on Empirical Methods Natural Language Processing, pages 180- 2023.
[26] Toru Lin, Jacob Huh, Christopher Stauffer, Ser Nam Lim, and Phillip Isola. Learning to ground multi-agent communication with autoencoders. Advances in Neural Information Processing Systems, 14:15020-1524,, 2021.
[27] Yat Long Lo, Biswa Sengupta, Jakob Nicolaus Foerster, and Michael Noukhovitch. Learning multi-agent communication with contrastive learning. The Twelfth International Conference on Learning Representations. 2023.
[28] Ryan Lowe, Abhinav Gupta, Jakob Foerster, Douwe Kiela, and Joelle Pineau. On the interaction between supervision and self-play in emergent communication. In International Conference on Learning Representations, 2019.
[29] Kyle Mahowald, Anna A Ivanova, Idan A Blank, Nancy Kanwisher, Joshua B Tenenbaum, and Evelina Fedorenko. Dissociating language and thought in large language models: a cognitive perspective. arXiv preprint #rXiv:2301.0661,, 2023.
12

---

[30] Reuth Mirsky, William Macke. Andy Wang, Harel Yedidsion, and Peter Stone. A penny for your thoughts: The value of communication ad hoc teamwork. International Joint Conference on Artificial Intelligence, 2020.
[31] Volodymyr Mnih, Koray Kavukeuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518(7540):52-533, 2015.
[32] Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. Grounding language for transfer in deep reinforcement learning, Journal of Artificial Intelligence Research, 63:849-874 874, 2018.
[33] Yaru Niu, Rohan R Paleja, and Matthew C Gombolay. Multi-agent graph-attentnn communica- tion and teaming. In AAMAS, volume 21, page 20th, 2021.
[34] Ini Oguntola, Dana Hughes, and Katia Sycara. Deep interpretable models of theory of mind. In 2021 30th IEEE international conference on robot & human interactive communication (RÓ-MAN), pages 657- IEEE, 2021.
[35] Frans A Oliehoek, Christopher Amato, al. concise introduction to decentralized POMDPs, volume 1. Springer, 2016.
[36] Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S Bernstein. Generative agents: Interactive simulacra of human behavior. In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology, pages 1-22, 2023.
[37] Amanpreet Singh, Tushar Jain, and Sainbayar Sukhbaatar. Learning when to communicate at scale in multiagent cooperative and competitive tasks. In International Conference on Learning Representations, 2018.
[38] Sainbayar Sukhbaatar, Rob Fergus, et al. Learning multiagent communication with backpropa- gation. Advances in neural information processing systems, 29, 2016
[39] Weihao Tan, Wentao Zhang, Shanqi Liu, Longtao Zheng, Xinrun Wang, and Bo An. True knowledge comes from practice: Aligning 11ms with embodied environments via reinforcement learning. arXiv preprint arXiv:2401. 14151, 2024.
[40] Mycal Tucker, Roger Levy, Julie A Shah, and Noga Zaslavsky. Trading off utility, informative- ness, and complexity in emergent communication. Advances in neural information processing systems, 35:222 14-22228, 2022.
[41] Mycal Tucker, Roger P Levy, Julie Shah, and Noga Zaslavsky. Generalization and translatability in emergent communication via informational constraints. In NeurIPS 2022 Workshop on Information- Information-Theorrtii Principles in Cognitive Systems, 2022.
[42] Mycal Tucker, Huao Li, Siddharth Agrawal, Dana Hughes, Katia Sycara, Michael Lewis, and Julie A Shah. Emergent discrete communication in semantic spaces. Advances in Neural Information Processing Systems, 34: 34:10574-10586 2021.
[43] Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine learning research, 9(11), ().1),22 2008
[44] Nicholas Waytowich, Sean Barton, Vernon Lawhern. Ethan Stump, and Garrett Warnell. Grounding natural language commands to starcraft ii game states for narration narration-guided rein- forcement learning. In Artificial intelligence and machine learning for multi-domain operations applications, volume 11006, pages 267-276. SPIE, 2019.
[45] Ronald J Williams. Simple statistical following algorithms for connectionist reinforce- ment learning. Machine learning, 8(3):229-256, 1992.
[46] Yue WI, Yewen Fan, Paul Pu Liang, Amos Azaria, Yuanzhi Li, and Tom M Mitchell. Read and reap the rewards: Learning to play atari with the help of instruction manuals, Advances in Neural Information Processing Systems, 36, 2024.
13

---

[47] Jiannan Xiang, Tianhua Tao, Yi Gu, Tianmin Shu, Zirui Wang, Zichao Yang, and Zhiting Hu. Language models meet world models: Embodied experiences enhance language models. Advances in neural information processing systems, 36, 2024.
[48] Shusheng Xu, Huaijie Wang, and Yi Wu. Grounded reinforcement learning: Learning to win the game under human commands. Advances Neural Information Processing Systems, 35:7504-7519, 2022.
[49] Shunyu Yao, Mo Yu, Yang Zhang, Karthik R Narasimhan, Joshua B Tenenbaum, and Chuang Gan. Linking emergent and natural languages via corpus transfer. arXiv preprint arXiv:2203.1333 13344, 2022.
[50] Changxi Zhu, Mehdi Dastani, and Shihan Wang. survey multi-agent deep reinforcement learning with communication. Autonomous Agents and Multi-Agent Systems, 38(1):4, 2024.
14

---

A Implementation details
A.1 Embodied LLM agents
Large language models are prompted to interact with the task environments in team tasks. We implement embodied LLM agents based on the pipeline proposed in (25), where agents are augmented with explicit belief state and communication for better team collaboration capability. Each agent keeps memory of his own observations from the environment and communication messages from other team members. Exact prompts can be found in the code within the supplementary materials. The design principles are that we only provide general rules about the task environments without explicitly instructing them on any coordination or communication strategy. We attempt to minimize the influence of prompt engineering to ensure the scamless applicability of our approach in diverse environments. In our language grounding data collection and ad- ad-hoc teamwork experiments, we use prrview --r-rviiwe the backbone pre- pre-trained model and set the temperature parameter to 0 to ensure consistent outputs.
OpenAl's API to call gpt-4-0125
Example prompts for LLM agents in the USAR environment are provided below:
Welcome our interactive text game! In this game, you'l Ill assume the role of a specialist on a search and rescue team. Alongside two other players, you 'Il navigate a five-room environment with a mission defuse five hidden bombs.
The Map: Imagine a network of rooms represented by a connected graph where each node corresponds to a room, and the edges between nodes depict hallways. The rooms are numbered 0, 3, 6, 5, and 8. Room 0 is connected to all other rooms. Room shares hallway with room 6. Room 3 is linked room 8, And room 8 is also connected with room 6. You can only travel to adjacent, directly connected rooms at each turn.
The Challenge: Scattered among these rooms are five bombs, each coded with different phases represented by colors. To defuse them, you'll 'll'll need to use the correct wire-cutting tools in the correct sequence. There are one-phase, two-phase, and three-phase bombs, needing I, 2, color-coded tool applications in sequence to disarm. For instance, a bomb with a red-green phase sequence requires the red tool first, then the green one. Points are awarded based on the number of tools used for defusing a bomb, with each tool use worth 10 points. Your task is to maximize the team score as soon as possible. The challenge is that the bomb locations and sequences are unknown to players at the start.
Each player equipped with two color-coded wire cutters. As player Alpha, you have red and green tools, player Bravo wields green and blue, and player Charlie possesses blue and red.
Tools:
Actions: Each round, you can opt to do one of the following: 1) Move to an adjacent room, 2) Inspect bomb's phase sequence in your current room, or 3) Apply your wire cutters to bomb in the current room.
Communications: In addition selecting an action to take from the above list, you can also send communication message texts to both of your teammates in each
round. The message text you sent will be shared with both of your teammates their observation in the next round.
Observation: While you can only see what's in your current room and read text messages from teammates. You'll 'Il also be informed of the current round number, team score and the current location of your teammates. Your teammates have the same observability as you. They will not be able to know your action and its consequences unless you explicitly communicate.
To facilitate our interaction, reply your action selection and communication mes- sages in this fixed format: Action selection: Your action. Message to Team: "Your Message". To move to an adjacent room, say: 'Move to Room X'. To inspect the sequence of a bomb in your current room, say: 'Inspect Bomb'. To apply wire cutter tool, say: 'Apply X Tool'. Remember, your replies must adhere strictly to these rules. Feel free to ask clarifying questions if needed. I'll supply the necessary information progress. Are you ready to take on this explosive challenge?
15

---

Example interactions between LLM agents and environments are provided below:
Env: You are now in Room 3 with Bomb 5. What is your next action?
Alpha: Action selection: Move to Room 5. Message to Team: am moving to Room help Charlie with Bomb 3.
Env: You can not directly move to Room because is not adjacent your current location; Room 3. Consider taking a detour to another room first and then move to your destination.
Example communication among LLM agents in USAR:
Alpha: am inspecting Bomb in Room 0. Bravo; please move to Room 3.
Charlie; please move to Room 5.
Bravo: am moving to Room as instructed. Alpha; please defuse Bomb in Room using the red tool.
Charlie: am moving to Room 5 to search for more bombs.
A.2 LangGround and baseline MARL agents
The implementation of LangGround and baseline MARL methods is based on open-sourced codebase 26 Agents were trained with the same setup and hyperparameters after empirical searching. The batch size is 500, and the number of update iterations in an epoch is 10. Training on pPe0 and USAR takes 2000 epochs and 1e7 timestamps, which takes about 4 hours to complete. Training pPv1 takes 500 epochs and 2.5c6 timestamps, which takes about 1.5 hours to complete. We use learning rate of 0.0001 for USAR and 0.001 for pp. MARL agent's s action policy an L.TTM with hidden layer of size 256. Communication vectors are exchanged one round at each timestamp. The supervised learning weight À is 1 in pp and 10 in USA. VQ-VIB and prototype communication agents discrete tokens to share information during teamwork. All experiments were conducted on a machine with 14-core Intel(R) Core(TM) j9- 1900H CPU and 64GB memory.
provided by the original paper authors 40 16 42 37
allowed to 58 different
B Environment details
Predator Prey
USAR
Room
Hallway
Predator
Prey
Bomb
Agent
Figure 7: Illustrations of the evaluation environments. Predator Prey gridworld environment conceptualizing team of predators with partial observation trying to search for static prey. The task goal is for all predators to reach the prey location within the time limit. USAR simulates a team of specialists searching for and defusing bombs an unknown environment. Because each specialist has the unique capability of defusing bombs in different colors, the team must coordinate to complete the task efficiently.
B.1 Predator Prey
In this task, n predators with limited range of vision v need to search for stationary prey on an by grrd-world environment. Each predator receives positive reward upon reaching the prey location.
16

---

Each episode ends when all predators reach the prey exceed the maximum number of steps T. The initial locations of predators and the prey might spawn anywhere on the map. At each timestamp, the predator agent receives partial observation of by u grids around its own location and may select movement action to navigate through the map. Since this a collaborative task, predators must learn to communicate their partial observations to allow for optimal navigation of the team. We consider Predator Prey to be a more challenging task since has a higher-dimensional observation space and more complex action space [26). Through this environment, we aim to demonstrate that aligning agents' communication with human language a straightforward yet effective method of grounding their communication with task observations.
B.2
USAR
The USAR task environment is designed to simulate the collaborative and problem-solviing dynamies of a search and rescue mission. Three agents (i.e., Alpha, Bravo, and Charlie) need to collaborate in locating and defusing color-coded bombs hidden in an unexplored environment. Each bomb has unique phase sequences in colors, which are not revealed until inspected by agents. Agents start with different colored cutters and must use them in the correct sequence to defuse bombs. The environment is represented as graph, where each of the rooms is a node, and the hallways connecting them are edges. At each timestamp, every agent can choose from three different types of actions: moving to one of the rooms, inspecting bomb's sequence in the current room, or utilizing one of the m wire-cutters. The size of the action space depends on the problem scale (i.e., + m + 1). Agents' observations are limited to their current room's 's contents and agent status. The team is rewarded 10 points when an z z-phase bomb is successfully defused. An episode ends when the team has defused all bombs or exceeded the time limit. This task is designed to force team coordination since each team member has unique observations and capabilities. For example, each agent only has subset of wire cutters and must coordinate with other teammates defuse bombs with multiple phases. Therefore, an effective communication protocol is required for efficient information sharing and team synchronization.
B.3 Environment configurations
For the predator and prey environment, we use map size of 5 by 5 with predators and prey. The predator's ' range of vision is manipulated to be either 0 or 1 to create two variants of the task environment. In the situation vision O. predators cannot observe the prey until they jump onto the same location. We set the maximum episode length to 20 based on previous research [17]. The USAR environment comprises five rooms (n = 5) and five bombs, including two single-phase two double- -hhase, and one triple-phase bombs. The bomb phase might have three different colors (m = 3). Each of the 3 agents spawns with 2 different wire cutters, forcing the team to collaborate in defusing bombs with multiple phases. Each successfully defused bomb awards the team 10 points per processed phase, resulting in 90 as the maximum score per mission. We set the maximum episode length 100 based on previous research 25
B.4
Text game interface
The initial task environments of Predator Prey and USAR are implemented for MARL agents based on Gym API 33 To facilitate interaction between LLM-based agents with the environment, we implement a rule-based text interface for each task. At each timestamp, LLM agents sequentially interact with the environment, receiving observations and performing actions via natural language interaction. Additionally, they are allowed to broadcast communication messages in natural language which are appended with the observation text and sent to all team members in the next round. It is worth noting that LLM agents receive equivalent information as MARL agents, limited to individual agent's 's partial observation.
The text interface facilitates communication between the game engine and the language model agents by converting game state observations into natural language descriptions and mapping agent responses back to valid game actions. To generate observations, the interface extracts relevant state features from the game engine, such as the current round number, cumulative team score, action feedback, visible objects, and communication messages from other agents. It then populates predefined sentence templates with these extracted features to produce a structured natural language
17

---

description of the current game state. Action encoding relies on keyword matching, as the language models are instructed format their responses using specific keywords and structures. The interface scans the agent's s response for these predefined keywords and maps them to corresponding game actions. In cases where an agent's 's response invalid or ambiguous, such as attempting to perform an action in an incorrect location, the interface generates informative error message based on predefined rules and templates. For instance, if an agent attempls to inspect a non-existent bomb, the interface might respond with the following error message: "There is no bomb in the current location, Room X, for you to inspect." This targeted feedback helps the agents refine their actions to comply with the game's 's rules and constraints.
C Data collection details
C.1 LangGround dataset
In order to construct dataset D, we collected expert trajectories from embodied LLM agents powered by GPT-4 in interactive task scenarios. As shown in Table 4 teams consisting of pure LLM agents perform reasonably well comparison to MARL methods. Therefore, we believe their action and communication policy can be used in guiding MARL agents. In USA AR, we collected 50 episodes resulting in 2550 pairs of (observation, action) and communication messages of individual agents. The number of data pairs is 1893 for pp-0 and 2493 for ppe1, respectively. To facilitate the alignment of agent communication space and natural language, we use OpenAI's word embedding api (i.c., -e-eeebbbddnnn---aargee to translate each natural language message into high-dimensional vector with the same dimension (i.e., D = 256) as agent communication vectors.
C.2 Ad-hoc teamwork
Due to the restrictions of resources and time, behaviors in human-agent teams. We match 2 MARL agents with unseen LLM agent a team and ask them complete the collaborative task in predator-prey and USAR environments. The LLM agent is powered by GPT-4-turbo and prompted to output action selection and communication messages given observation inputs, similar the data collection process introduced in Section4] The Gym environment is wrapped with a text interface to decode observations into English descriptions and encode LLM agent's output into concrete action selection. Both MARL agents and LLM agents interact with the same task environment in sequence. Natural language communication messages from LLMs are embedded using OpenAl's word embedding API and sent to MARL agents. The communication vectors from MARL agents are translated to English sentences via cosine similarity matching dataset D.
embodied LLM agents emulate human
D Additional Experiment Details
D.1
Topographic similarity
The topographic similarity is defined by the correlation between object distances the observation space and their associated signal distances in the communication space [2]. This property is usually associated with language compositionality and ease of generalization. The intuition behind this measure is that agents should emit similar communication messages given semantically similar observations. We calculate this measure following (20). based on agent trajectories collected from 100 evaluation episodes in pPoo- We first calculate 1) the cosine similarity between all pairs of communication vectors, and 2) the Euclidean distance between all pairs of agent locations. Then, we calculate the negative Spearman correlation p as the measure of topographic similarity. Table 3 indicates that our method (i.e., LangGround) results in the highest topographic similarity p 0.67 among all other baselines, exhibiting relatively more similar property as human language.
D.2 Additional ablation study
Here we run additional ablation study to attribute reinforcement learning signal and supervised learning signal to the agent's 's action and communication output the multi-ohjective optimization problem.
18

---

Methods
Topo Sim p
LangGround
0.67=0.07 0.54
IC3Net
aeComm
0.37+10.05
protoComm
0.35+
Table 5: Topographic similarity in PPo0
Because the LangGround agent trained end-to-end with combination of RL. and SL loss and uses the intermediate hidden state of its policy as the communication vector, it is very hard to separate the reasoning processes action and communication. However, we could provide indirect evidence to prove that RL and SL jointly contribute to both the agent's action and communication.
The MARL-comm agent uses gating function learn whether communicate at specific times- tamps. We could ablate this function to see its impact on team performance. As shown in the Fig. g removing the gating function harms the performance of LangGround more than IC3Net. This means both RL and SL signals influence the content and timing of LangGround communication.
In addition, could manipulate the weight of supervised learning loss, i.e. À in function 4 to illustrate the contribution of RI. and SI. signals. As shown in Fig. and [T0 in the PDF, À matters for both task performance and supervised loss. If the SL. loss weighted too high, the Lang agent cannot optimize its policy in completing the task. While if the RL loss is weighted too high, the LangGround agent cannot align its communication with human language. This result aligns with our claim that RL optimizes the communication for task utility and SL optimizes the communication for alignment.
USAR
USAR
US4R
120
F0
60-
402
LengCrrund
LangCrnund reGoting
Thaninng Tirnalopa
Traniigg Timesnaca
Tarinn Timastaps
Figure 8: Ablation study of Figure 9: Impact of supervised gating function and language grounding.
Figure 10: Impact of SL loss weight communication alignment.
learning loss weight on team performance.
19

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
We make reasonable claims based on the empirical results.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how be expected to generalize other settings.
much the results
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations of our work in the discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
20

---

Justification:
This paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We describe our proposed pipeline clearly in Section
4 In the Appendix, provide implementation details MARL agents and reproducibility details (e.g. model API, prompts, parameters) for LLM agents.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
21

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: We submit code in the supplementary materials for the review purpose. We will make the code public available upon paper acceptance, following the NeurlPS code and data submission guidelines.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.co cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
y /nips. /nips.cc//ppppp http://cccccccguuddeeeeeee/ee//e htabll/c/guideesodeeuuubiiis
for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/ettails
Question:
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
We provide those details in both the main text and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We provide error bars over evaluation episodes and random seeds in the main text for all tables and figures.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
22

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We reveal the computational resource used for experiments the Appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: No Code of Ethics are violated.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the potential societal impact of our work in the discussion section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
23

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This paper does not release any data models that have high risk for misuse.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We cite the original papers that produced the open-source code for baseline methods.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
24

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We provide detailed instructions and documentations with the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
25

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "t8sIMMPs6E", "reviewer": "Reviewer_NCiL", "summary": "The paper presents a method of using LLMs with synthetic data to generate human interpretable multi-agent communication protocols using zero-shot learning. The model, called language-grounded multi-agent communication (LangGround) aligns the communication of the multi-agent model with the LLM based communication.\n\nThe model is tested on 2 versions of the Predator-Prey and 1 version of the USAR environments against multiple baselines, showing that the model presented in the paper performs comparable or better than state-of-the-art models in Multi-Agent Communication. Moreover, the paper shows that the communication protocol is generalizable and allows for Human interpretation and the ability to work in teams with LLM-based agents.", "questions": "How is this model more advantageous than a pure LLM based environment as presented in [24]?\n\nHow open is this model to hallucinations from the LLM model?\n\nHow does the performance of the LLM Model used affect the performance of the trained LangGround Model?", "limitations": "Prompt engineering is required to allow for the model to be integrated into new environments.\n\nThe model that is trained depends on the performance of the Language Model.\n\nThe models are shown to trade off utility and informativeness, causing a decrease in performance for interpretability.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The model is novel and experiments show that it outperforms other state of the art baselines.\n\nThe related works is of sufficient length for a paper of given length.\n\nThe approach is clear and the figures are easy to understand.\n\nTesting against multiple MARL Environments and configurations provide a convincing argument.\n\nThe shared supplementary materials allow for easy reproducibility.", "weaknesses": "There could be a Human Subjects experiment to show that the model that is presented is actually preferred by humans over alternative models.\n\nThe experiments could be expanded to show the performance of the model at different scales.\n\nThe differences between the model presented in the paper and [24] can be highlighted more. While section 4.1 goes into detail that the Language model used is derived from the previous work, it may be better to stress the primary differences given that both models are tested on the USAR environment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the novelty and soundness of LangGround, our proposed method for MARL agents with human-interpretable communication. We appreciate your constructive comments and suggestions and seek to clarify them with the following responses and additional experiment results.\n\n > Q1: Human Subjects experiment\n\nWe acknowledge the necessity of evaluating our method with real human participants given the ultimate goal of learning human-interpretable communication. However, we decide to leave human subject experiments for future work based on the following considerations:\n\n - We want to focus the scope of this paper on the technical details of the proposed pipeline and evaluation of aligned communication space. Human experiment results might be influenced by many factors (e.g. demographics, background, attitude to AI) therefore need more considerations.\n - Many works have proved the effectiveness of LLMs in simulating human behaviors and replicating data collection from human participants [1]. This is our original motivation for using LLMs to generate synthetic data for LangGround. It is a reasonable first step to conduct a sanity check by evaluating LangGround in ad-hoc teamwork with embodied LLM agents, before directly testing it with human subjects.\n - Our method is the only model that learns an aligned communication space with natural language, facilitating direct conversation with humans. Alternative models either do not have any interpretability (e.g. IC3Net, aeComm), or only learn a semantically meaningful space independent from human language (e.g. protoComm, VQ-VIB). In the ad-hoc teamwork experiments, we actually give baseline models an advantage by allowing them to use the offline dataset of LangGround and OpenAI's word embedding model. Otherwise, no alternative models are able to communicate with embodied LLM agents or humans in natural language.\n\n\n > Q2: Evaluate LangGround in scaled environments\n\nPlease refer to our answer to Q1 in the overall rebutall.\n\n > Q3: What is the primary difference between LangGround and [2]\n\nThank you for pointing out potential directions for improving the presentation of our work. We will add more details to explain the method used in previous work [2] and compare it with LangGround.\n\nEssentially, [2] proposes a prompt engineering technique to improve the performance of embodied LLM agents in collaborative tasks, by allowing them to keep track of key task-related information. They evaluated this method in USAR environment and proved its effectiveness. While in this paper we propose a pipeline to improve the robustness and interpretability of emergent communication learned by Multi-Agent Reinforcement Learning agents. Therefore, the motivations and technical approaches of [2] and LangGround are inherently different, although both result in artificial agents capable of collaborating and communicating with humans.\n\n > Q4: Compare MARL-comm agents with embodied LLM agents in general\n\nRegarding this question, we provide the argument in two-fold:\n\n - MARL and LLM are trained with different learning objectives: MARL is optimized for maximizing the expected reward from environment, while LLMs are optimized to predict the next word given the context. Embodied LLM agents may generate infeasible action plans in interactive tasks due to hallucinations and bias from training data. Consequently, MARL agents are generally more efficient in completing tasks than pre-trained LLM agents [2].\n - LLMs have a large size of parameters thus take longer to run. In our case, calling OpenAI's API is ~50 times slower than running local inference of MARL models, and costs ~5 dollars for each eposide. Taking the massive time and resources required to train LLMs into consideration, MARL models are more affordable and eco-friendly in solving specific tasks.\n\n\n\n > Q5: Impact of LLM's hallucination and performance on LangGround\n\nPlease refer to our answer to Q2 in the overall rebutall.\n\n > Q6: Prompt engineering is required for generalization to new environments\n\nIn practice, yes. But the amount of effort required for generalization is much smaller when compared to alternative methods in either embodied LLM agents or MARL-comm literature. The rationale is similar to the argument we made for Q5, because LangGround only uses effective communication messages from LLMs, a mere amount of prompt engineering is required compared to alternative methods that require LLMs to generate actionable plans for each step. Learning communication protocol from scratch is known to be challenging in the MARL community. Our method stabilizes the learning process and reduces the amount of engineering work (e.g. hyper-parameter searching) when generalizing to new domains, by introducing language priors from general knowledge base, LLMs.\n\n\nWe sincerely appreciate the reviewer's time and effort in helping improve our paper. We hope that our responses have adequately addressed your concerns and clarified the contributions of our work. Thank you for your valuable feedback and consideration.\n\n[1] Using large language models to simulate multiple humans and replicate human subject studies. ICML 2023\n\n[2] Theory of Mind for Multi-Agent Collaboration via Large Language Models. EMNLP 2023", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eLDITyRMoZ", "reviewer": "Reviewer_hS12", "summary": "The paper presents a novel computational pipeline aimed at aligning the communication space of Multi-Agent Reinforcement Learning (MARL) agents with an embedding space of human natural language. The authors propose grounding agent communications on synthetic data generated by embodied Large Language Models (LLMs) in interactive teamwork scenarios.", "questions": "See weakness", "limitations": "See weakness", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The use of synthetic data generated by LLMs to align MARL agents' communication is a creative application of existing technologies in a new way, demonstrating originality in methodology.\n- While the paper does not present new theoretical results, it does provide a solid empirical foundation for its approach, which is well-supported by experiments.\n- The proposed computational pipeline appears technically sound, with a clear explanation of how it aligns with human language and the rationale behind the design choices.\n- The authors provide clear explanations of complex concepts, such as the alignment of communication spaces and the grounding process, making the paper accessible to a broader audience.", "weaknesses": "- The paper does not provide a theoretical framework or proofs to support the empirical findings. Developing a theoretical basis could strengthen the claims and provide deeper insights into why the approach works.\n- The experiments are conducted in controlled environments. To strengthen the claims, testing the approach in more diverse and complex scenarios could provide evidence of broader applicability.\n- The paper relies heavily on synthetic data generated by LLMs. There might be concerns about the representativeness of this data for real-world scenarios.\n- Some details regarding the implementation of the MARL agents and the interaction with LLMs could be better elaborated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for acknowledging the originality and soundness of LangGround, our proposed method for MARL agents with human-interpretable communication. We seek to clarify your questions with the following responses and additional experimental results.\n\n > Q1: No theoretical proof\n\n\nWe do not provide a proof for our work because the theories we used in deriving the proposed method are from previous literature in emergent communication and MARL-comm. However, we indeed provided analysis to explain why our method works in practice in the discussion section.\n\nThe core idea of LangGround is providing an auxiliary loss to directly regulate intermediate hidden layers of the RL agent's policy network. This technique has been proven effective in stabilizing the learning process via additional supervision [1]. In MARL-comm, previous work has used representation learning methods (e.g., autoencoder [2], contrastive learning [3]) to construct communication messages solely based on encoding observations. Our pipeline combines these two techniques by using language priors to regulate RL agent's communication via supervised learning. Our dataset $\\mathcal{D}$ consists of expert trajectories from LLM embodied agents with a well-established grounding on the task. We believe our method helps agent learn a semantic representation of task-related information grounded on word embeddings of example messages.\n\n\nThe other relevant theory is the Information Bottleneck Theory. In emergent communication, a speaker and a listener jointly learn a communication protocol to maximize the task reward. But this may compromise other properties of the learned language such as task-agnostic informativeness and human interpretability [4]. To overcome this issue, we align the agent communication space with the semantic space of human language by grounding them on a small set of data, and add the supervised learning loss to the task loss [5]. This explain the trade-off between utility and informativeness we observe across task environments, representing different solutions of the multi-objective optimization problem's Pareto front.\n\n > Q2: Evaluate LangGround in scaled environments\n\nPlease refer to our answer to Q1 in the overall rebutall.\n\n > Q3: LangGround relies on LLMs' performance\n\nPlease refer to our answer to Q2 in the overall rebutall.\n\n > Q4: Details about MARL implementation and ad-hoc teamwork are missing\n\n\nWe apologize for the confusion. Descriptions about ad-hoc teamwork experiments are presented in Section 6.3.2 and Appendix A.3.2 of the original paper. Implementation details of MARL agents are presented in Section 4.2 and Appendix A.1.2. We provide additional descriptions and a re-made framework illustration in Figure D in the PDF. We will add them to the final paper.\n\nAd-hoc teamwork refers to situations where agents collaborate with unseen teammates without pre-coordination. In this work, we use embodied LLM agents to emulate human behaviors in human-agent ad-hoc teams. We match 2 MARL agents with 1 unseen LLM agent in a team and ask them to complete the collaborative task in predator-prey and USAR environments. The LLM agent is powered by GPT-4-turbo and prompted to output action selection and communication messages given observation inputs, similar to the data collection process introduced in Section 4.1. The Gym environment is wrapped with a text interface to decode observations into English descriptions and encode LLM agent's output into concrete action selection. Both MARL agents and LLM agents interact with the same task environment in sequence. Natural language communication messages from LLMs are embedded using OpenAI's word embedding API and sent to MARL agents. The communication vectors from MARL agents are translated to English sentences via cosine similarity matching in dataset $\\mathcal{D}$. Their team performance is measured by the number of steps spent in completing the task.\n\nWe sincerely appreciate the reviewer's time and effort in helping improve our paper. We hope that our responses have adequately addressed your concerns and clarified the contributions of our work. Thank you for your valuable feedback and consideration.\n\n[1] Lee, C. Y., Xie, S., Gallagher, P., Zhang, Z., & Tu, Z. (2015, February). Deeply-supervised nets. In Artificial intelligence and statistics (pp. 562-570). Pmlr.\n\n[2] Lin, T., Huh, J., Stauffer, C., Lim, S. N., & Isola, P. (2021). Learning to ground multi-agent communication with autoencoders. Advances in Neural Information Processing Systems, 34, 15230-15242.\n\n[3] Lo, Y. L., & Sengupta, B. (2022). Learning to ground decentralized multi-agent communication with contrastive learning. arXiv preprint arXiv:2203.03344.\n\n[4] Tucker, M., Levy, R., Shah, J. A., & Zaslavsky, N. (2022). Trading off utility, informativeness, and complexity in emergent communication. Advances in neural information processing systems, 35, 22214-22228.\n\n[5] Tucker, M., Li, H., Agrawal, S., Hughes, D., Sycara, K., Lewis, M., & Shah, J. A. (2021). Emergent discrete communication in semantic spaces. Advances in Neural Information Processing Systems, 34, 10574-10586.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8jdAojRVdX", "reviewer": "Reviewer_YGBb", "summary": "This paper proposes a pipeline for aligning multi-agent communication with natural language by using an LLM to collect a synthetic dataset of natural language interactions. Within a MARL communication pipeline, this dataset is used to align the learned communication space with word embeddings of reference communications. This pipeline enables human interpretable messages and ad-hoc collaboration across multiple communication-intensive MARL environments.", "questions": "- The BLEU score in line 267 doesn't make sense to me. It is comparing the score between the natural language messages and reference messages in D with the most similar word embedding, but in the appendix (lines 653-654) it states the communication vectors are translated to English sentences via cosine similarity matching. In that sense, shouldn't the BLEU scores be perfect?\n- How does GPT-4 compare against your technique (and baseline emergent MARL communication) in the studied settings?", "limitations": "The limitations mentioned in the discussion are solid. I would additionally include how this technique depends on the notion that the grounding LLM \"knows\" what message is optimal.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The research direction proposed in this work is important since it connects the literature of emergent communication with natural language.\n- The paper is well written and easy to follow.\n- The proposed method is easily generalizable to new settings and is usable with different communication backbones and LLMs.", "weaknesses": "- This work is missing a discussion comparing the proposed technique to the prior work of CICERO[1] from 2022. CICERO used a dataset of human messages and actions in real games to build a task-specific language model and trained a policy network that communicates by sharing embeddings that are translated through the LLM. This work has many components in common with CICERO and needs to discuss its novelty (beyond replacing human data with GPT-4 data).\n- The supervised loss seems to imply that the communication protocol is just a distillation of GPT-4. Although the RL objective could theoretically help determine what information is useful to share, it seems more likely that RL would just learn to optimize actions while leaving the communication protocol to the supervised loss. Even if the RL objective does influence the communication protocol, the only alternative messages it could generate are messages corresponding to different (observation, action) pairs, so I doubt the claim in line 175 that RL determines useful information to share with other agents.\n  - That being said, I would agree with the notion that RL learns to \"interpret\" messages from other agents.\n- Section 6.3.1 seems misleading to me. Specifically, the way that communication vectors are translated to English sentences are via cosine similarity matching to the most similar message the dataset (according to lines 653-654 in the appendix). Therefore, the claim that communication messages are similar to natural language sentences in unseen locations (lines 285 to 287) must be true by construction.\n\n[1] FAIR et al. Human-level play in the game of diplomacy by combining language models with strategic reasoning. Science, 378(6624):1067–1074, 2022. doi: 10.1126/science.ade9097.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating the motivation and generalizability of our work, and we seek to clarify the framing of our research.\n\n > Q1: Missing comparison with CICERO\n\nThank you for pointing out this paper. We indeed came across it but decided not to include it in the comparison because the motivation and technical approach of CICERO are inherently different from our work. CICERO has an RL module for strategic reasoning and a language model for generating messages. The two modules are trained separately on different datasets, namely self-play trajectories and conversation data collected from human players. During inference, the strategic reasoning model outputs actions and intent estimations conditioned on game state and dialogue history, while the language model generates messages based on game state, dialogue history, and intent estimation. It is clear that the two modules in CICERO function independently, with the only connection being that the language model takes intention estimation from the planning module as input.\n\nHowever, in our work, both action and communication are generated by individual RL agents that are trained end-to-end with a combination of RL loss and supervised learning loss. As shown in Fig. 1, each LangGround agent is controlled by an LSTM policy, and its hidden state $h_t^i$ is used to generate vector $c_t^i$ for inter-agent communication. The gated average communication vector is taken by each agent's policy to output the action. We do not use an LLM to generate communication in our pipeline. Instead, a dataset of example messages is used to calculate the supervised learning loss.\n\nAlthough CICERO and LangGround both result in agents capable of communicating with humans in competitive/collaborative tasks, their major contributions are fundamentally different. CICERO focuses on empirically evaluating the proposed hierarchical framework in real-world scenarios, while our work focuses on proving the concept of aligning MARL agents' emergent communication space with the semantic space of human language. We hope our explanations address your concern.\n\n > Q2: Does RL determine useful information to share or not?\n\nPlease refer to our answer to Q3 in the overall rebuttal.\n\n\n > Q3: Concerns about Cosine similarity and BLUE socre\n\nWe apologize for the confusion and seek to clarify the design of the zero-shot generalization experiment in the context of the previous argument about our model structure.\n\nIn Section 6.3.1, we remove a subset of prey spawn locations during training and evaluate LangGround agents with those unseen states in a zero-shot manner. To accurately retrieve unseen English sentences from the dataset, the agent needs to generalize their communication vector in the learned semantic space in a similar way as in the word embedding space. \n\nFor example, in a hypothetical scenario, the agent was only trained on prey locations (0,0) and (2,2) but needs to communicate about (1,1) during evaluation. It must generate an interpolated vector that lies between vectors referring to known locations in the high-dimensional communication space. Because this space is aligned with the word-embedding space, the interpolated vector is likely to be close to the corresponding unseen messages about (1,1) in the dataset. In practice, the generalization might not be done via linear interpolation, depending on the properties of the aligned high-dimensional space.\n\nThe above-mentioned process is the prerequisite for achieving high cosine similarity and BLEU scores, as well as retrieving unseen sentences correctly in zero-shot generalization. It is clear that this process is not guaranteed 'by construction', but rather depends on the alignment between the agent communication space and word embedding space of human language. As mentioned earlier, the LangGround agent is solving a multi-objective optimization problem with a single LSTM model. The agent policy must trade off completing the task and aligning communication, and neither of them can be solved easily. Therefore, we design different criterias to evaluate the agents in both task performance and communication alignment respectively.\n\nWe also present additional zero-shot generalization experiment results in the overall rebuttal. Please refer to our answer to Q4 for more information.\n\n > Q4: Compare GPT-4 with LangGround\n\nThe comparison results are presented in Table 4 in the original paper. The LLM row corresponds to the team performance of 3 embodied agents powered by GPT-4-turbo. As analyzed in Section 6.4.2, LLM-only teams perform worse than MARL-comm teams, and better than ad-hoc teams.\n\n > Q5: LangGround relies on LLMs' performance\n\nPlease refer to our answer to Q2 in the overall rebutall.\n\n\nWe sincerely appreciate the reviewer's time and effort in helping improve our paper. We hope that our responses have adequately addressed your concerns and clarified the contributions of our work. Thank you for your valuable feedback and consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KGiD3UslD7", "reviewer": "Reviewer_oeK1", "summary": "The paper introduces a framework for communication in multi-agent RL. The communication space is aligned with the embedding space of natural language. The loss for training the underlying RL algorithm is augmented with a cosine similarity loss to align the communication message generated to the embeddings generated from LLMs.", "questions": "Refer to the weaknesses section for my questions.", "limitations": "The experimental evaluation in some of the experiments Tables 3 and 4 (refer weaknesses section above) are not extensive. Also, the procedure to obtain the results in Table 4 is not clear.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The approach is quite flexible in the sense that it can be used with any standard RL algorithm\n- The loss function is quite simple but effective.\n- The proposed method shows better performance in task-related metrics and cosine similarity/BLEU scores compared to other baselines.", "weaknesses": "- Figure 1 could have more details about the architecture in the caption. There is a reference to the figure on line 135 where it states that more information about the architecture can be found in Fig 1. But the caption and the figure were not clear and I had to read more about it in the text.\n- The experiments related to the zero-shot generalizability of the model are not convincing enough. I believe there isn’t much of a difference between $pp_{v0}$ and $pp_{v1}$ other than the vision range of the agents. Also, I am not sure if Table 3 just shows 4 instances of coming across different observations/locations in the environment. I am curious to see how this would perform in 10x10 test environments when they are trained on 5x5 grid environments. Will the communication messages be able to handle these new coordinates like (9,9), (7,8) which were previously unseen in the training scenarios?\n- I was not able to understand the exact setting used for the ad-hoc teamwork experiment. In what manner were the LLM agents controlled? How was the communication message decoded from LangGround to natural language? Was there a reason for evaluating the ad-hoc teamwork on just 24 episodes in Table 4?\n- The environments in which the model was tested are simple grid-world environments. The usage of LLMs to generate embedding data motivates testing it out in more complex environments (I think this has been pointed out in the future work section.).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for appreciating our work, and we seek to clarify our experiment design with the following responses.\n\n > Q1: Figure 1 can be better presented\n\nYour suggestion is well received. Thank you for pointing it out. We have remade the framework illustration in Figure D in the PDF and will add it to the final paper.\n\n\n> Q2: Zero-shot generalization experiment\n\nWe apologize for the confusion. We seek to clarify the design of the zero-shot generalization experiment to address your concern.\n\nIt's worth noting that there are two levels of zero-shot generalization in our work. The first level is communication alignment, where LangGround agents generate human-interpretable communication in task states they have never been grounded on during training. The second level is task completion, where LangGround agents are trained on environment A and are still able to complete the task in unseen environment B.\n\nIn Section 6.3.1, we remove a subset of prey spawn locations (i.e., 4 out of 25 cells) during training and evaluate LangGround agents with those unseen states in a zero-shot manner. This experiment requires both task-level and communication-level generalizability because those removed states are 1) unseen during training, and 2) not grounded with human language. Table 3 shows the inclusive list of 4 unseen states and the measurements of communication alignment with unseen ground truth. This is equivalent to your suggested experiment on a smaller scale, and we indeed found that \"the communication messages can handle new coordinates, e.g., (1,1), (1,3), (3,1), (3,3), which were previously unseen in the training scenarios\".\n\nWe acknowledge your suggestion about training on a 5x5 gridworld and testing on a 10x10 gridworld. However, such task-level generalization depends more on the backbone RL model structure and specific environment design, which is out of the scope of this work. Therefore, we present an alternative experiment focusing on communication-level generalization. We train LangGround agents on a 10x10 gridworld but only provide language grounding in a subset of states. The learning curves of LangGround agents with different levels of language grounding are presented in Fig. B and Fig. C in the PDF file. As shown in the figures, the more grounded states, the better the team performance, as well as the better the communication alignment. In addition, Table A in the PDF shows the results of communication alignment measurements in un-grounded states. Similarly, the more grounded states, the better alignment we observe in communication-level zero-shot generalization.\n\n\nTo summarize, the communication-level generalizability of LangGround stems from the assumption that we could ground the agent communication space with the word embedding space of human language on a limited number of instances, and expect LangGround to output interpretable messages in un-grounded states via topographic similarity of the aligned space. In practice, this assumption depends on many factors such as the number of grounded states, dimension of the communication space, scale of the problem, etc. The above experiment illustrates the impact of language grounding percentage (25%, 50%, 75%, 100%), and we leave further investigation to future work.\n\n\n\n> Q3: Ad-hoc teamwork experiment\n\nWe apologize for the confusion. Brief descriptions about ad-hoc teamwork experiments are presented in Section 6.3.2 and Appendix A.3.2 of the original paper. We will add more details to the revised paper.\n\nAd-hoc teamwork refers to situations where agents collaborate with unseen teammates without pre-coordination. In this work, we use embodied LLM agents to emulate human behaviors in human-agent ad-hoc teams. We match 2 MARL agents with 1 unseen LLM agent in a team and ask them to complete the collaborative task in predator-prey and USAR environments. The LLM agent is powered by GPT-4-turbo and prompted to output action selection and communication messages given observation inputs, similar to the data collection process introduced in Section 4.1. The Gym environment is wrapped with a text interface to decode observations into English descriptions and encode the LLM agent's output into concrete action selection. Both MARL agents and LLM agents interact with the same task environment in sequence. Natural language communication messages from LLMs are embedded using OpenAI's word embedding API and sent to MARL agents. The communication vectors from MARL agents are translated to English sentences via cosine similarity matching in dataset $\\mathcal{D}$. Their team performance is measured by the number of steps spent in completing the task.\n\nThe main reason for only evaluating each ad-hoc team composition on 24 episodes is the cost of calling OpenAI's API for LLM agents. In our case, each agent consumes 2k - 5k tokens per round, and each episode takes 20 - 50 rounds to complete. Evaluating 5 team compositions on 3 environments over 24 episodes costs around \\$500. We plan to conduct more comprehensive evaluations in the future with different backbone models and human participants.\n\n > Q4: Realistic environments\n\nYour suggestion is well received. As for future work, we plan to extend LangGround into more realistic task environments such as ALFWorld, RoCoBench, and ThreeDWorld Transport Challenge, where agents must physically interact with real-world objects based on visual and text inputs. Our proposed pipeline is agnostic to environments and backbone LLM and RL models, therefore it can be easily generalized to those scenarios.\n\nWe sincerely appreciate the reviewer's time and effort in helping improve our paper. We hope that our responses have adequately addressed your concerns and clarified the contributions of our work. Thank you for your valuable feedback and consideration.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank all the reviewers for their time and effort in helping us improve the paper. We appreciate your acknowledgment of the novelty and soundness of LangGround, our proposed method for MARL agents with human-interpretable communication. In this rebuttal, we seek to clarify a few common questions and provide additional experiment results upon request.


 > Q1: Evaluate LangGround in scaled environments

Upon your request, we ran additional experiments in the Predator Prey environment with a larger map size (10 by 10). The learning curves of LangGround and baselines are presented in Fig A in the rebuttal PDF file. As shown in the figure, our method outperforms ablating baselines without language grounding (i.e., IC3Net) or without communication (i.e., noComm). This result demonstrates the benefit of introducing LangGround in stabilizing the learning process of emergent communication of MARL agents in scaled environments.

As for future work, we plan to extend LangGround into more realistic task environments such as ALFWorld, RoCoBench, and ThreeDWorld Transport Challenge, where agents must physically interact with real-world objects based on visual and text inputs. Our proposed pipeline is agnostic to environments, LLMs, and MARL-comm model structure; therefore, it can be easily generalized to those scenarios.


 > Q2: LangGround relies on LLMs' performance

Essentially, LLM's hallucination and task performance do not directly affect LangGround's performance. We use embodied LLM agents to collect grounding communication datasets for LangGround as explained in Section 4.1. These datasets can also come from rule-based agents or human participants, as long as they show effective communication in solving collaborative tasks. Since only communication messages are used, LLM agents' task performance does not impact MARL's performance. For example, based on our observations during experiments in the predator-prey environment, due to hallucination LLM agents may incorrectly interpret the task as encircling the prey instead of reaching it, stopping one step away and saying, "I found the prey at (x, y), ready to encircle it." Despite not completing the task, their communication remains valid in reporting the prey location. During training, MARL agents optimize their policy for reaching the prey and aligning communication messages, so LLM agents' sub-optimal behaviors are not learned. The only minor issue is that MARL might learn to use the word 'encircle,' which could cause slight issues in ad-hoc teamwork with humans.

> Q3: Does RL determine useful information to share or not?

This is a very good question targeting the core mechanism of LangGround, attributing reinforcement learning signal and supervised learning signal to the agent's action and communication output in the multi-objective optimization problem. Thank you for bringing this up.

Because the LangGround agent is trained end-to-end with a combination of RL and SL loss and uses the intermediate hidden state of its policy as the communication vector, it is very hard to separate the reasoning processes of action and communication. However, we could provide indirect evidence to prove that RL and SL jointly contribute to both the agent's action and communication.

 - The MARL-comm agent uses a gating function to learn whether to communicate at specific timestamps. We could ablate this function to see its impact on team performance. As shown in the following table and Fig. E in the PDF, removing the gating function harms the performance of LangGround more than IC3Net. This means both RL and SL signals influence the content and timing of LangGround communication.

| Model | Performance loss after removing the gating function |
| ----------- | ----------- |
| IC3Net | -9.05 |
| LangGround | -34.09 |

 - In addition, we could manipulate the weight of supervised learning loss, i.e. $\lambda$ in function (4), to illustrate the contribution of RL and SL signals. As shown in Fig. F and G in the PDF, $\lambda$ matters for both task performance and supervised loss. If the SL loss is weighted too high, the LangGround agent cannot optimize its policy in completing the task. While if the RL loss is weighted too high, the LangGround agent cannot align its communication with human language. This result aligns with our claim that RL optimizes the communication for task utility and SL optimizes the communication for alignment.


> Q4: Zero-shot generalization experiment

We present additional experiments to verify the zero-shot generalization capability of LangGround. Specificlly, we train LangGround agents on 10x10 Predator Prey but only provide language grounding in a subset of states (i.e., 25%, 50%, 75%, 100%). The learning curves of LangGround agents with different levels of language grounding are presented in Fig. B and Fig. C in the PDF file. As shown in the figures, the more grounded states, the better the team performance, as well as the better the communication alignment. In addition, Table A in the PDF shows the results of communication alignment in un-grounded states during evaluation. Similarly, the more grounded states, the better alignment we observe in communication-level zero-shot generalization.

To summarize, the generalizability of LangGround stems from the assumption that we could ground the agent communication space with the word embedding space of human language on a limited number of instances, and expect LangGround to output interpretable messages in un-grounded states via topographic similarity of the aligned space. In practice, this assumption depends on many factors such as the number of grounded states, dimension of the communication space, scale of the problem, etc. The above experiment illustrates the impact of language grounding percentage, and we leave further investigation to future work.

Meta-review: This paper studies the problem on Multi-Agent Reinforcement Learning in the context of embodied Large Language Models (LLMs). The paper investigates the communication patterns of LLMs vis-a-vis human natural language, finding positive results for their technical approach by distilling and aligning grounded agent communication within an embedding space of human natural language.

Overall, the reviewers were mostly positive with only one borderline review. Multiple reviewers were positively impressed with the rebuttal and raised their scores. Concerns about the paper's weaknesses include the need for the LLM agents to be able to generate good messages in order to create a good dataset; further, it may be impossible to generate "novel" messages with the current approach. Nonetheless, the reviewers agreed that the paper's greatest strength was its novelty: it can make a contribution towards MARL and folks modeling natural language.
Decision: Accept (poster)