Entity Divider with Language Grounding in Multi-Agent Reinforcement Learning
Authors: Ziluo Ding
Wanpeng Zhang
Junpeng Yue
Xiangjun Wang
Tiejun Huang
Zongqing Lu
Venue: Submitted to ICLR 2023 (2023)
Keywords: language-based reinforcement learning
multi-agent reinforcement learning

Abstract: We investigate the use of natural language to drive the generalization of policies in multi-agent settings. Unlike single-agent settings, the generalization of policies should also consider the influence of other agents. Besides, with the increasing number of entities in multi-agent settings, more agent-entity interactions are needed for language grounding, and the enormous search space could impede the learning process. Moreover, given a simple general instruction, e.g., beating all enemies, agents are required to decompose it into multiple subgoals and figure out the right one to focus on. Inspired by previous work, we try to address these issues at the entity level and propose a novel framework for language grounding in multi-agent reinforcement learning, entity divider (EnDi). EnDi enables agents to independently learn subgoal division at the entity level and act in the environment based on the associated entities. The subgoal division is regularized by opponent modeling to avoid subgoal conflicts and promote coordinated strategies. Empirically, EnDi demonstrates the strong generalization ability to unseen games with new dynamics and expresses the superiority over existing methods. 

Under review as a conference paper at ICLR 2023
ENTITY DIVIDER WITH LANGUAGE GROUNDING IN MULTI-AGENT REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
We investigate the use of natural language to drive the generalization of policies in multi-agent settings. Unlike single-agent settings, the generalization of policies should also consider the influence of other agents. Besides, with the increas- ing number of entities in multi-agent settings, more agent-entity interactions are needed for language grounding, and the enormous search space could impede the learning process. Moreover, given a simple general instruction, e.g., beating all enemies, agents are required to decompose it into multiple subgoals and figure out the right one to focus on. Inspired by previous work, we try to address these issues at the entity level and propose novel framework for language grounding in multi-agent reinforcement learning, entity divider (EnDi). EnDi enables agents to independently learn subgoal division at the entity level and act in the environ- ment based on the associated entities. The subgoal division is regularized by agent modeling avoid subgoal conflicts and promote coordinated strategies. Empir- ically, EnDi demonstrates the strong generalization ability to unseen games with new dynamics and expresses the superiority over existing methods.
INTRODUCTION
The generalization of reinforcement learning (RL) agents to new environments is challenging, even environments slightly different from those seen during training (Finn al., 2017). Recently, language grounding has been proven to be an effective way to grant RI. agents the generalization ability (Zhong et al., 2019; Hanjie et al., 2021). By relating the dynamics of the environment with the text manual specifying the environment dynamics at the entity level, the language-baser agent can adapt to new settings with unseen entities or dynamics. In addition, language-based RL provides framework for enabling agents reach user-specified goal states described by natural language (Küttler et al., 2020; Tellex et al., 2020; Branavan et al., 2012). Language description can express abstract goals sets of constraints the states and drive generalization.
However, in multi-agent settings, things could be different. First, the policies of others also affect the dynamics of the environment, while text manual does not provide such information. Therefore, the generalization of policies should also consider the influence of others. Second, with the increasing number of entities in multi-agent settings, is the number of agent-entity interactions needed for language grounding. The enormous search space could impede the learning process. Third, some- times is unrealistic to give detailed instructions tell exactly what do for each agent. On the contrary, simple goal instruction, e.g., beating all enemies or collecting all the treasuries, is more convenient and effective. Therefore, learning subgoal division and cultivating coordinated strategies based on one single general instruction is required.
The key to generalization in previous works (Zhong al., 2019; Hanjie et al., 2021) is grounding language to dynamics at the entity level. By doing so, agents can reason over the dynamic rules of all the entities in the environment. Since the dynamic of the entity is the basic component of the dynamics of the environment, such language grounding is invariant to new distribution of dynamics or tasks, making the generalization more reliable. Inspired by this, in multi-agent settings, the influence of policies of others should also be reflected the entity level for better generalization. In addition, after jointly grounding the text manual and the language-bart goal (goal instruction) to environment entities, each entity has been associated with explicit dynamic rules and relationships

---

Under review as a conference paper at ICLR 2023
with the goal state. Thus, the entities with language grounding can also be utilized form better strategies.
We present two goal-based multi-agent environments based on two previous single-agent settings, i.e, MESSENGER (Hanjie et al., 2021) and RTFM (Zhong et al., 2019), which require generaliza- tion to new dynamics (i.e., how entities behave), entity references, and partially observable envi- ronments. Agents are given a document that specifies environment dynamics and a imguage-based goal. Note that one goal may contain multiple subgoals. In more detail, in multi-agent messenger, agents are required to bring all the messages to the targets, while in multi-agent RTFM, the general goal is to eliminate all monsters in a given team. Thus, one single agent may struggle or be unable to fimish it. In particular, after identifying relevant information in the language descriptions, agents need to decompose the general goal into many subgoals and figure out the optimal subgoal divi- sion strategy. Note that we focus on interactive environments that are easily converted to symbolic representations, instead of raw visual observations, for efficiency, interpretability, and emphasis on abstractions over perception.
In this paper, we propose a novel framework for language grounding in multi-agent reinforcement learning (MARL). entity divider (EnDi), to enable agents independently learn subgoal division strategics at the entity level. Specifically, each EnDi agent first generates a language-based rep- resentation for the enviromment and decomposes the goal instruction into two subgoals: self and others. Note that the subgoal can be described at the entity level since language descriptions have given the explicit relationship between the goal and all entities. Then, the EnDi agent acts in the environment based on the associated entities of the seif subgoal. To consider the influence of others, the EnDi agent has two policy heads. One is to interact with the environment, and another for agent modeling. The EnDi agent is jointly trained end-to-end using reinforcement learning and su- pervised learning for two policy heads, respectively. The gradient signal of the supervised learning from the agent modeling is used to regularize the subgoal division of others.
Our framework is the first attempt to address the challenges of grounding language for generalization to unseen dynamics in multi-agent settings. EnDi can be instantiated on many existing language grounding modules and is currently built and evaluated in two multi-agent environments mentioned above. Empirically, we demonstrate that EnDi outperforms existing language-basee methods in all tasks by a large margin. Importantly, EnDi also expresses the best generalization ability on unseen games, i.e., zero-shot transfer. By ablation studies, verify the effectiveness of each component, and EnDi indeed can obtain coordinated subgoal division strategies by agent modeling. We also argue that many language grounding problems can be addressed at the entity level.
2 RELATED WORK
Language grounded policy-learning, Language grounding refers to learning the meaning of nat- ural language units, e.g., utterances, phrases, or words, by leveraging the non-linguistic context. In Blukis et al., 2019; Janner et al., 2018; Küttler al., 2020; Tellex et al., 2020; Branavan et al., 2012), the text conveys the goal or instruction to the agent, and the agent produces behaviors in response after the language grounding. Thus, encourages a strong connection between the given instruction and the policy.
many previous works (Wang et al., 2019;
More recently, many works have extensively explored the generalization from many different per- spectives. Hill et al. (2020a; 2019; 2020b) investigated the generalization regarding novel entity combinations, from synthetic template commands to natural instructions given by humans and the number of objects. Choi et al. (2021) proposed a language-guided policy learning algorithm, en- abling learning new tasks quickly with language corrections. In addition, Co-Reyes et al. (2018) proposed to guide policies by language to generalize new tasks by meta learning. Huang et al. (2022) utilized the generalization of large language models to achieve zero-shot planners.
However, all these works may not generalize to a new distribution of dynamics tasks since they encourage a strong connection between the given instruction and the policy, not the dynamics of the environment.
Language grounding to dynamics of environments. A different line of research has focused on utilizing manuals as auxiliary information to aid generalization. These text manuals provide
2

---

Under review as a conference paper at ICLR 2023
descriptions of the entities in the environment and their dynamies, e.g., how they interact with other entities. Agents can figure out the dynamics of the enviromment based on the manual.
Narasimhan et al. (2018) explored transfer methods by simply concatenating the text description of an entity and the entity itself to facilitate policy generalization across tasks. RTFM (Zhong et al., 2019) builds the codependent representations of text manual and observation of the environment, denoted as tær22 based on bidirectional feature-wic wise linear modulation (FiLM²). key challenge in RTFM multi-modal multi-step reasoning with texts associated with multiple entities. EMMA (Hanjie et al.. 2021) uses an entity-conditioned attention module that allows for selective focus over relevant descriptions in the text manual for each entity in the environment called MESSENGER. A key challenge in MESSENGER is the adversarial train- rriin-vaaluation split without prior entity-text grounding. Recently, SILG (Zhong et al., 2021) unifies collection of diverse grounded language learning environments under common interface, including RTFM and MESSENGER. All these works have demonstrated the generalization of policies new environment with unseen entities and text descriptions.
Compared with the previous works, our work moves one step forward, investigating language grounding at the entity level in multi-agent settings.
Subgoal Assignment. In the goal-based multi-agent setting, in order to complete the goal more efficiently, agents need to coordinate with each other, e.g., making reasonable subgoal division. There are many sanguage-free MARL models (Wang et al., 2020; 2019; Jcon et al., 2022: Tang et al., 2018; Yang al., 2019) focusing on task allocation behavioral diversity, which exhibit similar effect as subgoal assignment.
However, without the help of inherited generalization of natural language, it is unlikely for the agent to perform well in environments unseen during training, which is supported by many previous works (Zhong al., 2019; Hanjie et al., 2021). In addition, it is hard to depict the goal state sometimes without natural language (Liu et al., 2022), which makes the subgoal assignment even more challenging.
PRELIMINARIES
Our objective is to ground language to environment dynamics and entities for generalization to unseen multi-agent environments. Note that an entity is object represented symbol in the observation, and dynamics refer to how entities behave in the environment.
POSG. We model the multi-agent decision-making problem game (POSG). For n agents, at each timestep t, agent i obtains its own partial observation o from the global state St, takes action d;,, following its policy (a(a;,,lo..) and receives reward (s,, az) where denotes the joint action of all agents. Then the environment transitions to the next state Se+1 given the current state and joint action according to transition probability function
partially observable stochastic
1 a.). Agent aims to maximize the expected return R² Σ ? Ti where y is the discount factor and T is the episode time horizon.
Wiki
(foni
Jecino pleaninng beat
essontial
atizel
lightiing
twild the
Bleseed, dreaeerr heat palson.
chrest onliee tweet.
the dead' enemy mage
owird
enders.
zombie
bindl poopenaeee the necrel di metitie..
laghtiing
Task
fornet
Agent
Coal
Mersagee
Figure 1: Illustrations of two modified multi-agent environments, i.e., multi auett RTFM (left) and multi-agent MESSENGER (right).
Multi-Agent RTFM is extended from Zhong al. (2019), where for each task all agents given the same text information based on collected written language templates, including docu- ment of environment dynamics and underspecified goal. Figure 1 illustrates instance of the
3

---

Under review as a conference paper at ICLR 2023
O,
Dar Naa
Ensryy
Concatenate O
Selt Pesttiet
B.
Otberd'
Bastey
X
L-m,
Others'
Took
Agent
Paopaptton
Mcatsiicatse
Text Manuii and Lanquage Goal:
Figure 2: Overview architecture of EnDi. Language grounding is first used to obtain language- based representation gouu-- Then, subgoal division module generates binary entity mask m to address important entities. The mask is applied directly to the observation and new language-basee representation Xpoliny for decision-making interact with the environment, and another is for agent modeling, The gradients from supervised learning and reinforcement learning jointly influence the formation of entity masks.
is obtained. EnDi has two policy heads. One to
game. Concretely, the dynamics consist of monsters (e.g., wolf, goblin), teams (e.g., order of the for- est), element types (e.g., fire, poison), item modifiers (e.g., fanatical, arcane), and items (e.g, sword, hammer). When the agents encounter a weapon/monster, they can pick up the weapon/engage in combat with the monster. Moreover, a monster moves towards the nearest observable agent with 60% probability, otherwise moves randomly. The general goal is to eliminate all monsters in a given team with the appropriate weapons. The game environment is rendered as a matrix of texts where each grid describes the entity occupying the grid.
Multi-Agent MESSENGER is built on Hanjie et al. (2021). For each task, all agents are provided with the same text manual. The manual contains descriptions of the entities, the dynamics, and the goal, obtained through crowdsourced human writers. In addition, each entity take one of three roles: an enemy, a message, or a target. There are three possible movement types for each entity, i.e, stationary, chasing, fleeing. Agents are required to bring all the messages to the targets while avoiding enemies. If agents touch an enemy in the game or reach the target without first obtaining the message, they lose the game. Unlike RTFM, the game environment is rendered matrix of symbols without any prior mapping between the entity symbols and their descriptions.
In both multi-agent RTFM and multi-agent MESSENGER, each agent i has an observation of at each moment. By taking action of, each agent can get reward rf. The specific action space and reward rules are shown in Appendix A. Note that we choose these two environments because the effect of language grounding is well suited on them in single-agent scenarios. Also, the difference in manuals and rules of these two environments can help us explore the generalization ability of our EnDi method in the face of different language texts.
4 METHODOLOGY
In goal-based MARL,, it is important for agents to coordinate with each other. Otherwise, subgoal conflicts (multiple agents focusing on the same subgoal) can impede the completion of the general goal. To this end, we introduce EnDi for language grounding in MARL to enable agents indepen- dently learn the subgoal division at the entity level for better generalization. For each task, agents are given text manual E Z describing the environment dynamics and a language-based goal g G as language knowledge. Apart from this, at each timestep I, each agent obtains h x w grid observation og, and outputs distribution over the action π (a;,r los.e F, g. Note that we omit the subscript the following for convenience. Note that parameters each module are not shared among agents.
4.1 OVERALL FRAMEWORK
Language Grounding Module, Given a series of language knowledge, the agent first grounds the language to the observation of the environment, adopting the existing language grounding module
4

---

Under review as a conference paper at ICLR 2023
to generate a language-ber based representation X = Ground(o, E R It captures the relationship between the goal, the manual, and observation. Thus, agents are likely to understand the environment dynamics instead of memorizing any particular information. which is verified by previous works (Zhong et al., 2019; Hanjie et al., 2021). This type of representation is then used to generate the subgoal division and policy.
Note that EnDi compatible with any language grounding module. In practice, we build our frame- work on the backbone of ixf2x (Zhong et al., 2019) and EMMA (Huang et al., 2022) and adopt their grounding modules, i.e. FiLM and multi-modal attention, respectively.
Subgoal Division Module. Since the language-basse goal is highly relevant the entities in the environment after grounding, the subgoal division should also be done at the entity level. To inde- pendently learn the good subgoal division, the goal is decomposed by each agent into two subgoals, the self subgoal it will focus on and the others subgoal which is exploited to avoid subgoal conflicts.
Taking agent to illustrate, it first generates two representations, i.e. X goal
Groundl(oo; 2,),2,9) 2. g) and X = Groundl ([o,, z, g). where 4, is the positional feature of agent which is the Manhattan distance to all the grids and L is the joint positional feature of all E, different agents have different
agents except When grounding the general goal to entities perspectives of how to achieve the goal, thus positional features are used to capture such differences. The two representations
then concatenated together and passed to 2D convolution extract mixture feature map. With the mixture feature map, we utilize a Gumbel-Softmax (Jang et al., 2016) layer to output a subgoal division distribution over all the entities pe Pe L 2, g). determining what entities should be addressed.
To guarantee the uniform entities coverage of subgoal division module, we match the subgoal dis- tribution Pe with target distribution P. as regularization term and aim to minimize DKL llp"). As there is not any prior information for the subgoal division, we assume the target distribution has the mean of lel/n with a relatively low variance in the meantime, where is the number of agents. Intuitively, we are willing to observe that all agents make contributions to the goal. Any distribution with different mean or high variance might violate this motivation.
Policy Module. Agents try to understand the dynamics and the general goal through interactions with the entities. It means that, as the number of entities increases in MARL, the grounding proce- dure could be complex since more interactions are required during training.
Normally, unlike the general goal, agents only need to interact with parts of the relevant entities to complete the subgoal. Intuitively, if agents can ignore trivial entities, the grounding procedure can be simplified and accelerated at its source, and agents
make decisions without distractions.
To this end, we sample a binary entity mask m, based on the subgoal division distribution. The mask is applied the observation directly and indicates the necessity of considering each entity when making decisions. Still, two representations are obtained as X policy = Ground2(o; @ mi, 2, policy Ground2(o; O (1 m,),,,9), where is the element-wise multiplication. There are two different policy heads in EnDi, Tself and Tothers The former one is to output actions for interacting with the environment and takes Xself
and Xothers
paliicy as input. On the other side, X Xothers is passed policy into another head for agent modeling, which will be elaborated on later. Note that Twelf trained using reinforcement learning, and more training details can be found in Appendix B. In addition, Gumbel-softun enables the gradients to flow through the binary entity mask to the subgoal division module.
4.2
AGENT MODELING
However, EnDi does not consider the influence of others, the subgoal division module is prone to converge to sub-optimum. In other words, agents may focus individual interests rather than the common interest. Moreover, we want to capture the influence of others at the entity level for better coordination.
To this end, we let each agent reason about others' intentions by agent modeling. Specifically, EnDi has policy head Tothers based on otheers to predict the joint actions of others a- at each timestep policy
5

---

Under review as a conference paper at ICLR 2023
and update the policy parameter in supervised manner,
[(o,, g) -y log oothers (a
(1)
where y is the concatenation of actual actions (one-hot vector) of others. By agent modeling, each agent can infer the entities that others plan to interact with since only accurate 1 - m, can guarantee the low supervised loss. In other words, it would force the agent to avoid subgoal conflicts at the entity level.
5 EXPERIMENTS
We evaluate EnDi in multi-agent MESSENGER and multi-agent RTFM environments, which we have extended from two previous single-agent environments, Section 3. For fair comparison, we build EnDi based on EMMA and tot2n, which are the models in MESSENGER and RTFM, respectively. More details about the environments can be found in Appendix A.
5.1 CURRICULUM LEARNING
Language-based tasks are much more complex compared with language-free tasks since agents must go through a long process of exploration learn maximize the expected return and ground lan- guage in the meanwhile. To solve multi-agent MESSENGER and RTFM tasks (movable entities, complex game rules, multiple task goals), we design curricula to help policy learning. Note that millions of training steps are required for each training stage. It again demonstrates the complexity of such language- based environments, even dealing with grid worlds.
In multi-agent MESSENGER, we have designed Stages 1, 2, and 3, following settings of MES- SENGER (Hanjie al., 2021).
Stage (S1): There are five entities (enemy, messagel, message2, goall, goal2) that are initialized randomly in five possible locations. Agents always begin in the center of the grid. They start without messages with probability 0.8 and begin with messages otherwise. On S1, agents are required to obtain different messages (start without messages/goalss (start with messages) to win. all agents interact with the correct entities within a given number of steps, a reward of +1 will be provided, and -1 otherwise.
Stage 2 (S2): The five entities are initialized randomly in five possible locations. The difference with is that entities on are mobile. In other words. they can change position randomly at every timestep.
Stage 3 (S3): The five entities are initialized randomly in five possible locations. All agents begin without the message, and the rule requires them interact with the messages first and then with the goals. Within a given number steps, each agent interacting successfully with the messages will be provided a reward of 0.2, and all agents interacting successfully with the goals will obtain a reward of +1, otherwise 1.
In multi-agent RTFM, there five stages for training. Note that Stage Stage 5 follow settings of RTFM (Zhong et al., 2019), while we empirically found Stage is beneficial for later training process in multi-agent setting. During every episode, we subsample set of groups, monsters, modifiers, and elements to use. We randomly generate group assignments of which monsters belong which team and which modifier is effective against which element. Note that each assignment represents a different dynamic.
Stage 1 (SI): There are only four entities in the enviromment, i.e., correct items and monsters. Note that there are one-to-one group assignments, stationary monsters, and no templated language descriptions. Agents need to pick up the correct items and use them to beat all the monsters to win within a given number of steps.
Stage (S2): We add two distractor entities (sampled from the group that is irrelevant the goal). One a distractor weapon that cannot used to beat the monsters. Another is distractor monster, with which agents will lose when engaging.
Stage 3 (S3): All monsters, including the distractor monsters, move towards the agents with 60% probability, otherwise
randomly.
Stage 4 (S4): We allow many-to-one group assignments to make disambiguation more difficult. Note that the difference between many-to-one and one-to-one groups is that the manual would also describe the entities that are not presented in the game.
Stage 5 (S5): The manual would use templated language descriptions.

---

Under review as a conference paper at ICLR 2023
Table 1: The final mean win rate (# stddev.) over five random seeds. All methods adopt the same curriculum training strategy for fair comparison. Note that multi-agent RTFM has five training stages, while multi-agent MESSENGER only needs three stages. tzz22, EMMA, and SILG are the single-agent baselines of previous works. Removing the supervised loss from the total loss, denoted as EnDi-sup, and removing the regularization term, denoted as EnDi-reg. EnDi(num) and EnDi(dis) are our methods with different form of regularization terms.
Model
S1
S2
tzr22m
0.99 + 0.01
0.11 + 0.06
0.05 + 0.05
0.05 + 0.03
0.02 + 0.01
SILG
0.99 + 0.01
0.05 - 0.01
0.04 + 0.01
0.01
0.04 = 0.00
0.04
multi-agent
EnDi-sup
0.99 1 0.01
0.53 1 0.08
0.36 0.03
0.41 1 0.12
0.41 1.113
RTFM
EnDi-reg
1.00 0.00
0.61 0.15
0.39 1.001
0.39 0.03
0.39 1.03
FnDi(num)
1.00 0.00
0.62
0.00
0.54
0.56 +
EnDiídis)
0.00
0.76
0.59
0.61 0.04
0.63 0.03
EMMA
0.57 0.03
0.04
0.02 0.00
SILG
0.01
0.32
0.02
multi-agent MESSENGER
EnDi-sup
0.99 0.01
0.52
0.21 0.03
EnDi-reg
0.67 0.03
0.02
EnDi(num)
0.99 1: 0.01
0.75 1 0.04
0.19
FnDi(dis)
0.99 +: 0.01
0.81 + 0.03
0.25 + 0.03
5.2 RESULTS
In the experiments, we compare EnDi with the following methods: 1) fxt2m (Zhong et al., 2019) which builds the codependent representations of text manual and observation of the environment based on bidirectional feature-wise linear modulation (FiLM²) in RTFM. 2) EMMA (Hanjie et al., 2021) which uses an matity-onnitionel attention module that allows for selective focus over relevant descriptions in the text manual for each entity in MESSENGER. 3) SILG (Zhong et al., 2021) that provides the first shared model architecture for several symbolic interactive environments, including RTFM and MESSENGER. For all methods, we train an independent model for each agent separately.
Performance. Table shows the win rates in both multi-agent RTFM and multi-agent MESSEN- GER environments. In each environment, there are two agents. In the multi-agent RTFM envi- ronment, we use EnDi with txf2 as the backbone lines. It is worth noting that as discussed in Section 4.1, to guarantee the uniform entities cov- erage of subgoal division module, we match the subgoal distribution Pe with target distribution Pa as regularization term. Specifically, we design two patterns for the regularization term, 1) EnDi(num) which minimizes ||e, -e-sl/(n - 1)] where |e, is the entity number of the entity mask for agent i, and 2) EnDi(dis) which minimizes the distance to reach the divided entities, i.e., Ex=(i--) Exce, (|I z&l + Ix, yx where (z, y) denotes the coordinate of entity/agent in the mask. In multi-agent RTFM, all models are able to succeed in the simplest $1, achieving a 99% 100% win rate. When gradually transferring to S2-S5, both EnDi(dis) and EnDi(num) consis- tently show significantly better win rates than baselines.
compare with both tzt2 and SILG base-
We also conduct similar experiments in multi-agent MESSENGER, where on EMMA to compare with both EMMA and SILG basclines. EnDi and SILG both succeed in SI while EMMA obtains win rate of 57%. Then, EMMA is only able achieve a 4% win rate in S2, while EnDi can obtain win rate at most 81%, much better than SILG and EMMA. In S3, the environment is dynamically changing and the tasks are complex, making it extremely difficult to win. Even in the single-agent version, the best result is only a 22% win rate (Hanjie et al., 2021). In the multi-agent version, the complex agent-entity interactions make S3 even more difficult. All models exhibit limited win rates due to the complexity of the environment and tasks, but EnDi(num) and EnDi(dis) still show a substantial win rate lead over baselines.
EnDi that builds
The consistent results in both environments highlight the importance and rationality of EnDi's design of subgoal division, demonstrating its effective performance multi-agent settings. In addition, the poor performance of dingle-agett methods indicates that without coordination behaviors the entity level, they cannot handle more challenging multi-agent settings.
Policy Analysis. To verify the effect of EnDi on the subgoal division, visualize the learned policies using EnDi. Figure 3 shows key snapshots from the learned policies on randomly sampled multi-agent RTFM and multi-agent MESSENGER tasks.

---

Under review as a conference paper at ICLR 2023
Slack consetoe sece Steciie
Figure 3: Illustration of learned subgoal division multi-agent RTFM (upper panel) and multi- agent MESSENGER (lower panel). The dash circle shows the chosen subgoal for each agent.
The upper panel in Figure shows an episode from multi-agent RTFM task. In this task, EnDi can reasonably factor teamwork into the proper subgoal division and complete the task quickly rather than simply finishing the subgoal nearby. In frames 1-3, although Agent1 has closer weapon in the upper left corner, it ignores this weapon and chooses another weapon the top of the map. This decision helps Agent2 to differentiate from Agentl in its subgoal division, and also avoids the distractor monster the left side of the map. In frames 3-5, it be seen that after Agent2 gets the weapon, it chooses to go down to the target monster, leaving the upper target monster to Agent and thus forming a coordinated strategy between the two agents.
The lower panel in Figure 3 shows an episode from multi-agent MESSENGER. We can observe similar subgoal division strategies as multi-agent RTFM. In more detail, the message at the upper left is closer to Agentl than the lower left one in frame 1. However, to avoid subgoal conflicts, Agentl chooses another subgoal, which the message at the lower left. For the rest frames, two agents go to complete their chosen subgoals so the general goal can be finished more efficiently.
Ablations. Compared to tzt2 or EMMA, EnDi has two additional modules: the subgoal divi- sion module and the agent modeling module. To investigate the effectiveness of each module, we compare EnDi(dis) with two ablation baselines. We remove the supervised loss from the total loss, denoted as EnDi-sup, and remove the regularization term, denoted as EnDi-reg. The results are shown in Table 1. Note that the model degrades to lx12 or EMMA if we remove both modules.
As shown in Table 1, EnDi(dis) achieves higher win rate than EnDi-sup and EnDi-reg in both environments. These results verify the importance of considering the influence of others and regu- larizing the subgoal division distribution. In more detail, we found that no subgoal division strategy can be learned in EnDi-reg, where the entity mask converges to a poor sub-optimum, i.e. covering almost all the entities. As for EnDi-sup, we observed that conflicts of the chosen subgoals between agents happen more often than the model with agent modeling.
Extension. EnDi can also adapt to environments with more than two agents. Specifically, the joint positional feature of all other agents is the Manhattan distance of all the grids their closest agents. For agent modeling, we have multiple policy heads for all other agents.
To verify this, we consider three-agent RTFM and set the num- ber of items and monsters to three. Figure 4 shows the learning curves of EnDi(dis) and txt2m. The result is presented in terms of the mean and standard deviation of five runs with different random seeds. Although the SI of multi-agent RTFM is easy for all the baselines, EnDi(dis) can still get a higher win rate. This demonstrates that EnDi(dis) can adapt to more complex settings, i.e., more agents. More thorough investigation is left as future work.
022
0.0
0.2
Figure 4: Learning curve of in terms of the win rate SI of three- agent RTFM.
5.3 GENERALIZATION
As mentioned before, the biggest advantage of language- MARL is that natural language drives the generalization of policies. To investigate the generalization ability of EnDi, zero-shot transfer the model learned from the training tasks to new tasks. Note that we follow the evaluation rules in RTFM (Zhong al., 2019) and MESSENGER (Hanjie al., 2021).
8

---

Under review as a conference paper at ICLR 2023
To assess the importance of language, EMMA (Hanjie al., 2021) has done the ablation studies by replacing the language descriptions with an auxiliary vector (Game ID) where each dimension corresponds to role. The results show the Game ID fails in the complex environment. We also test the generalization of language-free method, QMIX (Rashid et al., 2018). At the simplest S1 in multi-agent RTFM, the mean win rates of QMIX are only 64% and 41% in the training set and unseen tasks, respectively, much worse than EnDi. These results distinguish EnDi from language- free methods.
Multi-Agent RTFM. Importantly no assignments of
are shared between training and evaluation set (denoted as Eval). Thus, even with the same words, the new assignments still lead to new dynamics. However, we also design another evaluation set with the totally novel monsters and modifiers, denoted as Eval(new). Moreover, since we train our model in the 8 x 8 grid world, we also test the generalization to bigger 10 x 10 grid world in the original evaluation set, denoted as Eval(10x 10). Note that all results are obtained by models from S5.
Table 2: The mean win rates (#: stddev.) over five seeds on evaluation in multi-agent RTFM. Note that the result of each seed is obtained by running Ik episodes.
SS-Train
S5-Eval
S5-Eval(new)
S5-Eval(10x10)
EnDi-sup
0.38 0.11
0.39 0.10
0.36
0.10
0.41 0.11
EnDi-reg
0.39 0.02
0.38
0.02
0.35
0.02
0,41 + 0.00
EnDi(dis)
0.59 1 0.01
0.61 0.01
0.56 0.00
0.62 1 0.02
Table 2 shows that EnDi(dis) and its variants have demonstrated an extremely promising generaliza- tion ability to new dynamics unseen during training or a different size of grid world. This benefits from grounding language to the dynamics. In addition, we notice that EnDi-reg performs slightly worse in Eval(new) compared with others. We speculate the language grounding process may be slightly hard in EnDi-reg since the entity mask of EnDi-reg covers more entities than others.
Multi-Agent MESSENGER. For evaluation, we still keep the same entity types, but we make new assignments of word combinations for message, goal, and enemy to ensure that these combinations are unseen during training. We also use completely new description sentences in the text manuals.
The results are shown in Table 3. EnDi(dis) shows the promising generalization ability in multi- agent MESSENGER. EnDi(dis) can maintain 79% and 52% win rates on $1 and S2 respectively, playing with completely unseen entity combinations and text manuals. Due to the difficulty of the S3, all models underperform but EnDi(dis) still maintains the best generalization performance. In addition, the problem of EnDi-reg to handle language grounding is more magnified in multi-agent MESSENGER, yielding lower win rate in both training and evaluation. This result demonstrates the importance of focusing on relevant entities when making decisions.
Table 3: The mean win rates (+ stddev.) over five seeds evaluation in multi-agent MESSENGER. Note that the result of each seed is obtained by running Ik episodes.
S1-Train
S1-Eval
S2-Train
S2-Eval
S3-Train
S3-Eval
EnDi-sup
0.99 # 0.01
0.68 : 0.07
0.52 4.500.005 0.05
0.36 4 0.06
0.21 + 0.03
0.08 0.03
EnDi-reg
0.67 :: 0.03
0.39 0.04
0.07
4 0.02
0.04 + 0.01
0.02 +: 0.01
0.01 + 0.01
EnDi(dis)
0.99 0.01
0.79 0.05
0.03
0.52 a 0.02
0.25 4
0.11 0.02
6 CONCLUSION
We attempt to address the challenges of grounding language for generalization to unseen dynamics in multi-agent settings. To this end, we have proposed a novel language-based framework, EnDi, that enables agents independently learn subgoal division strategies at the entity level. Empirically, EnDi outperforms existing language-based methods in all tasks by large margin and demonstrates a promising generalization ability. Moreover, we conclude that many language grounding problems can be addressed at the entity level.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Valts Blukis, Yannick Terme, Eyvind Niklasson, Ross A Knepper, and Yoav Artzi. Learning to map natural language instructions physical quadcopter control using simulated flight. arXiv preprint arXiv:1910.09664 2019.
SRK Branavan, David Silver, and Regina Barzilay. Learning to win by reading manuals in a monte- carlo framework. Journal of Artificial Intelligence Research, 43:661-704, 2012.
Jongwook Choi, Archit Sharma, Honglak Lee, Sergey Levine, and Shixiang Shane Gu. Variational empowerment representation learning for goal-based reinforcement learning. arXiv preprint arXiv:2106.01404, 2021.
John D Co-Reyes, Abhishek Gupta, Suvansh Sanjeev, Nick Altieri, Jacob Andreas, John DeNero, Pieter Abbeel, and Sergey Levine. Guiding policies with language via meta-learning. arXiv preprint arXiv:181..07892, 2018.
Lasse Espeholt, Hubert Soyer, Remi Munos, Karen Simonyan, Vlad Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dumning, et Impala: Scalable distributed deep-rl with im- portance weighted actor-learner architectures. In International conference machine learning. pp. 1407-1416. PMIR, 2018.
Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning, pp. 1126-1135 1135. PMLR, 2017.
Austin W Hanjie, Victor Y Zhong. and Karthik Narasimhan. Grounding language to entities and dynamics for generalization in reinforcement learning. In International Conference on Machine Learning, pp. 4054 1062.. PMLR, 2021.
Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L. Mc- Clelland, and Adam Santoro. Envirommental drivers of systematicity and generalization in situated agent. arXiv preprint 1rXiv:19/0.00571, 2019.
Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human instruction-followinng with deep reinforcement learning via cansfer-learning from text. arXiv preprint wrXiv:2005.09335 2020a.
Felix Hill, Olivier Tieleman, Tamara Von Glehn, Nathaniel Wong, Hamza Merzic, and Stephen Clark. Grounded language learning fast and slow. arXiv preprint arXiv:2009.01171 2020b.
Wenlong Huang, Picter Abbeel, Deepak Pathak, and Igor Mordatch. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint arXiv:2201.07207, 2022.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameteriization with gumbel-softmax. arXiv preprint arXiv:1611.011 1..01144 144, 2016.
Michael Janner, Karthik Narasimhan, and Regina Barzilay. Representation learning for grounded spatial reasoning. Transactions of the Association for Computational Linguistics, 6:49-61, 2018.
Jeewon Jeon, Woojun Kim, Whiyoung Jung, and Youngchul Sung. Maser: Multi-agent reinforce- ment learning with subgoals generated from experience replay buffer. In International Conference on Machine Learning. pp. 10041-10052 PMLR, 2022.
Heinrich Küttler, Nantas Nardelli, Alexander Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. Advances in Neural Information Processing Systems, 33:7671-7684 2020.
Minghuan Liu, Menghui Zhu, and Weinan Zhang. coal-conditioned reinforcement learning: Prob- lems and solutions. arXiv preprint arXiv:2201.08299, 2022.
Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. Grounding language for transfer in
deep reinforcement learning. Journal of Artificial Intelligence Research, 8744, 2018.
10

---

Under review as a conference paper at ICLR 2023
Tabish Rashid, Mikayel Samvelyan, Christian Schroeder, Gregory Farquhar, Jakob Foerster, and Shimon Whiteson, Qmix: Monotonic value function factorisation for deep multi-agent reinforce- ment learning. In International conference machine learning, pp. 4295 1775-434.. PMLR, 2018.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347, 2017.
Hongyao Tang, Jianye Hao, Tangji Lv, Yingfeng Chen, Zongzhang Zhang, Hangtian Jia, Chunxu Ren, Yan Zheng, Zhaopeng Meng. Changjie Fan, et al. Hierarchical deep multiagent reinforce- ment learning with temporal abstraction. arXiv preprint arXiv: 1809v 1109.0809322 2018.
Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazio and Cynthia Matuszek. Robots that use lan- guage. Annual Review of Control, Robotics, and Autonomous Systems, 3(1), 2020.
Tonghan Wang, Heng Dong, Victor Lesser, and Chongjie Zhang. Roma: Multi-agent reinforcement learning with emergent roles. arXiv preprint 20Xiv:2003308039, 2020.
Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang. William Yang Wang, and Lei Zhang. Reinforced cross-modal matching and self-supervised im- itation learning for vision-language navigation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6629 6638,, 2019.
Jiachen Yang, Igor Borovikov, and Hongyuan Zha. Hierarchical cooperative multi-agent reinforce- ment learning with skill discovery. arXiv preprint 1912,0339 2019.
Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. Rufm: generalising to novel environment dynamics via reading. arXiv preprint arXiv: 2019.
Victor Zhong, Austin W Hanjie, Sida Wang, Karthik Narasimhan, and Luke Zettlemoyer. Silg: The multi-domain symbolic interactive language grounding benchmark. Advances in Neural Informa- tion Processing Systems, 50521555- 2021.
11

---

Under review as a conference paper at ICLR 2023
A ENVIRONMENT DETAILS
A.1 AGENI-AGENT RTFM
In multi-agent RTFM, the agents are given document of environment dynamics, observations of the environment, and an underspecified goal instruction. Concretely, we design set of dynamics that consists of monsters e.g, wolf goblin, teams, e.g. Order of the Forest, element types, egg, fire or poison, item modifiers, e.g., fanatical or arcane, and items, e.g,, sword or hammer. When the player in the same grid with monster or weapon, the player picks up the item or engages in combat with the monster. The player can possess one item a time, and drops existing weapons if they pick up new weapon. monster moves towards the nearest observable agent with 60% probability, otherwise moves randomly. The dynamies, the agents' inventories, and the underspecified goal are rendered as text. The game world is rendered as a matrix of texts in which each grid describes the entity occupying the grid. We use Human--ritten templates for stating which monsters belong to which team, which modifiers are effective against which element, and which team the agent should defeat.
For each episode, multi-agent RTFM subsamples a set of groups, monsters, modifiers, and elements use. Multi-agent RTFM randomly generates group assignments of which monsters belong to which team and which modifier is effective against which element. A document that consists of randomly ordered statements corresponding to this group assignment presented the agents. Multi-agent RTFM samples one element, one team, and a monster from that team, e.g., "fire goblin" from "Order of the forest", to be the target monster. Additionally, we sample one modifier that beats the element and an item to be the item that defeats the target monster, egg., "fanatical sword". Similarly, we sample an element, team, and a monster from a different team to be the distractor monster, e,g, poison bat, as well as an item that defeats the distractor monster, e.g,, arcane hammer.
In the multi-agent setting, we set the number of correct items and the number of target monsters to be the same as the number of agents. The number of the distractor item and the distractor monster is set to one. Under this setting, to achieve the goal more efficiently, the agents need to coordinate with each other. One agent can solve the game alone, but need more steps.
Game Rules. Once agents kill all the correct monsters, they win the game. Any agent who kills a correct monster get reward +1.
any agent engages with the distractor monster, they lose the game and whoever engages gets reward -1. Morcover, if they exceed the steps given by the game or they kill the distractor monster with the distractor item, they still lose the game. For each step, all agents get a penalty of -0.02. Note that the reward is not shared across all agents and this is different with multi-agent MESSENGER.
In order to win the game, each agent needs do the following things:
Identify the target team from the goal.
Identify the monsters that belong that team.
Identify which monster is in the world.
Identify the modifiers that are effective against this element.
Find which modifier is present and the item with the modifier.
Figure what item should be picked up among all the correct items in order to avoid miscoordination.
Pick up the correct item.
Figure out what monsters should be engaged among all the correct items in order to avoid miscoordination.
Engage the correct monster in combat.
Entities and Modifiers. Below is list of entities and modifiers contained in multi-agent RTFM:
Monsters: wolf, jaguar, panther, goblin, bat, imp, shaman, ghost, zombie.
Weapons: sword, axe, morningstar, polearm, knife, katana, cutlass, spear
12

---

Under review as a conference paper at ICLR 2023
Elements: cold, fire, lightning, poison.
Modifiers:
Grandmaster, blessed, shimmering, gleaming, fanatical, mysterious, soldier,
arcane.
Teams: star alliance, order of the forest, rebel enclave.
For the Eval-new set, have new list of entities and modifiers:
Monsters: tiger, bear, puma, elf, vampire, gremlin, witch, specter, robot.
Weapons: sabre, tomahawk, sunglow.
Modifiers: superstars, sacred, glittering, shiny, obsessive, bizarre, secret, esoteric.
Language Templates. Multi-agent RTFM collects human-writee language templates for the goal and the dynamics (Zhong et al., 2019). The goal statements in multi-agent RTFM describe which team the agent should defeat. Multi-agent RTFM collects 12 language templates for goal state- ments. The document of environment dynamics consists of two types of statements. The first type describes which monsters are assigned which team. The second type describes which modifiers, which describe items, are effective against which element types, which are associated with monsters. Multi-agent RTFM collects 10 language templates for each type of statement. The entire document is composed of statements, which randomly shuffled. Multi-agent RTFM randomly samples template for each statement, which multi-agent RTFM fills with the monsters and team for the first type and modifiers and elements for the second type.
A.2 MULTI-AGENT MESSENGER
Multi-Agent MESSENGER is built on Hanjie et al. (2021). In order to transform single-agent MES- SENGER into multi-agent MESSENGER, we keep the same entity, role, and adjective as used in single-agent MESSENGER, and increase only the number of entities. For each task, all agents are provided with the same text manual. The manual contains descriptions of the entities, the dynamics, and the goal, obtained through crowdsourced human writers. Crucially, while prior work assumes ground truth mapping (e.g., the word 'queen' in the manual refers to the entity name 'queen' in the observation), multi-agent MESSENGER does not contain priors that map between text and state ob- servations. To succeed in multi-agent MESSENGER, an agent must relate entities and dynamics of the environment their references in the natural language manual using only scalar reward signals from the environment. The overall game mechanics of multi-agent MESSENGER involve obtaining goal.
a message and delivering
In multi-agent MESSENGER, each entity can take one of three roles: an enemy, a message, or target. There are three possible movement types for each entity, i.e., stationary, chasing, fleeing. Agents are required to bring all the messages to the targets while avoiding enemies. If agents touch an enemy in the game or reach the target without first obtaining the message, they lose the game. There are twelve different entities. Each set of entity-role assignments is initialized on a 10 x 10 grid. The agent can navigate via up, down, left, right, and stay actions and interacts with another entity when both occupy the same grid. The same set of entities with the same movements may be assigned different roles different games. Thus, two games may have identical observations but differ in the reward function (which is not available to the agents) and the text manual (which is available). Thus, the agents must learn to extract information from the text manual to succeed consistently.
Game Rules. In order to win the game, each agent needs to do the following things,
Identify the entities that hold the message.
Map descriptions to the correct symbols the observation.
Observe the movement patterns of entities disambiguate which of the two entities holds
the message.
Pick up the messages from the entities that hold them.
Follow a similar procedure to the third step to disambiguate which mages are the goals and which the enemies.
13

---

Under review as a conference paper at ICLR 2023
Bring the messages to the goals.
In different stages, the reward functions also have the following differences:
SI & S2: If all the agents interact with the correct entities (i.e., agents started with message obtain goal and agents started without message obtain a message) within a given number of steps, reward of 1.0 will be provided, and 11.0 otherwise.
S3: The agents are required to get the messages first and then the goals that it can be considered win. Within a given number of steps, each agent interacting successfully with the messages will be provided reward of 0.2, and all the agents interacting successfully with the goals will receive a reward of 1.0, otherwise -1.0.
Entities. Below is a list of entities contained in multi-agent MESSENGER:
Enemy: enemy, opponent, adversary (Adjectives: dangerous, deadly, lethal).
Message: message, memo, report. (Adjectives: restricted, classified, secret)
Goal: goal, target, aim. (Adjectives: crucial, vital, essential)
Language Templates. We use the same corpus of text manual as in the original EMMA paper (Hanjie et al., 2021), ie.,, 82 templates with 2214 possible descriptions after filling in the blanks. We have three blanks per template, one each for the entity, role, and adjective. For each role, we have three role words and three adjectives that are synonymous. Each entity also described in three synonymous ways. Thus, every entity-role assignment can be described in 27 different ways on the same template. The raw templates are filtered for duplicates, converted lowercase, and corrected for typos to prevent confusion in specific tasks.
B IMPLEMENTATION AND TRAINING DETAILS
First, for both environments, we train 5 runs with different random seeds for each bascline report the mean and standard deviation. For generalization, we zero-shot transfer each of the 5 runs to the new task and once again report the mean and standard deviation.
B. MULTI-AGENT RTFM
We train using an implementation of IMPALA (Espeholt et al., 2018). In particular, we use 10 actors and batch size of 20. When unrolling actors, episode lasts for a maximum of 1000 steps. We optimize using RMSProp with learning rate of 0.005, which is annealed linearly for 100 million steps. We
maximum unroll length of 80 steps. Each
0.99 and 0.01.
During training, we apply a small negative reward for each time step of -0.02 and a discount factor of 0.99 facilitate convergence. We additionally include entropy regularization to encourage exploration. RTFM sets in the entropy loss with a weight of 0.005 and the baseline loss with weight of 0.5. The baseline loss is computed the mean square of the advantages (Espeholt et al., 2018).
For S1, we train the model for 1.5 x 10" steps. For the rest stages, i.e., S2-S5, we train the model for 5 x 10" steps.
B.2 MULTI-AGENT MESSENGER
The models end-to-end differentiable and we train them using proximal policy optimization (PPO) (Schulman et al., 2017) with 7 = 0.99, e = 0.01 and the Adam optimizer with learning rate 5 5 103. On S1, $2, and S3, limit each episode to 32, and 64 steps respectively. We train models for up to 12 hours on S1, 72 hours on $2, and 144 hours on S3. We use the validation games to save the model parameters with the highest validation win rate during training and use these parameters to evaluate the models on the test games.
For S1, train the models for 3 10? steps. For $2 and S3, train the model for 5 107 steps.
14

---

Under review as a conference paper at ICLR 2023
C MODEL DESIGN
We have two versions of EnDi. One takes the tæt2x as backbone. Another takes EMMA as back- bone.
C.I
12222 VERSION
Linear
Text Features
Linear
Linear
ReLU
O Output
Conv
ReLU
Visual Features
Conv
MaxPool
Summary
Conv
Figure 5: Architecture of FiLM?. FiLM? builds codependent representations of text and visual inputs by modulating each kind of input using representations of input in another modality.
For the language grounding module, we utilize FiLM² to capture the relationship between language knowledge and environment observations. The architecture of FiLM is shown Figure Not that each FiLM layer uses 3 x 3 convolutions and padding and stride sizes of 1. In more detail, the grounding module that is used to generate X and Xothers is shared and consists of two goal FiLM² layers with channel of 16 and 4. The convolution layer that processes the concatenation of X and Xothers uses kernel of x 3 and outputs 2 dimension features. Moreover, the grounding module that is used to generate Xself policy and X Xothers policy are shared and consists of one FiLM² layer with channel of 16, 32, 64, 64, and 64, with residual connections from the 3rd layer the 5th layer. The two representations need get polling operation over spatial dimension before going through the corresponding policy head (one fully-connected layer).
goal
For other text pre- vreprroesssing modules, we strictly follow the RTFM setting (Zhong et al., 2019). The Bidirectional LSTM (BiLSTM) that processes the inventory and the goal has hidden dimension of size 10. The BiLSTM that processes the document has a hidden dimension of size 100. Note that we use a word embedding dimension 30.
For more details about the tæt22 please refer the paper (Zhong al., 2019).
C.2 EMMA VERSION
We follow the EMMA settings to both capture the relationship between the language knowledge and the environment observations, and pre-process the language manual the MESSENGER environ- ment. The architecture of EMMA is shown in Figure 6. The EMMA model consists of components including Text Encoder, Entity Representation Generator and Action Module. Text Encoder, the input consists of h x grid observation with set of entity descriptions. EMMA encodes each description using BERT-base model whose parameters are fixed throughout training. Then the key and value vectors are obtained from the encoder. In Entity Representation Generator, EMMA embeds each entity's 'ss symbol into a query vector to attend to the descriptions with their respective key and value vectors. For each entity in the observation. EMMA places its representation. z, into tensor e wxxd at the same coordinates as the entity position the observation to maintain full spatial information. The representation for the agent is simply learned embedding of dimension
15

---

Under review as a conference paper at ICLR 2023
d. In Action Module, provide temporal information that assists with grounding movement dy- namics, EMMA concatenates the outputs of the representation generator from the three most recent observations to obtain tensor X' R Xxxxxd To get distribution over the actions, EMMA runs a 2D convolution on X' over the h, w dimensions. The flattened feature maps are passed through the possible actions.
fully-connected FFN terminating in softmax
For more details about EMMA, please refer to the paper (Hanjie et al., 2021).
token
key vector
Encode
Text Features
token
value vector
token
Embed
Visual Features
Attention
Conv + FFN
ReLU
Output
Figure 6: Architecture of EMMA.
D ADDITIONAL RESULTS
InDiddaa) EnDi(rum)
Intitda)
EnD(inu))
0.6
0.6
0.0
0.0
2
stop
1e7
stop
1e7
(a) Stage
(b) Stage
1.0
1.0
Jdis
Jrum)
(num)
0.8
0.8
EnDI rog
EnD rog
RITM
SILG
0.6
SIG
0.4
0.4
0.2
0.2
0.0
step
167
(c) Stage 4
(d) Stage 5
Figure 7: Learning curves in terms of the win rate of EnDi and baselines in multi-agent RTFM.
In this section, we provide the learning curve of EnDi and basclines. All results are presented in terms of the mean and standard deviation of five runs with different random seeds.
16

---

Under review as a conference paper at ICLR 2023
1.0
6.6
Growesan
1.7
28
0.2
546
167
107
(a) Stage
(b) Stage 2
(c) Stage
Figure 8: Learning curves in terms of the win rate of EnDi and baselines in multi-agent MESSEN- GER.
Figure 7 shows the results in multi-agent RTFM from Stage 2 to Stage 5. Note that Stage 1 is omitted since all baselines achieve nearly 100% win rate. Figure shows the results multi-agent MESSENGER from Stage I to Stage 3. The results again demonstrate the superiority of our method over baselines.
E RESULTS WITHOUT VALIDATION
Table 4: The mean win rates (+: stddev.) of EnDi(dis) without validation over five seeds on training in multi-agent MESSENGER. Note that the result of each seed is obtained by running 1k episodes.
EnDi(dis) train
w/ validation
w/o validation
S1
0.99 0.01
0.99 + 0.01
S2
0.81 + 0.03
0.76 + 0.02
S3
0.25 0.03
0.20 + 0.04
The mean win rates (+- stddev.) of EnDi(dis) without validation five seeds eval- uation in multi-agent MESSENGER. Note that the result of each seed is obtained by running Ik episodes.
Table 5:
EnDi(dis) test
w/ validation
w/o validation
S1
0.79 = 0.05
0.77 = 0.02
$2
0.52 + 0.02
0.48 + 0.03
S3
0.11 0.02
0.09 0.02
paper added the validation procedure. But to verify the performance of EnDi more strictly, conduct additional ablation experiments without the validation procedure. As shown Table 4 and 5, the performance of EnDi is only weakly affected by the removal of validation and still outperforms the baselines. Therefore, validation does not affect the conclusions in the experimental results of the paper.
order to make a fair comparison, we followed the original EMMA setting
F ERROR ANALYSIS
ERROR CASE IN MULTI-AGENT RTFM
Case (Figure 9, upper panely: the entity division is reasonable in Multi-agent RTFM: Fire Imp (tar- get monster), Shimmering Sword (good weapon) and Poison Zombie (distractor monster) belongs the entity division of agent 1, while Fanatical Kanata (good weapon). Lightning Ghost (target
17

---

Under review as a conference paper at ICLR 2023
Task
Agent 1
Agent
Agent
preur minoddon, hewend hest
day
atiemmnn pluun
bost
bell anguar are onter of troeet
zombis watf
Swerdd
Chical
Zambia
Agert
Agen
Agent
Agen
Wiki
Tass
Agent
Agent
Agent
mysueatous,,
blenned saminng best nn-
most defented
aniammeing qu so siees beet poison
ghosi
woll butl vlaa lince
Coandi mua
v
Agen
Agent
Agen
Figure 9: Error cases of EnDi in Multi-agent RTFM.
monster) and Blessed Spear (bad weapon) belongs to the entity division of agent 2. Each agent's subgoal includes good weapon and a target monster. However, agent 2 picks up the bad weapon (Blessed Spear), resulting in being killed by the monster (Lighting Ghost) that chased it on the way to the good weapon(Fanatica) Kanata), although Lighting Ghost is the target monster.
Case 2 (Figure 9, lower panel): the entity division also rational: Cold Imp (target monster), Arcane Kanata (good weapon), Lightning Zombie (distractor monster) and Gleaming Spear(bad weapon) belongs the entity division of agent while Fire Ghost(target monster) and Grandmas- ters Sword (good weapon) belongs to the entity division of agent 2. Agent 2 choose to pick up the good weapon (Grandmasters Sword), but at that point it's beat by the distractor monster (Lightning Zombie) that came after it.
All in all, we can blame the error case to that agents recognize the wrong entities. In other words, the language grounding procedure may suffer from the multi-agent settings.
F.2
ERROR CASE IN MULTI-AGENT MESSENGER
Manual
Agent
Agior
Agent
the devecly
the airplane holes the secret message.
Agentt
Mank Apertz
Agent
Agent
Manual
Agen:
Agent
Agent
essectial stade
the deadly onemy the rocet.
the secrei the magela hand.
Agent
Agent 1
Adent
Agand
Figure 10: Error cases of EnDi in Multi-agent MESSENGER.
18

---

Under review as a conference paper at ICLR 2023
In frame of Case 1 (Figure 10, upper panel), the entity divider reasonably assigns the entities to both agents. But in frame 3, the enemy is close to both agents but the entity divider can only assign the enemy to agent 1, resulting in agent 2 not paying good attention to the enemy and being killed by the enemy in frame 5. In the future, we consider to solve this problem by adding some sharing elements to the entity divider.
Another unexpected situation is shown in Case 2 (Figure 10, lower panel), where both agents are far away from the message on the right side in frame and the entity divider can only assign one agent. But in frame 3, due to the movements of the message and the goal, the entity divider chooses reassign the message, causing agent to change its direction from downward upward after frame 3. Although the agents finally complete the task. the invalid movement in the first few frames makes the agents less efficient. We will consider more reasonable assignment way to improve the entity divider.
19


Reviews:
[{"review_id": "lKRpZVbLFUV", "reviewer": "Reviewer_affK", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies using natural language as instructions and manuals for multi-agent reinforcement learning (MARL). The problem is challenging because the agents need to ground the language and consider other agents’ behavior at the same time. This paper proposed a language-grounding framework, which is referred to as ‘entity divider (EnDi)’, for MARL. Based on opponent modeling, the proposed approach enables agents to divide subgoals at entity level and act with the environment accordingly. The proposed EnDi is evaluated on multi-agent RTFM and multi-agent MESSENGER which are augmented from the popular RTFM and MESSENGER environment. The experimental results show that the proposed approach outperforms baselines and is able to generalize to unseen tasks. \n", "summary_of_the_review": "In summary, the reviewer found the problem and the proposed approach interesting. However,  it is unclear if the proposed approach is applicable to more complex environments, e.g. more agents, non-grid-world tasks, and limited visual range. Further clarification and revision are needed. \n\n", "strength_and_weaknesses": "### Strength:  \n1. Language grounding for multi-agent settings is an important area which draws less attention. The reviewer thinks this paper took a first step toward this goal. \n\n2. The reviewer found the proposed sub-goal division, which based on opponent modeling, simple yet interesting. The experimental results also demonstrate its effectiveness on two MARL environments. \n\n3. The paper is well-written and easy to follow.  \n\n### Weakness / Questions:\n\n1. To get $X^{others}_{goal}$, joint position feature of all other agents ($l_{-i}$) is needed.  Isn’t it a privileged knowledge that violates the partially observable assumption discussed in Section3 and the ‘decentralized execution’ of general multi-agent settings?\n\n2. Are the binary entity masks on the full grid? Or is it only on each agent’s visual range (local observation)? In addition, what is the visual range of each agent in the experiments?\n\n3. It seems that the opponent modeling (Eq. 1) is a critical component for avoiding sub-goal conflict. However, opponent modeling itself is a challenging problem. It is unclear how the proposed method ensures the modeling results are good and thus lead to good subgoal division? In addition, Suppose agent A never occurs in another agent’s, e.g. agent B, visual range. How could agent B model agent A’s behavior correctly?  If the agent is not able to model others’ behaviors accurately, how would it impact the quality of the binary entity mask?\n\n4. It is unclear how many agents each task. It seems that there are only two controlled agents in a task. The experimental section could be more convincing if there are more controlled agents in the tasks. \n\n5. In p. 8, the paper explains Fig. 3: “ it chooses to go down to the target monster, leaving the upper target monster to Agent1…”. Given the upper monster is left for agent 1, why is the upper monster assigned to the mask of agent 2?\n\n6. Is the proposed sub-goal approach applicable to environments that are not grid-world based?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Ew-1KAnZ4d", "reviewer": "Reviewer_xDyS", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper addresses the problem of collaborative MARL with language for specifying instructions and environmental dynamics, focusing on the problem of distributing subgoals among multiple agents. The approach proposed in the paper builds upon the grounding architecture from Zhong et al, 2019 and Hanjie et al. 2021, with two main architectural changes meant to enable individual agents to focus on different subgoals: (a) complementary masking over observations, leading to different agents attending to different entities in the observation; and (b) modeling of other co-operating agents as an auxiliary objective. \n\nIn the experiments on multi-agent versions of Messenger (Hanjie et al, 2021) and RTFM (Zhong et al, 2019), the proposed architecture outperforms independently trained agents (without the modifications to the architecture), also demonstrating generalization to completely new monsters / modifiers and increased grid size. ", "summary_of_the_review": "The paper proposes several architectural changes to enable subgoal division in collaborative MARL with language for specifying instructions and environmental dynamics. The results with two agents are promising, with the proposed method significantly outperforming independently trained agents with a non-modified grounding architecture. However, the paper is not ready to be accepted due to a lack of comparison to standard MARL baselines, methodological issues, and a lack of clarity about the method.", "strength_and_weaknesses": "I appreciated that the setting addressed in this paper is novel, in general, language-assisted RL is not well-studied and it's good to see more work in this area. The results are certainly promising, the proposed method strongly outperforms independently trained agents, with particularly interesting results in generalization to unseen entities / modifiers and grid-sizes. The proposed architecture overall makes sense. \n\nHowever, this work is not ready for publication due to a lack of comparison to standard MARL baselines, methodological issues, and a lack of clarity about the method (see Clarity section).  \n\nThe authors do not specify well which grounding-related problems are introduced by the addition of multi-agent dynamics to language-assisted RL beyond standard problems of MARL (i.e. problems related to incorporating language or attending to manual; instead of just increased non-stationarity and learning coordination strategies). The authors touch on this question in p2 of the Introduction, however, this needs to be elaborated on better. As it stands, the proposed method could be used in most MARL benchmarks and standard MARL methods (such as QMIX [1], MAPPO [2]) could be combined with architectures more suitable for grounding such as EMMA and text2pi when the task demands language grounding. So the authors could also examine the proposed architecture in standard non-language MARL tasks; but in either case, the baselines need to include standard MARL methods. While one of the experiments does contain a comparison to QMIX (Section 5.3), this is only in a language-free setting (i.e. using QMIX with a standard architecture that is not suitable for language grounding). If there is a reason why standard MARL methods can not be combined with grounding architectures, the authors need to elaborate on it in the paper. \n \nIf I'm interpreting this correctly, the results are reported over environment random seeds, but not training seeds. In section B.2 of the Appendix, the authors say that the validation games are used to select the highest-performing parameters instead of hyperparameters, with the reported results corresponding to this best-performing model on the random selections of test environments. This is very non-standard in RL for a variety of reasons [3], including not giving a sense of how stable the training is. The authors also do not seem to perform any hyperparameter search over either the proposed model or the baselines, which is a problem since the default baseline hyperparameters were optimized on another version of the environment. My suggestion is to perform a hyperparameter search, use the validation performance for hyperparameter selection, then report the results over both training and environment seeds for the best hyperparameters (but not the model). \n\nThe generalization results (Table 2 & 3) do not report baseline performance.\n\nLastly, there is only one experiment with more than 2 agents (it's only 3 agents, and the experiment is on the easiest version of one of the environments). The performance gap is much smaller there, with the proposed model barely outperforming the non-MARL baseline (why is this the case?). This sheds doubt on whether the proposed method works as well with more than 2 agents and in more challenging environments. \n\n[1] https://arxiv.org/abs/1803.11485\n[2] https://arxiv.org/abs/2103.01955\n[3] https://arxiv.org/abs/1709.06560", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "DRyL2EozBf", "reviewer": "Reviewer_dLxy", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors address the challenges of grounding language in generalization for unseen dynamics in multi-agent settings. They propose a new framework for language grounding called entity divider (EnDi).\nEnDi enables agents to independently learn subgoal division at the entity level and act in the environment. This division is regularized by the opponent\nmodeling to avoid subgoal conflicts and promote coordinated strategies. \nThe authors also conducted experiments to show that EnDi outperformed existing language-based methods in all tasks by a large margin.", "summary_of_the_review": "In summary, this is an interesting paper that provides a new method. However, the description and style may have room for improvement. It is better to make the abstract and conclusion more detailed and explicit.", "strength_and_weaknesses": "Strength\nThe experiment shows that their proposal is working better than the baselines.\n\nWeakness\nThe generality of the proposed model in broader MARL studies is not apparent.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "-FUzlPU2l9x", "reviewer": "Reviewer_imbM", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new method for improving generalization in multi-agent games, where game manuals describing entity and environment dynamics are provided, and high-level goals are specified in natural language. The proposed method learns a policy for each entity, trained with reinforcement learning, with an auxiliary supervised loss on predicting the dynamics of other agents in the environment. They modify two existing similar environments to be multi-agent (MESSENGER and RTFM) and evaluate with several comparisons to other methods and ablations.", "summary_of_the_review": "This paper was interesting, and I think the contribution is potentially useful, but I did find the description of the task itself to be relatively informal to the point of being somewhat difficult to understand the contributions. I am also not deeply familiar with multi-agent work, so cannot evaluate it thoroughly from that point of view.", "strength_and_weaknesses": "Strengths:\n- The use of \"opponent\" modeling as an auxiliary loss is intriguing (although I don't know how conventional it is in multi-agent RL already).\n- Description of POSG is clear (Section 3)\n- Performance of proposed method significantly outperforms existing methods\n- Experiments are relatively thorough\n\nWeaknesses:\n- See below for clarity concerns.\n- I'd like more discussion on what is functionally different between the two environments that make both of them good to study.\n- I found it a bit confusing that EnDi enabling agents to individually model subgoals also helps the agents coordinate. By modeling subgoal division, are the agents explicitly predicting how all of the other agents will act next? Does this mean it's possible for two collaborative agents to fail to complete a subgoal under the \"assumption\" the other will complete it, e.g., if they are both equally far away from the entity relevant to that subgoal?\n\nSome questions:\n- The opponent modeling head actually predicts multiple actions, corresponding to all of the other agents, right?\n- In Eq. 1, does each agent have access to the observations of the other agents, or is it predicting others' actions based on its own partial observation of the world?\n- How were the stages for the curriculum learning chosen?\n- Supervised loss ablation (EnDi-sup) is equivalent to ablating the opponent modeling, right?\n- Why evaluate QMIX, rather than ablating language as much as possible in EnDi? E.g., by removing access to the manual representation z?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject