Grounding Language to Entities for Generalization in Reinforcement Learning
Authors: Austin W. Hanjie
Karthik R Narasimhan
Venue:  (2021)
Keywords: reinforcement learning
language grounding

Abstract: In this paper, we consider the problem of leveraging textual descriptions to improve generalization of control policies to new scenarios. Unlike prior work in this space, we do not assume access to any form of prior knowledge connecting text and state observations, and learn both symbol grounding and control policy simultaneously. This is challenging due to a lack of concrete supervision, and incorrect groundings can result in worse performance than policies that do not use the text at all. 
We develop a new model, EMMA (Entity Mapper with Multi-modal Attention) which uses a multi-modal entity-conditioned attention module that allows for selective focus over relevant sentences in the manual for each entity in the environment. EMMA is end-to-end differentiable and can learn a latent grounding of entities and dynamics from text to observations using environment rewards as the only source of supervision.
To empirically test our model, we design a new framework of 1320 games and collect text manuals with free-form natural language via crowd-sourcing. We demonstrate that EMMA achieves successful zero-shot generalization to unseen games with new dynamics, obtaining significantly higher rewards compared to multiple baselines. The grounding acquired by EMMA is also robust to noisy descriptions and linguistic variation.

Under review as a conference paper at ICLR 2021
GROUNDING LANGUAGE TO ENTITIES FOR GENERALIZATION IN REINFORCEMENT LEARNING
Anonymous authors Paper under double-blind review
ABSTRACT
In this paper, we consider the problem of leveraging textual descriptions to im- prove generalization of control policies to new scenarios. Unlike prior work this space, do not assume access to any form of prior knowledge connecting text and state observations, and learn both symbol grounding and control policy simultaneously. This is challenging due lack of concrete supervision, and in- correct groundings can result in worse performance than policies that do not use the text at all. We develop Attention) which uses a multi-modal entity-conditioned attention module that al- lows for selective focus over relevant sentences in the manual for each entity in the environment. EMMA is end-to-end differentiable and can learn a latent grounding of entities and dynamics from text to observations using enviromment rewards as the only source of supervision. To empirically test our model, we design a new framework of 1320 games and collect text manuals with free-form natural lan- guage via crowd-sourcing. We demonstrate that EMMA achieves successful zero- shot generalization to unseen games with new dynamics, obtaining significantly higher rewards compared to multiple baselines. The grounding acquired by EMMA is also robust to noisy descriptions and linguistic variation
model, EMMA (Entity Mapper with Multi-modal
INTRODUCTION
Interactive game environments are useful for developing agents that learn grounded representations of language for autonomous decision making (Golland et al. 2010 Andreas & Klein 2015 Bah- danau et al., 2018). The key objective in these learning setups is for the agent to utilize feedback from the environment to acquire linguistic representations (e.g. word vectors) that are optimized for the task. Figure D provides an example of such a setting, where the meaning of the word fleeing in the context is to "move away", which captured by the movements of that particular entity (wizurd).
Learning useful grounding of concepts can also help agents navigate new environments with pre- viously unseen entities or dynamics. Recent research has explored this approach by grounding language descriptions to the transition and reward dynamics of an environment (Narasimhan et al. 2018 Zhong et al. 2020). While these methods demonstrate successful transfer to new settings, they require manual specification of some minimal grounding before the agent can learn (e.g. ground-truth mapping between individual entities and their textual symbols).
In this paper, we propose model to learn an effective grounding for entities and dynamics without requiring any prior mapping between text and state observations, using only scalar reward signals from the environment. To achieve this, there are two key inferences for an agent to make - (1) figure out which facts refer to which entities, and (2) understand what the facts mean to guide its decision making. To this end, we develop new model called EMMA (Entity Mapper with Multi- modal Attention), which simultaneously learns to select relevant sentences in the manual for each entity in the game as well incorporate the corresponding text description into its control policy. This is done using multi-modal attention mechanism which uses entity representations as queries to attend to specific tokens in the manual text. EMMA then generates a text-conditioned representation which is processed further by a deep neural network to generate policy. We train the entire model in multi-tast fashion using reinforcement learning to maximize task returns.
Code and data available sttps: //www. wwwpwwwwwwww....ow........ con http///rrrrrrreeessess///ooooo wttppedwddddde.eeoeeoooeeee

---

Under review as a conference paper at ICLR 2021
G2re /
Gane
Gome I Masuel
particsiar locale, there motoniess mongrel tomiidable adversary. the top -so---settttt paperworh the crook's possession, and he heading closer and closer to where you are.
the onicial target hold by witard and the witard is flesing from you. Game Manual
the dog is evadligg you while holdiigg the classiffedd in hiss hands.
the sword the enemy that' is dangeroue and moving cluser to you. thing
onject that budgo the queen
Two games from our multitask domain Messenger where the agent must obtain the mes- sage and delivers it to the goal (white dotted lines). The same entities may have different roles in different games which are revealed by the text descriptions.
Figure I:
To empirically validate our approach, we develop a new multi-task framework containing 1320 games with varying dynamics, where the agent is provided with a text manual in English for each individual game. The manuals contain descriptions of the entities and world dynamics obtained through crowdsourced human writers. The games are designed such that two environments may be identical except for the reward function and terminal states. This approach makes it imperative for the agent to extract the correct information from the text in order to succeed on each game.
Our experiments demonstrate EMMA is able to outperform three types of baselines (language- agnostic, attention-ablated, and Bayesian attention) with a win rate almost 40% higher on training tasks. More importantly, the learned grounding helps our agent generalize well to previously un- seen games without any further training (i.e. a zero-shot test), achieving up to a 79% win rate. Our model is also robust to noise and linguistic variation in the manuals. For instance, when provided an additional distractor description, EMMA still achieves a win-rate of 75% on unseen games.
2 RELATED WORK
Grounding for instruction following Grounding natural language to policies has been explored in
the context of instruction following in tasks like navigation (Chen & Mooney| [201 Hermann et al.
2017
Fried et al. 2018 Wang et al., 2019; Daniele et al. 2017 Misra et al. [2017] Janner et al.
2018
games (Golland et al., 2010 Reckman et al.] 2010] Andreas & Klein [013]] Bahdanau et al.,
2018
Hemachannaa et al.) 2014 Blukis 2019) (see Luketina al. (2019) and Tellex et al. (2020) for more detailed surveys). Recent
Küttler et al. 2020) or robotic control (Walter et
et al.
work has explored several methods for enabling generalization in instruction following, including environmental variations (Hill al. 2020a)). memory structures (Hill et al. 2020c) and trainre language models (Hill et al.] (2020b). In a slightly different setting, Co-Reyes et al. (2018) use incremental guidance, where the text input is provided online, conditioned on the agent's 's progress in the environment. [Andreas et al. (017) developed an agent that can use sub-goal specifications to deal with sparse rewards. [Oh al. (2017) use sub-task instructions and hierarchical reinforcement learning to complete tasks with long action sequences,
In all these works, the text conveys the goal to the agent (e.g. 'move forward five steps'), thereby encouraging a direct connection between the instruction and the control policy. This tight coupling means that any grounding learned by the agent likely to be tailored the types of tasks seen in training. making generalization to a new distribution of dynamics or tasks challenging. In extreme cases, the agent may even function without acquiring an appropriate grounding between language and observations (Hu et al. (209)). In our setup, we assume that the text only provides high-level guidance without directly describing the correct actions for every game state.
Language grounding by reading manuals A different line of work has explored the use of lan- guage as auxiliary source of knowledge through text manuals. These manuals provide useful descriptions the entities in the world and their dynamics (e.g. how they move or interact with other entities) that are optional for the agent to make use of and do not directly reveal the actions has to take. Branavan et al. (212)) developed an agent to play the game of Civilization more effectively by reading the game manual. They make of dependency parses and predicate la- beling to construct feature-base representations of the text, which are then used to construct the action-value function used by the agent. Our method does not require such feature construction.
2

---

Under review as a conference paper at ICLR 2021
Narasimhan et al. (2018) and Zhong al. (2020) used text deseriptions of game dynamics to learn policies that generalize to new environments, without requiring feature engineering. However, these works assume some form of initial grounding provided to the agent (e.g. mapping between object IDs and their descriptions, or the use of entity names in text as state observations). In contrast, our model figures out this fundamental mapping between entity IDs in observation space and their symbols in text entirely through interaction with the environment.
3
FRAMEWORK
Our objective is to demonstrate grounding of environment dynamics and entities in multi-task setup in order drive generalization unseen environments. Here and throughout, we refer to an entity as object represented as symbol in the observation that the agent may interact with.
Environment We model decision making in each environment as a Partially-Obse Obbsevablee Markov Decision Process (POMDP) with the 8-tuple (S, A, O, R, M). S and O are the set of all states and observations respectively where each o E O contains entities from the set of entities E. At each step I, the agent takes some action dg E A. P(t--11*;; (z) the transition distribution over all possible next states $1+1 conditioned on the current state and action ag. R(st. Qe, St+1) is function that provides the agent with a reward r. E for action de and transition from 8t $1+1- Z is set of text descriptions, with each E / providing information about an entity E E. M the map Ze -> which identifies the entity that each description describes. M, P. and are not available to the agent. Note that there might not be one-to-one mapping between Z and entities in the current state observation.
Reinforcement Learning (RI.) The objective of the agent is find policy n: : 0 - A to maxi- mize its cumulative reward in an episode. If π is parameterized by 0, standard deep RL approaches optimize to maximize the expected reward of following Ty. In our setup, we want the agent to learn policy (a o, Z) that conditions its behavior on the provided text. However, in contrast to previous work (Narasimhan et al.) 2018 Zhong et al., 2020), M is not available to our agent and must be learned through interaction.
4
MODEL
To learn latent mapping between text symbols and entities, we develop model, EMMA (En- tity Mapper with Multi-modal Attention). which employs a soft-attention mechanism over the text descriptions. At high level, for each entity description, EMMA first generates key and value vectors from their respective token embeddings obtained using pretrained language model. Each entity attends the descriptors via a symbol embedding that as the attention query. Then, instead of representing each entity with its embedding, we use the resulting attention-scaled values as a proxy for the entity. This approach helps our model learn control policy that focuses on entity roles (e.g. enemy, goal) while using the entities' identity (e.g. queen, mage) to selectively read the text. We describe each component of EMMA below and in Figure 2
Text encoder Our input consists of a x grid observation O with set of entity descriptions
Z. We encode each description 2 E Z using a BERT-base model whose parameters are fixed throughout training (Devlin al., 2018, Wolf et al., 2019). For a description z, let (, bn be its token embeddings generated by our encoder. We obtain key and value vectors ki,, Uz:
k Σ
WgWtt + bb.
a softmax (ux -t,)==1
0 3.W.t, buy
8 softmax ((y))tt)).. fz))==))
The key and value vectors simply linear combinations of Wit, + by and W.t, b, with weights a, 3 respectively, where W., W, are matrices which transform each token to d dimensions and by b, are biases. The weights a, 3 are obtained by taking the softmax over the dot products (ug 1,))_1
3

---

Under review as a conference paper at ICLR 2021
Attontion
de
Embed
query
softmax,
9t
v/d
Kx
Softmax
B
X
2i Pe Wiit + Bv
X
Eneade
Py: queen the enemy...
Sultmax
Conv - FFN
Egt The dig Boonin with the...
xy.
Z
Di a W&F
left
Figure 2: Schematic of our model EMMA, which creates a representation for entities using multi- modal attention the observations and text manual. Mechanisms for the key, query, and value are shaded in blue, green, and red respectively.
and (u, (1j) 1=1 respectively. These weights imbue our model with the ability to focus relevant tokens. All of Wx. by, uk. W.: by uy are learned parameters.
Entity representation generator To get representation for each non-avatar (non-player entity e, we embed its symbol into a query vector 9e of dimension d to attend to the descriptions E Z with their respective key and value vectors 12, We scaled dot-product attention (Vaswani et al. 2017) and denote the resulting representation for the entity as Te:
softmax ((2)) K2
j=1
where m [Z] is the number of descriptions in the manual. This mechanism allows EMMA to accomplish two forms of language grounding: the key and query select relevant descriptions for each object by matching entities names (e.g. mage), and the value extracts information relevant to the entities' behaviors in the world (e.g. enemy, chasing).
For each entity in the observation, we place its representation He into tensor X R exx the same coordinates as the entity position in the observation o to maintain full spatial information. The representation for the avatar (player entity) is simply learned embedding of dimension d.
Action Module To provide temporal information, we concatenate the outputs of the representation generator from the three most recent states to obtain tensor X' E x To get a distribution over the next actions (alo, 2), we run a 2D convolution on X' over the h, w dimensions. The flattened feature maps are passed through a fully-connected feed-forward network terminating in softmax over the possible actions.
y Flatten (Conv2D(X')) Taao, 2) softmax(FFN())
In contrast to previous approaches that use global observation features to read the manual (Zhong al. 2020), we build text-conditioned representation for each entity (x²). One advantage that Te can directly replace the entity embeddings typically used to embed the state observation in most models. This means EMMA's action module can easily swapped with existing models, such as txt2pi (Zhong et al., 2020) while still being completely end-to-end differentiable. Further details about EMMA and its design can be found in Appendix
EXPERIMENTAL SETUP
TASK
Motivation and Design We require domain in which grounding the text descriptions Z to dy- namics and learning the mapping M for all the entities in E is necessary to obtain a good reward. Moreover, there must be enough environments induce the mapping M.
4

---

Under review as a conference paper at ICLR 2021
With these requirements in mind, we devise new multi-task domain Messenger using the Py- VGDL framework (Schaul 2013). In Messenger, each entity can take on one three roles: a dangerous enemy, secret message, and crucial goal. The player's objective is to bring the message to the goal while avoiding the enemy. If the player encounters the enemy at any point in the game, or the goal without first obtaining the message, it loses the game and obtains reward of -1. Rewards of 0.5 and I are provided for obtaining and delivering the message to the goal respectively. Each of the enemy, message, and goal roles may be filled by one of twelve different entities, and each is assigned a stationary, chasing, or fleeing movement type to provide varying dynamics. Each set of entity-role assignments (henceforth referred to as a game) is initialized on 10 x 10 grid. The agent can navigate via up, down, left, right, and stay actions and interacts with another entity when both occupy the same cell. Some game examples are presented in Figure [
The same set of entities with the same movements may be assigned dif- ferent roles. Thus, two environments may have identical observations but differ in the reward function It (which is not available to the agent) and the text manual (which is available). Thus, our agent must learn to extract information from Z to succeed consistently.
human
robot
airplanc
thief
ship
sciemtist
queen
Grounding Entities via Interaction In previous work Zhong et al. 2020). every combination of entities is possibly observed during train- ing. Thus, for an entity c, its symbol in the observation (e.g. 8) is the only one that always appears together with its text symbol (e.g. queen). This co-occurrence provides strong inherent bias towards the correct grounding without needing to act in the environment. We denote these games in which each entity can appear with every other entity multi- combination (MC) games.
nature
dog
bird
fantasy
fish
mage
orb
sword
The MC assumption may not always be realistic in practice some entities are very unlikely to appear together (e.g. airplane, thief, sword) while others may co-occur exclusively with each other (e.g. mage, orb, sword). We denote games in which the same three entities always appear together as cingle-combinationn (SC) games.
Figure 3: Entities and their subdivision into hu- man, nature and fantasy sub-worlds. Each K₃ subgraph is a combina- tion of entities that may appear during training.
For SC games, any text symbol in the manual (e.g. mage, enemy, the, etc.) co-occurs the same number of times with all entities present, pro- viding the agent no information about M. Thus, the agent must ground these entities entirely via interaction. That is, the agent must align the behavior of the entities with its grounding of enemy, message, goal. For example, if the agent has the message, interacts with entity e and obtains a reward of 1, it must infer from the description "The mage is the goal" that e must be a mage (assuming it has grounded goal correctly).
We divide the entities in Messenger into human, nature and fantasy sub- worlds (Fig. 3 and exclude from training any games in which entities from different sub- aorld appear together. In particular, the nature and fantasy subworlds form SC games and the human subworld forms the MC games.
Text Descriptions Unlike previous work on lan- guage grounding in grid environments (Zhong et al.) et al.) 2019), we do not use templated or rule-generated text. We collected 3,881
the Hlyng machine remains still, and is also the note of upmost secrecy. the airplane is coming in your direction. that airplane is the pivitol target.
2020 Chevalier-Boissweeeeeeeeeeeeeree
unique free-form entity descriptions in English via Amazon Mechanical Turk (Buhrmester et al., 2016) by asking workers to paraphrase prompt sentences.
the winged creature escaping from you is the vi- tal target.
the flecing plan is a critical target.
To increase the diversity of responses, the prompts were themselves produced from 82 crowdsourced templates When constructing the prompts, we in- tentionally inject multiple synonyms for each entity. Workers often further paraphrased these synonyms, resulting in multiple ways to describe the same tity (e.g. airplane, jet, flying machine, aircraft. airliner etc.). Learning to map these different text symbols to the same entity is another unique challenge in Messenger.
Table Example deseriptions for Messen- ger. Note the use of synonyms, multiple sen- tences per description, typos (plane, plan) and the need to disambiguate similar words (flying machine, winged creature).
5

---

Under review as a conference paper at ICLR 2021
Each training manual consists of a set of three descriptions, for each enemy,, message. and goal entities. In total there are over 1.3 x 10 possible manuals each with an average length of 30 words. The total vocabulary size of the free-form descriptions is 1,016. Besides lower-casing the worker responses, we do not do any preprocessing. Example descriptions can be found in Table T Further details regarding data collection can be found in appendix
Evalu-Eval Split Each entity is trained on two out of three possible roles, with the third role reserved for validation and testing. This forces models to make compositional entity-role gen- eralizations for every entity to succeed on the evaluation games. In total we have 44 training, 32 validation, and 32 test games. We train on 2,557 of the text descriptions and reserve 652 each for validation and testing respectively.
Comparison with Previous Environments The primary focus of our work is on robust ground- ing of entities with rich language, variety of entity synonyms, in interactive setting without any priors or co-oceurrence biases to help induce this grounding. To this end, Messenger features numerous challenges not found in prior work.
In Narasimhan et al. (2018), an oracle used to concatenate the text representation to its corre- sponding entity representation. Access such oracle is strong assumption in the wild and eliminates the need to ground the entities altogether.
In RTFM ZZhng et al., 2020), the observation is grid of text which shares set of symbols with the manual and both manual and observations are embedded into the same space. key challenge unique Messenger is learning map between the observation and manual when they are embed- ded into different spaces, Furthermore, in RTFM all combinations of entities appear during training, which we explained earlier, provides an additional signal that may simplify the grounding prob- lem.
RTFM uses small number rule-based templates to construct each manual, and each entity is re- ferred to in a single way (e.g. goblin is always goblin). In contrast, Messenger features thousands of completely free-form descriptions and each entity may be referenced in multiple ways. For a more detailed comparison of RTFM and Messenger, including why we do not simply extend RTFM can be found in Appendix B.T
5.2 BASELINES
1) Mean-Bagg Bag of Sentences (Mean-BOS) This is variant of EMMA (our model) with the attention mechanism ablated. We average the value vectors obtained from equation 2 for each descriptor to obtain ö which is used by the action module.
C= y =Fllatte(Coov2D(EEill
(alo, Z) softmax (FFNn (5)
ZEZ
2) Game ID-Conditioned (G-ID) To assess the importance of lan- guage in our setup, we test a model with no language understanding Messenger. We provide an auxillary vector / of integer IDs that reveals the mapping between entity symbols and roles (Fig. 4. These integers are embedded and concatenated to form the vector which is used by the action module generate a distribution over the next actions:
Immovable
chooiar
Swing
goal type
Embed
y Flatten (Conv2((Emb(oo))
(6)
D1
concat
x (alo, Z) softmax (FFN
Figure 4: G-ID model
3) Bayesian Attention Module (BAM) This baseline uses a hard-attention mechanism with naive Bayes classifier trained to learn M. This approach is similar to a word alignment model used in machine translation approaches such as the IBM Model (Brown et al. 1993). Specifically, for some set of observed entities E' C E in the current environment:
C(t,) E) P(tle)
BAM(Z, E")
arg max P(e|2) 62
P(zee) II P((ee) (E2
C(1,e) (7)

---

Under review as a conference paper at ICLR 2021
where t are tokens in 2, t' is any token in the manual vocabulary and C refers to co-oceurence counts. We use Bayes' rule to flip the conditional. We let Ze = 8g from equation 2 for the 2 that maps to e. If two descriptions map to the same entity, we take the one with higher P(elz), and an entity receives no assignment we represent it with a learned default embedding Emb(e). We pretrain BAM on 1.5 10 episodes.
4) Oracle-Map (O-Map) To get a sense of the upper-bound on EMMA'S performance, we consider model that has access to the descriptor entity map M, similar to Narasiiinn et al. (2018). Thus, it is identical to EMMA except that the representation for each entity z is obtained as in equation g
Tee Σ (=) =|eu ZEZ
5.3 TRAINING AND EVALUATION
Curriculum Learning the entity groundings directly on Messenger is too difficult for the models consider. Our initial experiments revealed that EMMA could not learn entity grounding efficiently in this setting (Figure 5 (middle)). Thus, we introduce two-stage curriculum to train our models (Bengio et al. 2009). In stage 1 (S1), all entities begin two steps from the agent and are immovable. The agent either begins with or without the message and must interact with the correct entity. It is provided a reward of 1 if does so, and - otherwise. Models are first pretrained on $1, and all model parameters are transferred to stage 2 (S2), where entities are mobile and the agent always begins without the message. In each training game there one chasing, one fleeing and one immovable entity. On both S1 and S2 train models in multi-task fashion by sampling random game and appropriate manual at the start of each episode.
All models are end-to-end differentiable and train them using proximal policy optimization (PPO) (Schulman et al. 2017) and the Adam optimizer (Kingma & Ba 2014) with a constant learn- ing rate of 5 x 10 Additional details can be found in Appendix
Evaluation On S1 test games, the entities start in the same locations as the training games and are also immovable. Thus, model can apply the same policy used during training provided it can infer the (unseen) entity-role assignments, effectively testing the state estimation capabilities of models.
On S2 test games, we introduce new combinations of object movements to adaptation to new dynamics. Specifically, compared to training there are two chasing entities during testing instead of one. We also consider simpler state estimation test (S2-SE) where entity movements identical to those seen during training. Environment details can be found in Appendix B
6 RESULTS
G-ID
Mean-BOS
BAM
EMMA
O-Map
S1-All
89 3.3
90 7.2
81
88 2.3
97 1.3
6.1 MULTI-TASK PERFORMANCE
S1-MC
9.5
91
97 1:
88
97
S1-SC
8.7
90 6.8
51
1.4
96 0.6
Figures 5 (left and middle) show re- wards for both train and validation games as function of training steps. The advantage of textual understand- ing is clear; on both SI and S2, EMMA and the O-Map baseline converge to good policies much faster than the other baselines. However, EMMA trained directly on $2 with pretraining on S1 (EMMA-(no curriculum), Fig. 5) indicate that fitting directly onto $2 remains a challenge.
S2-All
3.6
2.1
69 4
95 0.4
96 - 0.8
S2-MC
0.7
2.9
1.4
85 0.9
96
1: 0.2
96
+: 0.4
S2-SC
3.9 1.5
2.4
0.6
22
# 0.5
Table 2: Win rates on training games. All denotes
overall win rates, MC and SC denote multi and single- combination respectively.
T details win rates on the training games, with a breakdown over single (SC) and multi com- bination (MC) games. We observe that the naive Bayes classifier can assign over 99% of training descriptors correctly ing, win rates are up to 60% lower. Our model (EMMA) can consistently win on both multi and single combination games, demonstrating EMMA's 's ability to ground entities using interaction and reward signals alone, without co-occurrences statistics between entity and text symbols to guide its grounding.
MC games. However, on SC games that require interactive entity ground-
7

---

Under review as a conference paper at ICLR 2021
Model
EMRA
G-Map
BAX
Mian 825
GID
ENKA-(19 curiiclumm
1.00
tram
1.3
2,0
1.5
0.50
1.0
0.25
0.5
0.00
0.5
provious
-0.75
-0.50
-0.5
-2.5
0.0
0.5
1.0
1.5
2.0
2.5
3.0
5
15 Train (K(10))
10
20
25
train (X110))
transfer (xTOP))
Figure 5: Average episodic rewards on S1 (left) and S2 (middle) on training (solid line) and valida- tion (dotted line) games, function of training steps (X-axis). Reward combination of both single and multi-combination games. EMMA-(nO curriculum) denotes EMMA trained directly on S2. We evaluate transfer performance of EMMA on games with novel entities and reward mechanics not found in Messenger (right). Our model trained on Messenger (transfer) learns the new games much faster than a model trained from scratch (scratch). All results are averaged over three seeds and shaded area indicates standard deviation.
G-ID
Mean-BOS
BAM
EMMA
O-Map
S1
18 8.2
6.7 2.7
66 1 1.5
85 + 1.4
97 0.3
6.2 GENERALIZATION
S2
1511.11
16414
39 238
79
82 32211 1.7 93 -3:0.4
S2-SE
0.1
1.7 0.7
47
90
1.8
New Entities To assess EMMA'S ability at picking up novel game me- chanics which are not specified in the provided text, we introduce two new stationary collectibles into Messen- ger ver-aa trap and gold which provide additional rewards of -1 and 1 respectively. We finetune EMMA the 32 validation games with these new entities. The model learns to collect the gold and avoid the trap while accomplishing the original objectives in Messenger (Figure (right)). (right)) Compared training from scratch, EMMA pretrained on the original Messenger games is able to achieve much higher reward in this modified setting in the same amount of steps, exceeding the previous maximum reward of 1.5 in 1 x 10 steps.
Table 3: Win rates on test games over three seeds. S1, S2 denotes stage one and stage two respectively. SE denotes state-estimation. EMMA generalize to unseen games and almost matches the O-Map model on $2.
Test games Results test games presented in Table Both the G-ID and Mean-BOR baselines fail to generalize in all cases. The G-ID baseline has complete access to distin- guishing information between games that is necessary to suc- ceed. However it overfits the entity-role assignments it has seen during training, resulting in poor test performance. BAM demonstrates some ability to generalize test games, but per- formance on games with dingle-combination entities are con- siderably lower, bringing the average down.
BAM
EMMA
Append Delete
34 0.2
75 + 1.7
20 1.1
36 + 1.8
Synonyms
11 0.5
72 3.1
Table 4:
Win rates S2 test games over three seeds for Append, Delete and Synonym cases.
In contrast, EMMA can win 85% and 79% of test games on SI and S2 respectively, almost match- ing the performance of the O-Map model. It also performs the best on the S2-SE test games. By extracting information from the relevant descriptor for each entity, EMMA is able to drastically sim- plify each task simply needs learn a policy for how to interact with enemy, messenger and goal archetypes instead of memorizing policy for each entity. This facilitates knowledge sharing between games, and generalization to unseen games.
6.3 ROBUSTNESS
Test-Time We first assess the robustness of trained BAM and EMMA models against text manual variations on S2 test games in table We test each model's ability to: (1) handle an extra descriptor for an entity not found in the game (Append), (2) reason about the role of objects without a descriptor by deleting a sentence from the input at random (Delete) and (3) generalize to unseen synonyms (Synonyms). For the last case, we use (unseen) templated descriptions filled in with entity synonyms not seen during training.
8

---

Under review as a conference paper at ICLR 2021
Both models can retain their performance when presented with an extraneous description, and suffer considerably when description is deleted. A key difference, however, is in our model's 's ability to generalize to unseen entity synonyms. EMMA wins almost 72% of games compared to 11% by the BAM model in this setting.
Train
Test
Train-Time We test the ability to learn entity groundings with added neutral entities and negated descriptions (Table
SI-Neu
92 1.0
88 0.7
S2-Neu
95 + 0.4
75 - 4.0
Neutral entities. At the start of each episode, we randomly select one of five neutral entities and insert into the obser- vation. The neutral entities are not described by the text, do not interact with the agent and provide no reward signal. The neutral entities are distinct from the objects in figure
S1-Neg
87 3.8
67 + 29
S2-Neg
88 + 8.8
58 + 28
Table 5: Percent win rates on train and test games for EMMA on the negation (Neg) and neutral (Neu) training cases on stage one (S1) and stage two (S2).
Negation. On each training episode with probability 0.25 we select description, negate it, and change the role. (e.g. the mage is an enemy becomes the mage is not the message).
This case forces the model to consider the roles of the other two entities deduce the role of the entity with the negated description. While EMMA can ground entities and performs well with neutral entities, it sometimes fails to ground the entities correctly with negated descriptions, affecting its performance on test games.
airptene robut
Swerr Orb
6.4 ANALYSIS OF GROUNDING
sirplane
robot
We visualize the attention weights for EMMA in Figure 6
ship
0.8
Our model is always trained on set of three descriptors, but to assess the overall latent mapping learned by our
queen
59 wo unguin11
scientist
thief
model, we evaluate the attention weights over 12 descrip- tions, one for every entity. EMMA able to place most weight for entity onto its descriptor Ze- In particular, EMMA learns a grounding for dog, bird, fish, mage, sword and e for which co-occurrence statisties pro- vide no meaningful alignment information, demonstrat- ing that our model can learn groundings for these entities via interaction alone. This also hints that the acquired grounding can enable EMMA to comfortably scale up environments containing larger sets of entities.
dog
bird
fish
mage
sword
entty description
Figure 6: Attention weights for EMMA computed from equation 3 Each row shows the attention weights for one tity over 12 randomly selected descrip- tors, each of which describe a separate entity indicated by the column label.
CONCLUSION
In this paper, we develop a new model, EMMA (Entity Mapper with Multi-modal Attention) to lever- age textual descriptions for generalization of control policies to new environments. Unlike prior work, we do not assume access to any form of prior knowledge connecting text and state obser- vations, and learn both symbol grounding and control policy simultaneously. EMMA employs a multi-modal ennity-oodiitioned attention module and learns a latent grounding of entities and dy- namics using only environment rewards. Our empirical results on a newly developed multi-task game framework with crowdsoureed text manuals demonstrate that EMMA shows strong generaliza- tion performance and robust grounding of entities. We hope that these results can lead to further research in enabling generalization for RL using natural language.
Studying entity grounding in interactive settings (with humans or other agents) will be important for agents that act in the world and automatically learn new con- cepts. While EMMA generalizes well to unseen games, its performance when additional reasoning is required (negation, deleted descriptions) is lower and warrants further study. Furthermore, training EMMA directly on Messenger without curriculum results in very poor performance. This demon- strates that learning stable entity groundings Messenger remains difficult problem with long trajectories and sparse rewards (training directly on S2) and when additional reasoning is required (negation). We leave this future work.
Applications and Future Work

---

Under review as a conference paper at ICLR 2021
REFERENCES
Jacob Andreas and Dan Klein, Alignment-based compositional semantics for instruction following. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pp. 1165-1174, 2015.
Jacob Andreas, Dan Klein, and Sergey Levine. Modular multitask reinforcement learning with policy sketches. ICMI., 2017.
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefen- stette. Learning to follow language instructions with adversarial reward induction. arXiv preprint arXiv:1806.01946. 2018.
Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In Proceedings of the 26th anmual international conference machine learning, pp- 41-48,4488 2009.
Valts Blukis, Yannick Terme, Eyvind Niklasson, Ross A Knepper, and Yoav Artzi. Learning to map natural language instructions to physical quadcopter control using simulated flight. arXiv preprint arXiv: 1910. 09664,, 2019.
SRK Branavan, David Silver, and Regina Barzilay. Learning win by reading manuals in a monte- carlo framework. Journal of Artificial Intelligence Research, 43:661-704, 2012.
Peter F Brown, Stephen A Della Pietra, Vincent J Della Pietra, and Robert L Mercer. The mathe- matics of statistical machine translation: Parameter estimation. Computational linguistics, 19(2):
263-311, 1993.
Michael Buhrmester, Tracy Kwang, and Samuel D Gosling, Amazon's mechanical turk: A new source of inexpensive, yet high-quality data? 2016.
David L Chen and Raymond J Mooney. Learning to interpret natural language navigation instruc- tions from observations. San Francisco, CA, pp. 859 865, 2011.
Maxime Chevalier-Boisse
Dzmitry Bahdanau, Salem Lahlou, Lucas Willems, Chitwan Saharia, Thien Huu Nguyen, and Yoshua Bengio. Al: First steps towards grounded language learning with a human in the loop. In International Conference on Learning Representations, 2019. URL
net htro///orr/////// iideidddrrrr 1ww.ddderrrecc.
John D Co-Reyes, Abhishek Gupta, Suvansh Sanjeev, Nick Altieri, Jacob Andreas, John DeNero, Pieter Abbeel, and Sergey Levine. Guiding policies with language via meta-learning. arXiv preprint 1.1.00/811.078822 2018.
Andrea F Daniele, Mohit Bansal, and Matthew R Walter. Navigational instruction generation as inverse reinforcement learning with neural machine translation. In 2017 12th ACM/IEEE Inter- national Conference Human-Robot Interaction (HRI, pp. 109-118. IEEE, 2017.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv: / 1810008805 2018.
Daniel Fried, Jacob Andreas, and Dan Klein. Unified pragmatic models for generating and follow- ing instructions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Compulational Linguistics: Human Language Technologies, Volume 1 (Long Pa- pers). pp. 1951-1963, New Orleans, Louisiana, June 2018. Association for Computational Lin- guistics. doi: 10.18653/v1/N18-I11 1177. URL https: www. acwweb. ezggantthoogy// N18-1177
Dave Golland, Percy Liang, and Dan Klein. A game-theoretic approach to generating spatial de- scriptions. In Proceedings of the 2010 Conference Empirical Methods in Natural Language Processing, pp. 410-419, Cambridge, MA, October 2010. Association for Computational Lin- guistics. URL //www//wwwwwww//////// wwwwwb.w.wwoooooec...eee........ www.ro/atggtggggnoioieeeooeeeeee 1040
Sachithra Hemachandra. Matthew R Walter, Stefanie Tellex and Seth Teller. Learning spatial- semantic representations from natural language descriptions and scene classifications. In Robotics and Automation (ICRA), 2014 IEEE International Conference on, pp. 2623-2633 IEFE, 2014.
10

---

Under review as a conference paper at ICLR 2021
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer. David Szepesvari, Wojciech Marian Czarnecki, Max Jaderberg. Denis Teplyashin, et al. Grounded lan- 2017.
guage learning in simulated 3d world. arXiv preprint arXiv:1706.0000
Felix Hill, Andrew Lampinen, Rosalia Schneider, Stephen Clark, Matthew Botvinick, James L Mc- Clelland, and Adam Santoro. Environmental drivers of systematicity and generalization a situated agent. In International Conference Learning Representations. 2020a.
Felix Hill, Sona Mokra, Nathaniel Wong, and Tim Harley. Human instruction-follloinn with deep reinforcement learning via transfer-learning from text. arXiv preprint arXiv:2205.009382 2020b.
Felix Hill, Olivier Tieleman, Tamara von Glehn, Nathaniel Wong. Hamza Merzic, and Stephen Clark. Grounded language learning fast and slow. arXiv preprint arXiv:2009.077/1 2020c.
Ronghang Hu, Daniel Fried, Anna Rohrbach, Dan Klein, Trevor Darrell, and Kate Saenko. Are you looking? grounding to multiple modalities in vision-and-lenn lnd-lagguage navigation. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 6551-6557, 2019.
Michael Janner, Karthik Narasimhan, and Regina Barzilay. Representation learning for grounded spatial reasoning. Transactions of the Association for Computational Linguistics, 6:49-61, 2018.
Diederik Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint 142.,6942 2014.
Heinrich Küttler, Nantas Nardelli, Alexander H Miller, Roberta Raileanu, Marco Selvatici, Edward Grefenstette, and Tim Rocktäschel. The nethack learning environment. In Advances in Neural Information Processing Systems, 2020.
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefen- stette, Shimon Whiteson, and Tim Rocktäschel. survey of reinforcement learning informed by natural language. International Joint Conferences on Artificial Intelligence, 2019.
Dipendra Misra, John Langford, and Yoav Artzi. Mapping instructions and visual observations to actions with reinforcement learning. arXiv preprint 174.007704..8755, 2017.
Karthik Narasimhan, Regina Barzilay, and Tommi Jaakkola. Grounding language for transfer in deep reinforcement learning. Journal of Artificial Intelligence Research, 63:849-874, 2018.
Junhyuk Oh, Satinder Singh, Honglak Lee, and Pushmeet Kohli. Zero-shot task generalization with multi-task deep reinforcement learning. arXiv preprint wwX.vv:706.005665 2017.
Hilke Reckman, Jeff Orkin, and Deb Roy. Learning meanings words and constructions, grounded in virtual game. Semantic Approaches in Natural Language Processing, pp- 67, 2010.
Tom Schaul. A video game description language for model-based interactive learning. In Com- putational Intelligence in Games (CIG), 2013 IEEE Conference on, pp. 1-8. IEEE, 2013.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347 2017.
Stefanie Tellex, Nakul Gopalan, Hadas Kress-Gazit, and Cynthia Matuszek. Robots that use lan- guage. Annual Review of Control, Robotics, and Autonomous Systems, 3:25-55, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Infor- mation Processing Systems, pp. 5998-6008, 2017.
Matthew R Walter, Sachithra Hemachandra, Bianca Homberg, Stefanic Tellex, and Seth Teller. Learning semantic maps from natural language descriptions. Robotics: Science and Systems, 2013.
Xin Wang, Qiuyuan Huang. Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang. and Lei Zhang. Reinforeed cross cross-modal matching and self-supervised imita- tion learning for uision-languaa navigation. In Proceedings of the IEÉE Conference on Computer Vision and Pattern Recognition, pp. 2019.

---

Under review as a conference paper at ICLR 2021
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R' e'mm Louf, Morgan Funtowicz, and Jamie Brew. Huggingface trans- formers: State-of-thee thrtt art natural language processing. ArXiv, abs/1910.03771, 2019.
Victor Zhong, Tim Rocktäschel, and Edward Grefenstette. Rtfm: Generalising to new environment dynamics via reading. In International Conference on Learning Representations, 2020.
A
TEXT MANUAL
Example Input
The bird that is coming near you is the dangerous enemy.
The secret message is in the thief's hand as he evades you.
The immovable object is the mage who holds goal that is crucial.
Enemy Descriptions
Adjectives: dangerous, deadly, lethal
Role: enemy, opponent, adversary
Message Deseriptions
Adjectives: restricted, classified, secret
Role; message, memo, report
Goal Descriptions
Adjectives: crucial, vital, essential Role: goal, target, aim
Table 6: Example template descriptions. Each underlined word in the example input indicate blanks that may be swapped in the template. Each template takes a word for the object being described (bird, thief, mage), its role (enemy, message, goal) and an adjective (dangerous, secret, crucial).
To collect the text manual, we first crowdsource 82 templates (with 2,214 possible descriptions af- ter filling in the blanks). Each Amazon Mechanical Turk worker is asked paraphrase prompt sentence while preserving words in boldface (which become the blanks in our templates). We have three blanks per template, one each for the entity, role and an adjective. For each role (enemy, mes- sage, goal) we have three role words and three adjectives that are synonymous (T) Each entity also described in three synonymous ways. Thus, every entity-role assignment can described in 27 different ways the same template. Raw templates are filtered for duplicates, converted to lowercase, and corrected for typos to prevent confusion on downstream collection tasks.
To collect the free form text for specific entity-role assignment, we first sample random template and fill each blank with one of the three possible synonyms. The filled template becomes the prompt that is shown to the worker. Aside from lower-casing the free-form descriptions, do no further pre- pre-processing.
On all tasks (template and free-form) provide example prompt (which is distinct from the one provided) and examples to provide additional task clarity. For each prompt, we obtain two distinct paraphrased sentences to promote response diversity. To ensure fluency in all responses, workers to those located in the United States with at least 10,000 completed HITSs and an acceptance rate of > 99%. On all data collection tasks, we limit single worker a maximum of 36 responses to ensure worker diversity. In total, 297 workers participated in the data collection process.
limited
B ENVIRONMENT DETAILS
Details about Messenger can be found in table 7 On stage (SI), the three entities start randomly in three out of four possible locations, two cells away from the avatar. The agent always begins in the center of the grid. It starts without the message with probability 0.8 and begin with the message otherwise. When the avatar obtains the message, symbol in the observation. On S1. we limit each episode to four steps and provide a reward - the agent does not complete the objective within this limit.
capture this information by changing the avatar
12

---

Under review as a conference paper at ICLR 2021
Objects
bird, dog, fish, scientist, queen, thief, airplane, robot, ship, mage, sword, orb
Roles
enemy, message, goal
Movements
chasing, fleeing, immovable
Total games
3(12.3) = 1320
Total variants
1320 x 3! 7920
Initial States/variant
24
Table 7: Basic information about our domain Messenger. Each game features out of 12 possible non-player entities, each assigned a role of enemy, message goal. Each training game has 3! variants corresponding to the assignment of chaser, fleeing and immovable movement types to each entity.
On stage 2 (S2), the avatar and entities are shuffled between four possible starting locations at the start each episode. On S2, the mobile entities (fleeing, chasing) move at half the speed of the agent. On S2, we limit each episode to 64 steps and like in SI, we provide reward of -1 if the agent does not complete the objective within this limit.
Since there only 4 single combbnicomiinnioon (SC) training games and 40 multi- cultiicomminatioon (MC) training games, we sample the games non-uniformly at the start each episode to ensure that there is enough interaction with SC entities to induce entity grounding. On both SI and S2, sample an SC game with probability 0.25 and an MC game otherwise. Not all descriptions have movement type information (e.g. "chasing"). We also collect unknown type descriptions with movement type information. During training, each description is independently an unknown type description with probability 0.15
Negation We procedurally generate the negated text by negating existential words (e.g. "is an en- emy" becomes "is not an enemy"). We manually negate those descriptions not captured by the rules. During both training and evaluation, scription with 0.75 probability, and randomly select a description in the manual to negate otherwise. When we negate "...i not a goal", for example). Thus the information present in the manual has not changed, but the agent must look at the remaining two descriptions to deduce the role of with description 2/--
provide complete text manual without any negated de-
entity description 2e to z/x, also change the role ("....i enemy" becomes
Transfer Learning We test transfer by introducing two new entities - trap and a gold which provide rewards of -1 and respectively. Both collectables are randomly shuffled between two possible starting locations at the start of each episode and do not move. We train the models in this new setting in a multi-tass task fashion on the 32 validation games. After the agent encounters either the trap or gold, the collected item disappears. Neither item terminates the episode and the agent can still win or lose the current episode regardless of whether it has picked up the gold or trap.
B. COMPARISON WITH RTFM
The main novelty of our work (both the Messenger environment and our model) is in specifically tackling the issue of entity grounding without any prior knowledge. To do this, Messenger in contrast RTFM (1) does not have any signal connecting entities to text symbols, (2) features much richer language, and (3) requires interaction in the environment to ground entities to text. We describe these in more detail:
1. RTFM's observation space consists of a grid of text which shares set of symbols with the text manual. Thus, both the text in the manual and the observation are embedded into the same space (e.g. using the same word vectors), essentially providing models with the entity grounding upfront. In contrast, our environment has a separate set of symbols for the entities with relation the text in our manual. Thus, the entities and text embedded into different spaces, and learning to map between these two spaces is the key challenge in our environment that has not been explored before.
2. RTFM features only 32 total rule-based templates for the text, and each entity can only be referred to in a single way (goblin is always goblin). In contrast, we crowdsourced thousands of completely free-form descriptions in two rounds using Amazon Mechanical
13

---

Under review as a conference paper at ICLR 2021
Turk. After obtaining the seed templates from the first round, intentionally inject mul- tiple synonyms for each entity to construct each prompt for the second round. Workers often further paraphrased these synonyms, resulting in 5, 6 or often more ways to describe the same entity (e.g. airplane, jet, flying machine, aircraft, airliner etc.). The need to map these different text symbols the same entity further complicates the entity ground- ing problem in our case and more closely mirrors the challenges of grounding entities in the real world. We believe Messenger provides a much closer approximation "natural" language compared to RTFM.
3. RTFM features all possible combinations of entities during training, which as we explained in Sec. 5.1 provides an additional signal that may simplify the grounding problem.
4. Furthermore, each entity in RTFM only moves in a single way, whereas in Messenger, each entity may have different dynamics such as fleeing, chasing, and immovable entities (and this also described in the text). This also allows us to test our model's ability to generalize to unseen dynamics with unseen entity movement combinations, whereas in RTFM the evaluation on unseen games is essentially state-estimation.
Messenger is quite similar to RTFM and shares several aspects (e.g. grid-world with different entities and goals). That said, there are numerous reasons why we were not able to adapt the original RTFM environment to meet our requirements. We enumerate them here:
1. The dynamics in RTFM make entity grounding (the primary focus of our work) difficult. Messenger requires much simpler reasoning than RTFM, and it is already too difficult to ground entities directly in Messenger without a curriculum. RTFM sidesteps the issue by providing this grounding beforehand.
2. Obtaining enough crowdsourced descriptions is hard with RTFM because of the more com- plicated dynamics. In RTFM, there are monsters, weapons, elements, modifiers, teams, variable goals and different weaknesses between entity types that need to be specified. Col- lecting enough descriptions that are entirely human written would be challenging. (RTFM sidesteps this issue by using templates generate their text manual). In contrast, there only entities, 3 roles, and a fixed goal in Messenger, making the text-collection task much more tractable.
3.
The entities in our Messenger environment are carefully chosen to make entity grounding harder. In RTFM, each entity is referred to in a single way, and it is not clear how to refer to them in multiple ways (e.g. there are not too many other ways to say goblin). In contrast, we specifically chose a set of entities that allowed for multiple ways of description, and actively encouraged this during data collection.
4. The combination of entities that appear during training in Messenger is carefully designed. This is so that we can introduce ongle-combination games and the associated grounding challenges that come with it.
5. We have different movement types for each entity. These different movements are referred to in our text manual and significantly increase the richness and variety of descriptions we collected, and also allow us to test generalization to unseen movement combinations. In RTFM, the entity movements are the same and fixed for all entities.
6. Each entity's attribute is referenced in the observation in RTFM, e.g. the grid has entries such as fire goblin. We could add to the cell an extra symbol for fire, but this further obfuscates the entity grounding problem we are focusing on, because would also need to obtain a grounding for all the attributes such as fire.
C IMPLEMENTATION DETAILS
For all experiments we use 256. When multiple entities F' overlap in the observation, we fill the overlapping cell with the average of the entity representations 1 [ Σ Zeeg Te- The convolutional layer consists of 2 kernels with stride and 64 feature maps. The FFN in the action module is fully-connected with 3 layers and width of 128. To give the Mean-BON BOS and G-ID baselines the ability to handle the additional conditioning information, we introduce an additional layer of width
14

---

Under review as a conference paper at ICLR 2021
512 the front of the FFN for those baselines only. Between each layer, leaky ReLU the activation function. All models were trained for a maximum of x 10 episodes on S1 and 36 hours on S2. All experiments were conducted on single Nvidia RTX2080 GPU.
D MODEL DESIGN
The weights ug and u, were introduced to make sure that the token embeddings for filler words such as the, and, or do not drown out the words relevant to the task when we take the average in equations observe that up learns focus tokens informative for identifying the entity (e.g. mage, sword) while up learns to focus on tokens that help identify the entities' roles (e.g.
1 and Qualitatively,
enemy, message).
We also found that using a pretrained language model was critical for success due to the large number ways to refer single entity (e.g. airplane, jet, flying machine, aircraft, airliner etc.).
D.1 MODEL VARIATIONS
We consider variation to EMMA. Instead of obtaining token weights 3 in equations | and by taking softmax over the toben-cmb and vector products uy t and uh t, we consider independently scaling each token using sigmoid function. Specifically, will obtain key and value vectors k, and v2 using:
o oup ti) ouu 1;)
k-
Wkts bk
(uyy ti)
2/ -i=1 ouu (2)
Wot, + b.
(10)
iml
where is the sigmoid function, and all other details are identical to EMMA. We call this model EMMA-O, We notice that both EMMA and EMMA-o are able to obtain good training and validation performance, whith EMMA-0 obtaining higher rewards on S2. However, on S1, EMMA is able to obtain higher validation reward faster (Fig. [7]. Moreover, EMMA can learn robust groundings with neutral entities, while EMMA-o often overfits to spurious grounding with neutral entities (Fig. S). Although the independent scaling EMMA-o allows the model to consider more tokens simultaneously. the softmax selection of EMMA facilitates more focused selection of relevant tokens, and this may help prevent overfitting.
15

---

Under review as a conference paper at ICLR 2021
0.75
1.0
0.50
reward
0.25
0.5
0.25
EMMA
0.50
train
val
-1.00
10
15
20
25
train steps (x10%)
train steps (x10%)
Figure 7: Average episodic rewards on S1 (left) and S2 (right) on training (solid line) and validation (dotted line) games, function of training steps (x-axis) for both EMMA and EMMA-O. Both models are able to perform well, however, EMMA is able to obtain a good validation reward faster. All results averaged three seeds and shaded area indicates standard deviation.
0.75
0.50
0.50
0.25
0.25
0.00
0.00
Model
-0.25
-0.5
EMMA
EMMA-o
0.50
Games
train
vall
train (x10%)
train (x10f)
Figure 8: Average episodic rewards S1 games with negation (left) and neutral entities (right) on training (solid line) and validation (doted line) games, as function training steps (x-axis) for both EMMA and EMMA-O. Both models struggle negation, but EMMA is able perform well with neutral entities. All results are averaged over three seeds and shaded area indicates standard deviation.
16


Reviews:
[{"review_id": "ml43hct0nqq", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper is studying the problem of learning to interpret manuals / textual information about the task, with the goal of faster learning and generalization in RL. Unlike recent prior work (Narasimhan et al 2018, Zhong et al 2020) where the entities in the environment are already partly grounded to text (e.g. by representing them as their textual description), the agent here needs to learn the mapping between entities and corresponding text. In order to study this problem, the authors use a new environment and dataset of natural language descriptions, as well as propose a self-attention model that matches entities to relevant sentences. The proposed model performed comparable to a partly grounded policy (as in Narasimhan et al 2018).\n\nReasons for score:\n\nI find the topic interesting and relevant, and the use of human-generated descriptions (in contrast to procedurally generated manuals as in Zhong et al 2020) is welcomed. However, there is relatively little in terms of novelty both in the proposed model and the environment. I think the paper would benefit from identifying and focusing on one or two of the more specific issues and exploring them in-depth.\n\nMore specifically: \n(1) The proposed model is a fairly straightforward application of self-attention to text, which has been similarly used (albeit with BiLSTMs) in Zhong et al 2020. \n(2) While the challenge of needing to learn which entities map onto which words is greater, each sentence refers to exactly one entity, so the agent just needs to match 3 sentences to 3 entities, which is a marginally greater challenge (if at all). \n(3) With slight modifications, the environment proposed in Zhong et al 2020 should be suitable for addressing the main topic of this paper, why introduce a new environment? \n\nQuestions and comments: \n- there should be a more detailed comparison to the two of the closest works (Narasimhan et al 2018, Zhong et al 2020), i.e. how exactly is your model and environment different?\n- the MDP notation for transition and reward function is imprecise, actions are a random variable in both expressions (i.e. use r(s, a, s'))\n- why is the curriculum introduced, is it necessary to solve the task?\n- the use of the term _validation_ here is confusing, it seems you are both interested in zero-shot (on test games) and transfer learning performance (which is referred to as validation)?  \n- in Fig 5 right, why is learning from scratch so much worse? Shouldn't learning from scratch be as easy as in Fig 5 right/middle?  \n\n------------------------------------\n\nUPDATE:\nAfter reading the author's response, the reviewer's comments, and the revised version, I have increased my score. \nSee my response to the authors for more comments.   ", "extra_scores": {}, "extra_text": {}}, {"review_id": "_hft6YJJHjm", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a model for entity grounding from its textual description for a text-based language game. To test their new model named EMMA (Entity Mapper with Multi-modal Attention), they also present a new toy game framework and crowdsourced data composed of 1320 games (from 3,881 entity descriptions with crowdsourcing). In this game, the agent has to associate an entity with its description. The game is described in Figure 1 and section 5.1: there are three entities to each game (messenger, enemy, goal), and each entity can be stationary, chasing, or fleeing movement. The game has a one-sentence description for each entity (if I’m interpreting it correctly). The space of action is up, down, left, right, and stay. The goal of the player is always static — bring a message to the goal while not touching the enemy. From what I understand, the game here itself is substantially simpler than that of previous work (Zhong et al 20), which involves varying goals and modifiers. \n\nTheir architecture deviates from existing approaches (Zhong et al 20, Narasimhan et al 18) which did not require the model to learn a mapping from an entity to its description either by providing a mapping between objects and their textual descriptions or using entity names plainly. This is the main focus of the model — to learn a mapping between entities and their descriptions. The evaluation is also done in a manner that between train/validation/testing there’s no overlap between entity - role combination. \n\nThe model architecture is pretty standard, using a bert-base encoder with attention. Figure 2 clearly explains their model, where each character takes a weighted sum of descriptions as value vectors to decide the action. \n\nThe game feels a bit retrospectively designed to test the research idea. It would have been a lot more convincing if they use their model on the existing game environment, instead of creating a whole new game specifically designed to test their idea. The paper is clearly written and easy to follow, except for some parts (BAM parts weren’t easy to follow). \n\nThe paper presents nice studies throughout section 6, looking into learning in a more challenging evaluation set up such as generalization and supplementary material is also thorough.\n\nFor Figure 5, graphs are on the union of single and multi-combination games? \nHow much does it help to use pre-trained BERT encoder vs. training from scratch? \n\nI’m not sure what future work can be done in this space. The model seems to be doing well in this constrained setting (Table 2). Could the authors provide how this model can be applied to more challenging and complex game scenarios, where there's no one-to-one mapping between entity description and an entity, and the goal is more complex?", "extra_scores": {}, "extra_text": {}}, {"review_id": "R0BaqDgsHi", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "## Summary\nThis is a significantly improved extension of prior work in this area. The authors significantly improve upon the previous techniques: see below. I have a few questions on clarity but overall the paper was well-presented.\n\n## Quality & Clarity\nThe paper is generally fairly clear. There are a lot of details in this paper and some of them did not quite feel fully developed (understandable given the space constraints), such as justification for certain modeling choices.\n\nFor example, in section 4 under “Entity representation generator”, the authors say:\n>  For each entity e in the observation we place its representation xe into a tensor X at the same coordinates as the entity position in the state s to maintain full spatial information\nThis implies to me that tensor X is a 2D grid matching the coordinate system. The next section says that the three most recent observations are concatenated together and a 2D convolution is run over them. Which of the three potential dimensions are used in the 2D convolution and which is flattened: the x/y coordinates or the time dimension?\n\nIn Section 4: Model, Text Encoder, you say that “These [alpha and beta] weights imbue our model with the ability to focus on relevant tokens”. Can you provide an intuition for what u_k and u_v are learning?\n\nIn Section 6.1: Multi-task Performance, I was not quite able to grasp why the single-combination tasks are more difficult. Can you explain the difference between these two kinds of tasks a bit more?\n\nOverall though, I was able to easily understand the main points of the paper and the data was clearly visualised in graphs and tables.\n\n## Originality & Significance\nThere are a lot of good contributions in this paper:\n- natural language crowd-sourced manuals with two layers of “turk-indirection”: not only was the text crowdsourced but first the templates for the texts were crowdsourced\n- a comparison of several different techniques with different attention mechanisms\n- experiments with additional evaluations of generalisation and robustness, including more difficult test scenarios, adding neutral distractor entities, fine-tuning with additional sources of punishment/reward, negating the text, and replacing entities with synonyms\n- a visualization of attention to demonstrate that the attention mechanism is working properly\n\nThese are original and significant contributions that go beyond what was done prior.\n\n**Suggestion**: While I didn’t understand the difference in difficulty between the multi-combination / single-combination games, what I did hope to see when they were presented was an evaluation of how the agent was able to transfer learning from one of these enabled transfer to the other (e.g. learning in the fantasy setting transferred to the nature setting). This would demonstrate that the kind of structure present in the tasks is learned and can be applied to different kinds of entities / a different vocabulary.", "extra_scores": {}, "extra_text": {}}, {"review_id": "KfAklaShlhV", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 1, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": " \nNatural language grounding is an interesting research direction and has attracted many researchers in recent years. Previous work mainly considered grounding the text to image objects. This paper considers collaboratively to learn “entity” representations and natural language explanations with a reinforcement learning framework. Specifically, a multi-modal attention network is proposed to model the interaction between the entity representation and the text descriptions. The entire framework is trained over multiple games in a multi-task manner. Experiments are conducted on a newly designed benchmark. The proposed RL framework achieves reasonable performance in domain games (training & test are from the same games) and also has a strong zero-shot generalization to unseen games and \"entities\" (thanks to the parameter sharing and multi-task learning). Besides, the newly released dataset may facilitate future research in natural language grounding.\n\n I am not familiar with the natural language grounding literature, so this might be an educated guess, which is listed as follows,\n\n\n- The writing is relatively clear, although some specific illustrations are sometimes hard to follow. For example, the term “entity” is very ambiguous. In the NLP field, an entity might relate to a phrase that has specialized semantic meaning. I read several recent papers in the natural language grounding field but still puzzled about the term (for example, the semantic meaning of “entity” in  [2] is different from this paper). \n \n- After reading this paper, I am not sure which part is the main novelty. The proposed reinforcement learning framework is relatively standard (no specialized designed state, policy, and reward). The interaction module is designed in a slightly straightforward way. Utilizing parameter sharing to encourage stronger generalization ability also seems reasonable. The novelty might be further clarified.\n- The proposed approach seems to be similar to [1] (cited). The general idea is to learn a parameterized policy model that inputs the pair (entity [this paper] or visual representation [1] and text) and outputs the action of the agent in the game (correct me if I am wrong). Depending on different scenarios, the encoding network can be slightly different. It might be beneficial to have a detailed discussion of the difference between the methods proposed in the two papers.\n- In Figure 5, the reward of different approaches seems to have small differences. However, the final winning rate of games varies a lot. It might be beneficial to have some discussions on this part.\n- If my understanding is correct, the contribution of the paper is mainly focused on the strong generalization ability of the proposed approach across different games. However, it may need to clarify which part of the proposed approach contributes to the most significant influence. \n\nOther minor points:\n - I am interested in the O-Map baseline, which seems to be a very good upper bound of the proposed approach, I am surprised to see that the EMMA approach outperforms the O-Map in the training settings. Some intuitive explanations might be useful. The second question is that, though utilizing the upper bound (O-Map), the winning rate still has some room to improve. It is suggested to have some further analysis.\n - I do not quite understand why the overall training process contains two rounds of stages. Is S1 stage training a warm-up training?\n\nI would like to hear the authors’ responses to make my decision.\n\n[1] Zhong et al. RTFM: Generalising to New Environment Dynamics via Reading. ICLR 2020\n\n[2] Lai et al. Contextual Grounding of Natural Language Entities in Images. NeurIPS 2019 workshop\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "FE0EmnU2dFS", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper considers the task of training an agent to act following a manual expressed in natural language. The manual describes the roles and the behaviors of the entities in the environment. Each entity can be the goal, the message or the enemy and it can also be either fleeing, chasing or not moving. The agent has to bring the message to the goal while avoiding the enemy. A model called EMMA is proposed to change entity representations based on the manual, thereby making the agent aware of the entities’ roles. It is shown that EMMA is more effective than simple baselines.\n\nThe paper is mostly clearly written.  The proposed model makes a lot of sense for the considered task. The paper can be seen as another proof-of-concept paper for acting based on a manual, but it should be noted that it is not the first one of its kind [1]. \n\nSome things were not entirely clear to me.\n\n- In several places in the paper it is mentioned that the model learns “mapping between entity IDs in observation space and their symbols in text entirely through *interaction* with the environment.” I am not sure what interaction means here. Looking at the formulas, my understanding is that EMMA establishes a correspondence between the observed object symbols and the entities based on the entities’ descriptions. What role does interaction play in determining the object-role mapping? Same question about a similar sentence in Paragraph “Multi-Task Setup” of Section 5.1.\n- Why is G-ID baseline performing so poorly on validation and test data? Doesn’t it have the complete access to the roles and behaviour of every object?\n- Why is the paper not using a setup from the prior work, e.g. from [1]? \n\nWhile the paper is clear and seems to be very accurately and correctly executed, I have concerns (which might be considered subjective) about the limited real-world impact of this kind of work. The paper does not discuss how this work is related to any real world RL applications, such as e.g. robotics or even building RL agents for games that people actually play. It does not discuss what the manual would be in these cases. Would it look anything like the description of the objects’s roles? It appears common in the field of RL to not ask for such justifications and happily accept gridworld studies motivated by generalization to “new environments”. But I would like to flag that for an outsider this disconnect from reality may seem troublesome. \n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: Like the reviewers, I find this paper extremely borderline. On the one hand, it is clearly written, about a topic I find fascinating, and generally well motivated if not shockingly novel (i.e. removing some of the simplifying assumptions from Zhong et al. 2020, e.g. requiring grounding to be learned, use of real language rather than synthetically generated). On the other hand, I agree with the leitmotiv present amongst the reviews that the problem at the centre of the experimental setting is very, very simple (3 objects, 3 descriptions). I am mindful of the fact that access to computational resources is unevenly distributed, and am not expecting a paper like this to immediately scale their experiments to highly complex settings with photorealism, etc, but I can't help but feel that a more challenging task, with a deeper analysis of the problems presented by both grounding and the use of non-synthetic language, would both have been highly desirable to make this paper uncontroversially worth accepting.

As a result, the decision is to not accept the paper in its present form. Work on this topic should definitely be presented at ICLR, but it's a shame this paper did not make a stronger case for itself.
Decision: Reject