RTFM: Generalising to New Environment Dynamics via Reading
Authors: Victor Zhong
Tim Rocktäschel
Edward Grefenstette
Venue:  (2020)
Keywords: reinforcement learning
policy learning
reading comprehension
generalisation

Abstract: Obtaining policies that can generalise to new environments in reinforcement learning is challenging. In this work, we demonstrate that language understanding via a reading policy learner is a promising vehicle for generalisation to new environments. We propose a grounded policy learning problem, Read to Fight Monsters (RTFM), in which the agent must jointly reason over a language goal, relevant dynamics described in a document, and environment observations. We procedurally generate environment dynamics and corresponding language descriptions of the dynamics, such that agents must read to understand new environment dynamics instead of memorising any particular information. In addition, we propose txt2π, a model that captures three-way interactions between the goal, document, and observations. On RTFM, txt2π generalises to new environments with dynamics not seen during training via reading. Furthermore, our model outperforms baselines such as FiLM and language-conditioned CNNs on RTFM. Through curriculum learning, txt2π produces policies that excel on complex RTFM tasks requiring several reasoning and coreference steps.

Published as a conference paper at ICLR 2020
RTFM: GENERALISING TO NOVEL ENVIRONMENT DYNAMICS VIA READING
Victor Zhong* Paul G. Allen School of
Tim Rocktäschel
Edward Grefenstette Facebook Al Research & University College London egrefen cb.com com
Facebook Al Research & University College London rockt fa.ktffb.com com
Computer Science & Engineering University of Washington vzhengêes, washington.com edu
ABSTRACT
Obtaining policies that can generalise to new environments in reinforcement learn- ing is challenging. In this work, we demonstrate that language understanding via a reading policy learner is a promising vehicle for generalisation to new envi- ronments. We propose grounded policy learning problem, Read to Fight Mon- sters (RTFM), in which the agent must jointly reason over language goal, rele- vant dynamics described in a document, and environment observations. We pro- cedurally generate environment dynamics and corresponding language descrip- tions of the dynamics, such that agents must read to understand new environ- ment dynamies instead of memorising any particular information. In addition, we propose txt2n, a model that captures three-way interactions between the goal, document, and observations. On RTFM, txt2m generalises new en- vironments with dynamics not seen during training via reading, Furthermore. our model outperforms baselines such as FiLM and aunguage-conditione CNNs on RTFM. Through curriculum learning, txt produces policies that excel on complex RTFM tasks requiring several reasoning and coreference steps.
INTRODUCTION
Reinforcement learning (RL) has been successful in variety of areas such continuous trol (Lillicrap et al., 2015), dialogue systems (Li et al., 2016), and game-playing (Mnih et al., 2013). However, RI. adoption in real-world problems is limited due poor sample efficiency and failure to generalise to environments even slightly different from those seen during training. We explore language-conditione policy learning, where agents use machine reading discover strategies re- quired to solve a task, thereby leveraging language as a means to generalise to new environments.
Prior work on language grounding and language-bassee based RL (see Luketina al. (2019) for a recent survey) are limited to scenarios in which language specifies the goal for some fixed environment dynamics (Branavan et al., 2011; Hermann et al., 2017; Bahdanau al., 2019; Fried et al., 2018; Co-Reyes et al., 2019), or the dynamics of the environment vary and are presented in language for some fixed goal (Branavan et al., 2012). In practice, changes to goals and to environment dynamics tend to occur simultaneoully-giieee some goal, we need to find and interpret relevant information to understand how to achieve the goal. That is, the agent should account for variations in both by selectively reading, thereby generalising environments with dynamics not seen during training.
Our contributions are two-fold. First, we propose a grounded policy learning problem that we call Read Fight Monsters (RTFM). In RTFM, the agent must jointly reason over language goal, a document that specifies environment dynamics, and environment observations. In particular, must identify relevant information in the document shape its policy and accomplish the goal. To necessitate reading comprehension, we expose the agent to ever changing environment dynam- ies and corresponding language descriptions such that it cannot avoid reading by memorising any particular environment dynamics. We procedurally generate environment dynamics and natural lan- guage templated descriptions of dynamics and goals produced combinatorially large number of environment dynamics to train and evaluate RTFM.
Work done during intersship Facebook AT Research.

---

Published as a conference paper at ICLR 2020
Second, we propose txt22 to model the joint reasoning problem in RTFM. We show that txt2 generalises to goals and environment dynamics not seen during training, and outper- forms previous -anguage-condttione models such as language-conditioee CNNs and FiLM (Perez et al., 8; Bahdanau et al., 2019) both in terms of sample efficiency and final win-rate on RTFM. Through curriculum learning where we adapt 2xt trained simpler tasks to more complex tasks, we obtain agents that generalise to tasks with natural language documents that require five hops of reasoning between the goal, document, and environment observations. Our qualitative anal- yses show that txt2n attends to parts of the document relevant to the goal and environment ob- servations, and that the resulting agents exhibit complex behaviour such as retrieving correct items, engaging correct enemies after acquiring correct items, and avoiding incorrect enemies. Finally, we highlight the complexity of RTFM in scaling to longer documents, richer dynamics, and natural language variations. We show that significant improvement in iaguage-grounder policy learning is needed to solve these problems in the future.
2 RELATED WORK
Language-conditioned policy learning. A growing body of research is learning policies that fol- low imperative instructions. The granularity of instructions vary from high-level instructions for application control (Branavan, 2012) and games (Hermann et al., 2017; Bahdanau et al., 2019) to step-by-step navigation (Fried et al., 2018). In contrast to learning policies for imperative instruc- tions, Branavan et al. (2011; 2012); Narasimhan et al. (2018) infer policy for fixed goal using features extracted from high level strategy descriptions and general information about domain dy- namics. Unlike prior work, we study the combination of imperative instructions and descriptions of dynamics. Furthermore, we require that the agent learn to filter out irrelevant information to focus on dynamics relevant to accomplishing the goal.
Language grounding. Language grounding refers to interpreting language in a non-linguistic con- text. Examples of such context include images (Barnard & Forsyth, 2001), games (Chen & Mooney, 2008; Wang et al., 2016), robot control (Kollar et al., 2010; Tellex et al., 2011), 1), and navigation (An- derson et al., 2018). We study language grounding in interactive games similar to Branavan (2012) 2); Hermann et al. (2017) or Co-Reyes et al. (2019), where executable semantics are not provided and the agent must learn through experience. Unlike prior work, we require grounding between un- derspecified goal, a document of environment dynamics, and world observations. In addition, we focus on generalisation to not only new goal descriptions but new environments dynamics.
READ TO FIGHT MONSTERS
We consider a scenario where the agent must jointly reason over a language goal, relevant envi- ronment dynamics specified in text document, and environment observations. In reading the document, the agent should identify relevant information key to solving the goal in the environment. A successful agent needs to perform this language grounding generalise to new environments with dynamics not seen during training.
To study generalisation via reading, the environment dynamics must differ every episode such that the agent cannot avoid reading by memorising a limited set of dynamics. Consequently, we proce- durally generate large number of unique environment dynamies (e.g. effect ive (blessed i items, poison monsters) ), along with language descriptions of environment dynamics (e.g. blessed items are effective against poison monsters) and goals (e.g. Defeat the order of the forest). We couple a large, customisable ontology inspired by rogue-like games such as NetHack or Diablo, with natural language templates to create a combinatorially rich set of environment dynam- ics to learn from and evaluate
In RTFM, the agent is given a document of environment dynamics, observations of the environment, and underspecified goal instruction. Figure illustrates an instance of the game. Concretely, we design a set of dynamics that consists of monsters (e.g. wolf, goblin), teams (e.g. Order of the For- est), element types (e.g. fire, poison), item modifiers (e.g. fanatical, arcane), and items (e.g. sword, hammer). When the player is in the same cell with a monster or weapon, the player picks up the item engages in combat with the monster. The player can possess item at time, and drops existing
2

---

Published as a conference paper at ICLR 2020
Doc:
The Rebel Enclave consists of jackal, spider, warg, Arcane, blessed useful for paison monsters. Star Allince contains bat, panther, and wolf. Goblin, jaguar, and lynx are on the same Order the Gleaming and mysterious weapons beat cold monsters. Lightning monsters are weak against Grandmester's and Soldier's weapons. Fire defeated by fanatical and shimmering weapons.
You
they
Fire
goolin
1.P
4
Defeat the Order of the Forest
Ihu. bo0e
T92: Dc00
Polson
Prison
bat
Arcano fermatee
Polson
bust
gobbin
Aroano
Arcone
5
10
11
Inc. amard
Juy.
Jos. Fatutical sward
Figure RTFM requires jointly reasoning over the goal, a document describing environment dy- namics, and environment observations. This figure shows key snapshots from trained policy one randomly sampled environment. Frame shows the initial world. In 4, the agent approaches "fanatical sword", which beats the target "fire goblin". In 5, the agent acquires the sword. In 10, the agent evades the distractor "poison bat" while chasing the target. In 11, the agent engages the target and defeats it, thereby winning the episode. Sprites used for visualisation the agent observes cell content in text (shown in white). More examples are in appendix A.
weapons if they pick up a new weapon. A monster moves towards the player with 60% probability, and otherwise moves randomly. The dynamics, the agent's inventory, and the underspecified goal are rendered as text. The game world is rendered as matrix of text in which each cell describes the entity occupying the cell. We use Homann-writeen templates for stating which monsters belong to which team, which modifiers defeat (see appendix H for details on collection and G for list of entities in the game). In order to achieve the goal, the agent must cross-reference relevant information in the document and well as in the observations.
effective against which element, and which team the agent should
During every episode, subsample set of groups, monsters, modifiers, and elements to We randomly generate group assignments of which monsters belong to which team and which modifier effective against which element. A document that consists of randomly ordered statements responding to this group assignment is presented to the agent. We sample one element, one team, and monster from that team (e.g. "fire goblin" from "Order of the forest") " to be the target monster. Additionally, we sample one modifier that beats the element and an item to be the item that defeats the target monster (e.g. "fanatical sword"). Similarly, we sample an element, a team, and monster from different team to be the distractor monster (e.g. poison bat), well item that defeats the distractor monster (e.g. arcane hammer).
In order to win the game (e.g. Figure 1), the agent must
1. identify the target team from the goal (e.g. Order of the Forest)
2. identify the monsters that belong that team (e.g. goblin, jaguar, and ghost)
3. identify which monster is in the world (e.g. goblin), and its element (e.g. fire)
4. identify the modifiers that are effective against this element (e.g. fanatical, shimmering)
5. find which modifier is present (e.g. fanatical), and the item with the modifier (e.g. sword)
3

---

Published as a conference paper at ICLR 2020
6. pick up the correct item (e.g. fanatical sword)
7. engage the correct monster in combat (e.g. fire goblin).
If the agent deviates from this trajectory (e.g. does not have correct item before engaging in combat, engages with distractor monster), it cannot defeat the target monster and therefore will lose the game. The agent receives a reward of +1 if it wins the game and -1 otherwise.
RTFM presents challenges not found in prior work in that it requires large number of grounding steps in order to solve task. In order to perform this grounding, the agent must jointly reason over language goal and document of dynamics, as well as environment observations. In addition to the environment, the positions of the target and distractor within the document are randomised the agent cannot memorise ordering patterns in order to solve the grounding problems, and must instead identify information relevant to the goal and environment at hand.
We split environments into train and eval sets. No assignments of monster- tealter--eam--ee teattttaaeeeeeeee calum--mdiiier-llemeee are shared between train and eval to test whether the agent is able to generalise to new environments with dynamics not seen during training via reading. There more than 2 million train eval environments without considering the natural language templates, and 200 million otherwise. With random ordering of templates, the number of unique documents exceeds 15 billion.
MODEL
We propose the txt2x model, which builds representations that capture three-way interactions between the goal, document describing environment dynamics, and environment observations. We begin with definition of the Bidirectional Feature-wise Linear Modulation (FiLM²) layer, which forms the core of our model.
4.1 BIDIRECTIONAL FEATURE- WISE LINEAR MODULATION (FILM?) LAYER
Feature-wise linear modulation (FiLM), which modulates visual inputs using representations of textual instructions, is an effective method for image captioning (Perez et al., 2018) and instruction fol- lowing (Bahdanau
Linear,
Text features
Lineary
RaLUU
Tinenhue
output
Convvis
al., 2019). In RTFM, the agent must not only filter concepts in the visual domain using language but filter concepts in the text domain using visual observa- tions. To support this, FiLM2 builds codependent representations of text and visual inputs by further incorporating conditional representations of the text given visual observations. Figure 2 shows the FiLM² layer.
Visual
Coti
MaxPool
summary
Cotw.
Figure 2: The FiLM layer.
We use upper-case bold letters to denote tensors, lower-case bold letters for vectors, and non-bold letters for scalars. Exact dimensions these variables are shown in Table in appendix B. Let Ptext denote a fixed-length d -imeensinaii representation of the text and Xvis the representation of visual inputs with height H. width W, and dvis channels. Let Conv denote convolution layer. Let + and symbols denote element- wise addition and multiplication operations that broadcast over spatial dimensions. We first modulate visual features using text features:
Tre
Wrere + by
(1)
B
W WoDeext + b
(2)
V ReLU((1 + Ttext) * Convvis (Xvix) + .roxt)
(3)
Unlike FiLM, we additionally modulate text features using visual features:
Dvis Conv, (Xnv,(Xvis) (Xvis)
(4)
B = Conva( (Xvis)
(5)
Vixtt ReLU((1 + + vvis) (W Brext + bbexx) B
4

---

Published as a conference paper at ICLR 2020
Postional
Palay MLP
polayy
Inventory
Inventocy Fi STM
Sellatth
cat
Gosl
Goal BLS
Sellattn
FILM
FILM
FILM
Linear
Linear
Document
Gioal doc BILSTM
Atth
Pool
Adi
Attn
Vie-doo
Visual features
Figure 3: txt22 models interactions between the goal, document, and observations.
The output of the FiLM layer consists of the sum of the modulated features V. as well as a max-pooled summary s over this sum across spatial dimensions.
V Vvis + Viext
(7)
MaxPool(V)) (8)
4.2 THE 2x MODEL.
We model interactions between observations from the environment, goal, and document us- ing FiLM2 layers. We first encode text inputs using bidirectional LSTMs, then compute summaries using self-attention and conditional summaries using attention. We concatenate text summaries into text features, which, along with visual features, are processed through consecutive FiLM layers. In this case of a textual environment, we consider the grid of word embeddings as the visual features for F The final FiLM output is further processed by MLPs compute a policy distribution over actions and a baseline for advantage estimation. Figure shows the txt22 model.
Let Eobs denote word embeddings corresponding to the observations from the environment, where Eo ;, i, represents the embeddings corresponding to the labs- string that describes the objects location (i. j) in the grid-world. Let Edoc: Einv: and Egeat respectively denote the em- beddings corresponding to the wood document, the wiod inventory, and the goat- word goal. We first compute a fixed-length summary Ggoal of the the goal using a bidirectional LSTM (Hochre- iter & Schmidhuber, 1997) followed by self-attention (Lee et al., 2017: Zhong et al., 2018).
H goal BiLSTM goal Eggal)
(9)
dgoal, = Wgoat goal, + Ogoal
(10)
Lgonl
agoul softmax (oaa))
(11)
Cgoal
Agoal sgoal,
(12)
We abbreviate self-attention over the goal as Cgoul selfattn(H goat). We similarly compute
a summary of the inventory as Ciuv selfattnBiI.STT EEnv)), Next, we represent the document encoding conditioned on the goal using dot-product attention (Luong et al., 2015).
O = Cyoath
(14)
Hane BiLSTM goal-doe Edon)
(13)
Laoc
adoc softmax(d'ac))
(15)
Cdloe:
actoc, h
(16)
We abbreviate attention over the document encoding conditioned the goal summary as Cdoc = attend Haoc: Cgoal)) Next, we build the joint representation the inputs using succes- sive FiLM layers. At each layer, the visual input to the FiLM" layer is the concatenation of the output of the previous layer with positional features. For each cell, the positional feature X consists of the and y distance from the cell to the agent's position respectively, normalized by the width and height of the gord-world. The text imput is the concatenation of the goal summary, the inventory summary, the attention over the document given the goal, and the attention over the document given the previous visual summary. Let (a: denote the feature-wist concatenation of
5

---

Published as a conference paper at ICLR 2020
1,0
exp brl2m
Win rate
Model
Train
Eval 6x6
Eval
no boxt, mmod yo_yi__se
10x
conv
24
25 +
13 + 1
0.8
FiLM
49 -
49 +
32 =
win
no ho.task at
49 2
49 1
35 15:6
no vis attn
49 +
49 +1
40+:12
0.4
no_text_mod
49 49++1
49 +
35 +
Ext2
84121
83121
66
Table 1: Final win rate on simplest variant of RTFM. The models are trained on one set of dynamics (e.g. training set) and evaluated on
fremes
1e7
another set of dynamics (e.g. evaluation set). "Train" and "Eval" show final win rates on training and eval environments.
Figure 4: Ablation training curves on simplest variant of RTFM. Individual runs light
colours. Average win rates are in bold, dark lines.
and b. For the ith layer, we have
R( [V(i-l); pool
(17)
T() [craal Cinvi Cdoc; attend (Eaor) (
(18)
V ( 2iL (R (T))
(19)
BiLST viiadoo [doe) another encoding of the document similar to goal. produced using sep- arate LSTM, such that the document is encoded differently for attention with the visual features and with the goal. For 0, we concatenate the bag-of-wordhh embeddings of the grid with positional features the initial visual features ( [C, Eobs,ji p. We max pool linear transform of the initial visual features to compute the initial visual summary s MaxPool V + bini).
Let s (last) denote visual summary of the last FiLM² layer. We compute the policy Ypolicy and baseline Ybaseline
ReLUUW,s (Iest) bo)
(20)
Ypolicy = MLP policy (o)
(21)
Vbaseline MLP baseliae
(22)
where MLP policy and MLPPbaseliee baseline are 2-layer multi-layer perceptrons with ReLU activation. We
train using TorchBeast (Küttler al., 2019), implementation of IMPALA (Espeholt al., 2018).
Please refer to appendix D for details.
5 EXPERIMENTS
We consider variants of RTFM by varying the size of the grid-world (6 x 6 vs 10 x 10), allowing many-to-one group assignments make disambiguation more difficult (group), allowing dynamic, moving monsters that hunt down the player (dyna), and using natural language templated docu- ments (n1). the absence of many-to-one assignments, the agent does not need perform steps 3 and 5 in section 3 as there is no need to disambiguate among many assignees, making it easier to identify relevant information.
We compare txt22 to the FiLM model by Bahdanau et al. (2019) and language-conditioned residual CNN model. We train on one set of dynamics (e.g. group assignments monsters and modifiers) and evaluated In no aat.askktttn. the document attention conditioned on the goal utterance (equation 16) is re- moved and the goal instead represented through self-attention and concatenated with the rest of the text features. In no vis attn, we do not attend over the document given the visual output of the previous layer (equation 18), and the document is instead represented through self-attention.
held-out set of dynamics. We also study three variants of txt22.

---

Published as a conference paper at ICLR 2020
Transfer from
Transfer to
6
6 x 6 dyna groups
6 x 6 group n1
6 x 6
6 6 dyna
6 6
x 6
dyna
x 6
dyna
groups
n1
nl
group
n1
random
84 + 20
26 +
25 + 3
45
23 + 2 64 12
25 =
23 +
23 + 2
+6 x 6 +dyna
85 -
82 = 19
78 = 24
52 = 13
53 + 18
40 8
77 + 10
65 +
16
43 + 4
+gzoup
65 + 17
Table 2: Curriculum training results. We keep 5 randomly initialised models through the entire curriculum. cell row and column j shows transfer from the best- best-performing setting in the previous stage (bold in row - 1) to the new setting in column j. Each cell shows final mean and standard deviation win rate on the training environments. Each experiment trains for million frames, except for the initial stage (first row, 100 million instead). For the last stage (row 4), we also transfer to a 10 x 10 + dyna + group + nl variant and obtain 61 + 18 win rate.
In no-text.nmee mood, text modulation using visual features (equation 6) is removed. Please see ap- pendix C for model details on our model and baselines, and appendix for training details.
5.1 COMPARISON BASELINES AND ABLATIONS
We compare txt2m to baselines and ablated variants on simplified variant of RTFM in which there are one-to-one group assignments (no group), stationary monsters (no dyna), and no nat- ural language templated descriptions (no n1). Figure 4 shows that compared to baselines and ab- lated variants, 2xt more sample efficient and converges to higher performance. Moreover, no ablated variant is able to solve the tasks it is the combination of ablated features that ables txt2 to win consistently. Qualitatively, the ablated variants converge to locally optimum policies in which the agent often picks up random item and then attacks the correct monster, sulting in a ~ 50% win rate. Table shows that all models, with the exception of the CNN baseline, generalise new evaluation environments with dynamics and world configurations not seen during training, with 2x outperforming FiLM and the CNN model.
We find similar results for txt2m, its ablated variants, and baselines on separate, language-based rock-paper-seee sccper-scissors task in which the agent needs to deduce cyclic dependencies (which type beats which other type) through reading in order to acquire the correct item and defeat monster. We observe that the performance of reading models transfer from training environments to new envi- ronments with unseen types and unseen dependencies. Compared to ablated variants and baselines, txt2n is more sample efficient and achieves higher performance ronment dynamics. When transferring to new environments, t 2x remains more sample efficient than the other models. Details these experiments are found in appendix E.
both training and new envi-
5.2 CURRICULUM LEARNING FOR COMPLEX ENVIRONMENTS
Due to the long sequence of co-references the agent must perform in order to solve the full RTFM (10 10 with moving monsters, many-to-one group as- signments, and natural language templated docu- ments) we design a curriculum to facilitate policy learning by starting with simpler variants of RTFM. We start with the simplest variant (no group, no dyna, no n1) and then add in an additional di- mension of complexity. We repeatedly add more complexity until we obtain 10 x 10 worlds with mov- ing monsters, many-to- natural language templated descriptions. The per- formance across the curriculum is shown in Table 2
Train
Eval
Win rate
env
env
Train
Eval
6 6
6 x
65
+ 22
10 x 10
55 + 27
10 x 10
10 10 61 + 18
43 + 13
Table 3: Win rate when evaluating on dynamics and world configurations for 2x on the full RTFM problem.
group assignments and

---

Published as a conference paper at ICLR 2020
Al
2000
speu
apps in
(a) The entities present are shimmering morning star, mysterious spear, fire jaguar, and lightning ghost.
The
(b) The entities present are soldier's 's axe, shimmering axe, fire shaman, and poison wolf.
Figure 5: txt2n attention on the full RTFM. These include the document attention conditioned on the goal (top) as well as those conditioned on summaries produced by intermediate FiLM layers. Weights are normalised across words (e.g. horizontally). Darker means higher attention weight.
(see Figure 13 in appendix F for training curves of each stage). We see that curriculum learning is crucial to making progress on RTFM, and that initial policy training (first row of Table 2) with addi- tional complexities in any of the dimensions result significantly worse performance. We take each of the 5 runs after training through the whole curriculum and evaluate them dynamics not seen during training. Table shows variants of the last stage of the curriculum which the model was trained on 6 6 versions of the full RTFM and in which the model was trained on 10 x 10 versions of the full RTFM. We see that models trained on smaller worlds generalise to bigger worlds. Despite curriculum learning, however, performance of the final model trail that of human players, who can consistently solve RTFM. This highlights the difficulties of the RTFM problem and suggests that there is significant room for improvement in developing better language grounded policy learners.
Attention maps. Figure 5 shows attention conditioned on the goal and observation summaries produced by intermediate FiLM layers. coal-conditioned attention consistently locates the clause that contains the team the agent is supposed attack. Intermediate layer attentions focus regions near modifiers and monsters, particularly those that are present in the observations. These results suggests that attention mechanisms in txt help identify relevant information the document.
Analysis of trajectories and failure modes. We examine trajectories from vell-performing poli- cies (80% win rate) as well as porly-performing policies (50% win rate) on the full RTFM. We find that well-performing policies exhibit a number of consistent behaviours such as identifying the correct item to pick up to fight the target monster, avoiding distractors, and engaging target monsters after acquiring the correct item. In contrast, the poorly- poorlypprforming policies occasionally pick up the wrong item, causing the agent lose when engaging with monster. In addition, it occasionally gets stuck in evading monsters indefinitely, causing the agent to lose when the time runs out. Replays of both policies can be found in GIFs in the supplementary materials'.
CONCLUSION
We proposed RTFM, grounded policy learning problem in which agent must must jointly reason over a language goal, relevant dynamics specified in document, and environment observations. In order to study RTFM, we procedurally generated combinatorially large number environment dynam- ics such that the model cannot memorise set of environment dynamics and must instead generalise via reading. We proposed txt22, a model that captures three-way interactions hetween the goal,
Trajectories 2xt:2 RTFM can he found https:: i /t 9k74kk
8

---

Published as a conference paper at ICLR 2020
document, and observations, and that generalises environments with dynamics not dur- ing training. txt2n outperforms baselines such as FiLM and manguage-conditions CNNs. Through curriculum learning, txt performs well complex RTFM tasks that require several reasoning and coreference steps with natural language templated goals and descriptions of the dynamics. Our work suggests that language understanding via reading promising way learn policies that generalise to new environments. Despite curriculum learning, our best models trail performance of human players, suggesting that there is ample room for improvement in grounded policy learning on complex RTFM problems. In addition to jointly learning policies based on external documentation and language goals, we are interested in exploring how to use supporting evidence in external doc- umentation to reason about plans (Andreas et al., 201 8) and induce hierarchical policies (Hu et al., 2019; Jiang et al., 2019).
ACKNOWLEDGEMENT
We thank Heinrich Küttler and Nantas Nardelli for their help in adapting TorchBeast and the FAIR London team for their feedback and support.
REFERENCES
Peter Anderson, Qi Wu, Damien Teney, Jake Bruce, Mark Johnson, Niko Sünderhauf, Ian D. Reid, Stephen Gould, and Anton den Hengel. Vision-and-language navigation: Interpret- ing visually-grounded navigation instructions in real environments. In CVPR, 2018.
Jacob Andreas, Dan Klein, and Sergey Levine. Learning with latent language. In NAACL, 2018.
Dzmitry Bahdanau, Felix Hill, Jan Leike, Edward Hughes, Pushmeet Kohli, and Edward Grefen- stette. Learning to follow language instructions with adversarial reward induction. In ICLR, 2019.
K. Barnard and Forsyth. Learning the semantics of words and pictures. ICCV, 2001.
S. R. K. Branavan, David Silver, and Regina Barzilay. Learning to win by reading manuals in monte-carlo framework. In ACL, 2011.
S. R. K. Branavan, Nate Kushman, Tao Lei, and Regina Barzilay. Learning high-level planning from text. In ACL, 2012.
S.R.K. Branavan. Grounding Linguistic Analysis in Control Applications. PhD thesis, MIT, 2012.
David L. Chen and Raymond J. Mooney. Learning to sportscast: test of grounded language acquisition. In ICML, 2008.
John D. Co-Reyes, Abhishek Gupta, Suvansh Sanjeev, Nick Altieri, John DeNero, Pieter Abbeel, and Sergey Levine. Guiding policies with language via meta-learning. In ICLR, 2019.
Lasse Espeholt, Hubert Soyer, Rémi Munos, Karen Simonyan, Volodymyr Mnih, Tom Ward, Yotam Doron, Vlad Firoiu, Tim Harley, lain Dunning, Shane Legg, and Koray Kavukcuoglu. IMPALA: scalable distributed deep-rl with importance weighted actor-learner architectures. In ICML, 2018.
Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell. Speaker-follower models for vision-and aaw-aaddlagagg language navigation. In NeurIPS. 2018.
Karl Moritz Hermann, Felix Hill, Simon Green, Fumin Wang, Ryan Faulkner, Hubert Soyer, David Szepesvari, Wojeiech Marian Czamecki, Max Jaderberg, Denis Teplyashin, Marcus Wainwright, Chris Apps, Demis Hassabis, and Phil Blunsom. Grounded language learning in a simulated 3d world. CoRR, aws//700.000 706.06551,11. 2017.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural Compututation, 9()), 1997.

---

Published as a conference paper at ICLR 2020
Hengyuan Hu, Denis Yarats, Qucheng Gong, Yuandong Tian, and Mike Lewis. Hierarchical decision making by generating and following natural language instructions. CoRR, 00744, 2019.
Yiding Jiang, Shixiang Gu, Kevin Murphy, and Chelsea Finn. Language as an abstraction for hier- archical deep reinforcement learning. CoRR, abs/1906.07343 2019.
Thomas Kollar, Stefanie Tellex, Deb Roy, and Nicholas Roy. Toward understanding natural language directions. In HRI, 2010.
Heinrich Küttler, Nantas Nardelli, Thibaut Lavril, Marco Selvatici, Viswanath Sivakumar, Tim Rocktäschel, and Edward Grefenstette. TorchBeast: PyTorch Platform for Dis- tributed RL. arXiv preprint arXiv:1910.03552, 2019. URL https://gitte com/ facebookresearcc/ wtrerrcch/trcbbbas....
Kenton Lee, Luheng He, Mike Lewis, and Luke Zettlemoyer. End-to-end neural coreference reso- lution. EMNLP, 2017.
Jiwei Li, Will Monroe, Alan Ritter, Dan Jurafsky, Michel Galley, and Jianfeng Gao. Deep reinforce- ment learning for dialogue generation. In EMNLP, 2016.
Timothy P. Lillicrap, Jonathan J. Hunt, Alexander Pritzel, Nicolas Manfred Otto Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. Continuous control with deep reinforcement learn- ing. CoRR, abs/1 509. 009901,,, 2015.
Jelena Luketina, Nantas Nardelli, Gregory Farquhar, Jakob Foerster, Jacob Andreas, Edward Grefen- stette, Shimon Whiteson, and Tim Rocktäschel. A Survey of Reinforcement Learning Informed by Natural Language. In IJCAI, 2019.
Minh-Thang Luong. Hieu Pham, and Christopher D Manning. Effective approaches to attention- based neural machine translation. In ACL, 2015,
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, loannis Antonoglou, Daan Wierstra, and Martin A. Riedmiller. Playing atari with deep reinforcement learning. CoRR, abs/1312.5602, 2013.
Karthik Narasimhan, Regina Barzilay, and Tommi S. Jaakkola. Deep transfer in reinforcement learning by language grounding, JAIR, 2018.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville, Film: Visual reasoning with a general conditioning layer. AAAI, 2018.
Stefanie Tellex, Thomas Kollar, Steven Dickerson, Matthew R. Walter, Ashis Gopal Banerjee, Seth Teller, and Nicholas Roy. Understanding natural language commands for robotic navigation and mobile manipulation. In AAAI, 2011.
T. Tieleman and G. Hinton. Lecture 6.5 Ros-RmmsPPPp Divide the gradient by running average of its recent magnitude. COURSERA: Neural Networks for Machine Learning, 2012.
Sida I. Wang, Percy Liang, and Christopher D. Manning. Learning language games through inter- action. In ACL, 2016.
Victor Zhong, Caiming Xiong, and Richard Socher. Global-locally self-attentive dialogue state tracker. In ACI, 2018.
10

---

Published as a conference paper at ICLR 2020
A PLAYTHROUGH EXAMPLES
These figures shows key snapshots from a trained policy on randomly sampled environments.
Doc:
should arcane mysterious items beat lightning monsters, Panther, warg, and wolf are from the Order of the Forest. Blessed and Grandmaster's Items monsters. Cold good against fanatical and shimmering
Ancone spear
spoar
PD
Ligoning chaman
Lighting chaman
weapons. The Star Alliance team is made up bestle, jackal, and sheman. Imp, jaguar, lynx up the Rebel Enclave. Gleeming and Soldiar's weapons best fire monsters.
Defeat the Star Alliance
3
Fiow Ire
Lighing
Lighinn XD
Sakiier' sword
7
You Fiu imp
8
9
Figure 6: The initial world is shown in In 4, the agent avoids the target "lightning shaman" because it does not yet have "arcane spear", which beats the target. In 7 and 8, the agent is cornered by monsters. In the agent is forced to engage combat and loses.
Doc:
Cold monsters and Soldier's weapens. The Order of the Forest team consists of ant, lynx, and woll. Mysterious and shimmering weapons are good against lightning monsters. Poison monster are defeated by blessed and fanatical itemss Get arcane and Grandmaster's weepons to panther, zombie Star Alliance, Jackal, jaguar, and ghost are on the Rebel Enclave. Goal:
gleaming
Peseen
Cold
ghand
Saldier
slay monsters.
Fanatical
Cold ghocl
Fanotical keciaro
Fight
Rebel Enclaav.
5
Polion
zombbo
Diles
Possan
Fanstical
Fanstical
chosc
Faneticcal
kalana
cold
11
13
14
Solddors
Soldio's
Figure 7: The initial world shown in In 5 the agent evades the target "cold ghost" because does not yet have "soldier's knife", which beats the target. In 11 and 13, the agent obtains "soldier's knife" while evading monsters. In 14, the agent defeats the target and wins.
11

---

Published as a conference paper at ICLR 2020
B VARIABLE DIMENSIONS
Let Prext E Rddeett denote a fixed-length deex-- representation of the text and X E Rd denote the representation of visual inputs with
Variable
Symbol
Dimension
dimxt-dim text representation
Deextt
diext
dvis-dim visual representation with height H, width W, dvis channels
X
dvis x H X
Environment observations embeddings
Enba
Lobs X demb x x W
lobs lobs-word string that describes the objects in location (f,j) in the grid-world
lobe X demb
dooe- word document embeddings
Edoc
ldoc X demb
liw--word word inventory embeddings
Eun
linv demb
goal word goal embeddings
Egoal
lgoal damb
Table 4: Variable dimensions
12

---

Published as a conference paper at ICLR 2020
C MODEL DETAILS
C.I
TXT2m
Hyperparameters. The 2xt22 used in our experiments consists consecutive FiLM layers, each with 3x3 convolutions and padding and stride sizes of 1. The txt2 layers have channels of 16, 32, 64, 64, and 64, with residual connections from the 3rd layer to the 5th layer. The Goal-doc LSTM (see Figure 3) shares weight with the Goal LSTM. The Inventory and Goal LSTMs have hidden dimension of size 10, whereas the Vis-doc LSTM has a dimension of 100. We use a word embedding dimension of 30.
C.2 CNN WITH RESIDUAL CONNECTIONS
ficnal featuree
Pollcy MLP
policy
Inventory
MILSTT TM
Salkatn
Coal
Gooal BILSTM
Selaattn
Corw
call
Conw
cel
Corv
Linea
Document
Sallattn
Baseline
baseline
Figure 8: The convolutional network baseline. The FiLM baseline has the same structure, but with convolutional layers replaced by FiLM layers.
Like 2xr, the CNN baseline consists of 5 layers of convolutions with channels of 16, 32, 64, 64, and 64. There are residual connections from the 3rd layer to the 5th layer. The input to each layer consists of the output of the previous layer, concatenated with positional features.
The input to the network is the concatenation of the observations V and text representations. The text representations consist of self-attention over bidirectional LSTM-encoded goal, document, and inventory. These attention outputs are replicated over the dimensions of the grid and concatenated reature-wise with the observation embeddings in each cell. Figure 8 illustrates the CNN baseline.
C.3 FILM BASELINE
The FiLM baseline encodes text in the same fashion as the CNN model. However, instead of using convolutional layers, each layer is a FiLM layer from Bahdanau et al. (2019). Note that in our case, the language representation is self-attention over the LSTM states instead of concatenation of terminal LSTM states.
13

---

Published as a conference paper at ICLR 2020
D TRAINING PROCEDURE
We train using an implementation of IMPALA (Espeholt et al., 2018). In particular, we use 20 actors and batch size of 24. When unrolling actors, we use a maximum unroll length of 80 frames. Each episode lasts for maximum of 1000 frames. We optimise using RMSProp (Tieleman & Hinton, 2012) with a learning rate of 0.005, which is annealed linearly for 100 million frames. We set = 0.99 and 0.01.
During training, we apply a small negative reward for each time step -0.02 and discount factor of 0.99 to facilitate convergence. We additionally include a entropy cost to encourage exploration. Let Ypalicy denote the policy. The entropy loss is calculated as
poliey
Σ poliicy log Ppolicy,
(23)
In addition policy gradient, we add in the entropy loss with a weight of 0.005 and the baseline loss with weight of 0.5. The baseline loss is computed as the root mean square of the advantages (Es- peholt al., 2018).
When tuning models, we perform a grid search using the training environments to select hyperpa- rameters for each model. We train runs for each configuration in order report the mean and standard deviation. When transferring, we transfer each of the runs to the new task and once again report the mean and standard deviation.
14

---

Published as a conference paper at ICLR 2020
graphs
# edges
# nodes
Scenario
train
dev
unscen
train
dev
% new
train
dev
% new
permutation
30
30
y
20
20
n
60
60
new edge
20
20
y
48
36
y
17
13
new edge+nodes
60
60
y
20
20
5
Table 5: Statisties of the three variations of the bock-paper-scissooe task
seen envirenment dynamics
new environment dynamics
win rate
0.6
conv
film
film
film22
0.0
10x10
11x11
11x11
Figure 10:
Performance the Rock-paper- pappppppppsstte scies-scissors task across models. Left shows final perfor- mance on environments whose goals and dynamics were seen during training. Right shows perfor- mance the environments whose goals and dynamics were not during training.
In addition to the main RTFM tasks, we also study simpler formulation called Rock-paper-scissoos that has a fixed goal. In Rock-paper-scissors the agent must interpret document that describes the environment dynamics in order to solve the task. Given an set of characters (e.g. a-z), we sample 3 characters and set up pock-paper-scissonn-lii dependency graph between the characters (e.g. "a beats beats c, beats a²"). We then spawn a monster in the world with a randomly as- signed type (e.g. "b goblin"), as well an item corresponding to each type (e.g. "a", "b", and "c"). The attributes of the agent, monster, and items set up such that the player must obtain the correct item and then engage the monster in order to win. Any other sequence of actions (e.g. en- gaging the monster without the correct weapon) results a loss. The winning policy should then be to first identify the type of mon- ster present, then cross-reference the document find which item defeats that type, then pick up the item, and finally engage the monster in combat. Figure shows an instance of Rock- bock-ppper-seissorr
Doc:
e beats d.
c beats e.
d beats c.
goblin
Inventory: empty
You
B
Figure 9; The Rock-paper-seissors task requires jointly reasoning over the game observations and a document describing environment dynamics. The agent observes cell content in the form of text (shown in white).
Reading models generalise new environments. We split environment dynamies by per- muting 3-character dependency graphs from an alphabet, which we randomly split into training and held-out sets. This corresponds to the "permutations" setting Table 5.
15

---

Published as a conference paper at ICLR 2020
permutations
new edge
new edge node
1.0
0.8
win
0.4
0.2
conV
film
film"2
conv transfer
film transfer
film transfer
3
5
frames
1e7
frames
1e7
frames
1e7
Figure 11: Learning curve while transferring to the development environments. Win rates of indi- vidual runs are shown in light colours. Average win rates are shown in bold, dark lines.
We train models on the the training set and evaluate them on both seen and not seen during training. The left of Figure 10 shows the performance of mod- els on worlds of varying sizes with training en- vironment dynamics. In this case, the dynam- ies (e.g. dependency graphs) were seen during training. For 9 x 9 and 11 11 worlds, the world configuration not seen during training. For 10 10 worlds, there 5% chance that the initial frame was seen during training. Fig- ure 10 shows the performance on held-out envi- ronments not seen during training. We see that all models generalise to environments not seen during training, both when the world configura- tion is not seen (left) and when the enviromment dynamics not seen (right).
worlds from
exp
film
conv
no. text,
vis attn
0.6
no goal_attn
0.0
1
frames
1e7
Figure 12: Ablation training curves. Win rates of individual runs are shown in light colours. Aver- age win rates are shown in bold, dark lines.
Reading models generalise to new concepts. In addition splitting via permutations, we de- vise two additional ways of splitting environment dynamics by introducing new edges and nodes into the held-out set. Table 5 shows the three different settings. For each, we study the transfer be- haviour of models on new environments. Figure 11 shows the learning curve when training model on the held-out environments directly and when transferring the model trained on train environments to held-out environments. We observe that all models significantly more sample-efficient when transferring from training environments, despite the introduction of new edges and new nodes.
txt2n is sample-efficient and learns better policies. In Figure 10, model outperforms the CNN model on both training environment dynamics and held-out environ- ment dynamics. txt22 further outperforms FiLM, and does more consistently in that the final performance has less variance. This behaviour is also observed in the in Figure 11. When training on the held-out set without transferring, 2xt is more sample efficient than FiLM and the CNN model, and achieves higher win-rate. When transferring to the held-out set, txt remains more sample efficient than the other models.
that the FiLM
"hhere are 24360 unique grid configurations given particular dependency graph, 4060 unique dependency graphs in the training set, and 50 million frames during training. After training, the model finishes an episode in approximately 10 frames. Hence the probability of seeing redundant initial frame is Se7/10 5%.
24360-1060
16

---

Published as a conference paper at ICLR 2020
F
CURRICULUM LEARNING TRAINING CURVES
rand
rand +6x6
1.0
0.8
win rate
0.6
0.4
0.2
0.0
0.0
0.4
0.6
frames
1e8
frames
1e7
rand + 6x6 +dyna
rand +6x6 +dyna +groups
1.0
0.8
win rate
0.6
0.4
0.2
move 6
ni 6
move,nl
extra,move.nt .nl
0.0
10
extra
extra,nl 6
extra,move 6
extra,mous .n 10
frames
1e7
frames
1e7
Figure 13: Curriculum learning results for txt RTFM. Win rates individual runs shown in light colours. Average win rates are shown in bold, dark lines.
G ENTITIES AND MODIFIERS
Below a list of entities and modifiers contained in RTFM:
Monsters: wolf, jaguar, panther, goblin, bat, imp, shaman, ghost, zombie
Weapons: sword, axe, mornnggtaa, polearm, knife, katana, cutlass, spear
Elements: cold, fire, lightning, poison
Modifiers: Grandmaster's, ss blessed, shimmering, gleaming, fanatical, mysterious, Soldier's, 's arcane
Teams: Star Alliance, Order of the Forest, Rebel Enclave
H LANGUAGE TEMPLATES
natural language templates for the goal and the dynamics. The goal statements in RTFM describe which team the agent should defeat. We collect 12 language templates for goal statements. The document of environment dynamics consists of two types of statements. The first type describes which monsters are assigned to with team. The second type describes which modifiers, which describe items, effective against which element types, which are associated with monsters. We collection 10 language templates for each type of statements. The entire document is composed from statements, which randomly shuffled. We randomly sample template for each statement, which we fill with the monsters and team for the first type and modifiers and element for the second type.
We collect
17


Reviews:
[{"review_id": "SJerltuHcB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "First of all, I should acknowledge that this is a last-minute emergency review. Second, I should also acknowledge that I haven’t actively followed the specific research topic handled in this paper, although I work quite a lot in reinforcement learning and dialogue systems. The last paper I read on the topic was Branavan et al’s “Learning to win by reading manuals in monte-carlo framework” (ACL 2011).\n\nThis paper is about language-conditioned reinforcement learning, where the agent is required to perform machine reading of documents to learn policies to solve a task in new environments. The contribution of this paper is seemingly two fold: first, a new benchmark test suite is proposed (named “read to fight monsters” funny acronym), and second, an improved neural model (named “txt2pi”) which extends FiLM proposed for related tasks. Various experiments on txt2pi are conducted to show how the model performs against baselines, as well as it behaves under different learning settings.\n\nOverall, this paper makes a good contribution to the research on deep models for language-conditioned reinforcement learning, but I see a number of things I miss in this paper. \n\nThe first one is about the test suite RTFM. The authors claim that it poses a new challenge that the agent has to understand the language specification of the goal, the environment dynamics, and the observations *all together* (compared to prior work that misses one or two), I am not confident that it is not challenging enough. \n\n- The main concern I have is that they are all machine-generated, and soon or later, this kind of test suite could be beaten-to-death. A more ideal test suite would be the texts collected by humans, aiming for more diversity, ambiguity, and sometimes incorrectness. \n- The RTFM would combinatorily generate a large number of environments, but still, they all look pretty simple. It would be interesting to have “partially specified” documents, such as “lightning monsters are known to be weak against grandmaster’s or soldier’s weapons, but it is not specifically known”. It would be interesting to have partially observable environments, where the agent has to do some information gathering in the environment to behave optimally.\n\nThe second one is about txt2pi model proposed in the paper.\n- txt2pi adequately extends FiLM to handle multiple sources of information, but what is the lesson being learned? \nI find the discussions in page 15 very enlightening. Could you move the content to the main text and defer the curriculum learning details to the appendix?\n\nIn summary, this paper proposes an interesting test suite for language-conditioned reinforcement learning, but I wish it is more realistic in the language data and more complex in the RL tasks. \n\n--- \nI have updated my rating based on the reviewer response. Thank you for clarifying some of my concerns. I am still a bit reserved because of the use of synthetic corpus. However, at the same time, I think there could be a lot of follow-up papers based on this work.  \n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}, {"review_id": "S1gpAKB15H", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper constructs a new game that requires combining visual reasoning with text understanding to win.  The authors propose a new model txt2π, based on a new layer called FiLM², which combines text and visual features in a way that allows visual features to be encoded with knowledge of the text features (as in the FiLM layer from previous work), as well as text to be encoded with knowledge of the visual features. The model is trained to play the game using IMPALA.  Ablation studies show that the FiLM² layer leads to a substantial improvement, and also shows the necessity of curriculum learning.  Performance is still below human performance suggesting this is a promising area for future work.\n\nThe work here is complicated, but the writing and explanations are very clear. A new problem domain is introduced, which I think will be interesting to many other researchers.  I am not an expert in this area, so it's hard for me to give a confident assessment of the significance of this paper, so I give this paper a Weak Accept, despite enjoying the paper and having no specific points of criticism.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I made a quick assessment of this paper.", "review_assessment:_checking_correctness_of_experiments": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BylH2AYTFS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work proposes a new environment, Read to Fight Monsters (RTFM), and correspondingly a new algorithm, txt2\\pi, for solving this problem. The RTFM requires the agent to read a description of the rules (x beats y, etc) and a description of the goal (to eliminate y), and perform the task correctly to win the game. The txt2\\pi algorithm uses the newly proposed FiLM^2 module and consists of integrating the visual input (grid world configuration) and the text input (descriptions) to learn a policy and baseline (actor-critic). \n\nPros\n- The presentation is relatively clear.\n- A new environment that can be impactful for the field.\n\nCons\n- Certain design choices are not discussed properly.\n- The experimental evidence is relatively weak.\n\n(1) For RTFM, a complete list of possible elements would be helpful, at least in the supplementary. For example, the possible monsters/elements/weapons, etc.\n\n(2) Some proposals lack motivation or explanation. \n(2.1) How is Eq.(8) a summary?\n(2.2) Why the attention in (18) should base on the vis-doc embedding instead of the goal embedding, or goal-conditioned doc embedding?\n(2.3) It would be helpful to provide an example of the inventory description.\n(2.4) What is the difference between the Goal and Goal-doc in Fig.3?\n\n(3) Experiment\n(3.1) \"no group\", \"no dyna\" and \"no nl\" seem to be suggesting a simpler environment. How is no natural language templated descriptions (no nl) considered as an easier scenario than with a template? The description would be harder to parse or understand without structure (template).\n(3.2) It is mentioned several times that the models are \"trained on one set of dynamics and evaluated on another set of dynamics\". Specifically, what are the differences?\n(3.3) For the curriculum training (Table 2), adding dyna does not hurt the performance (from 84 to 85). Further explanation would be helpful.\n(3.4) How are the baselines and alternative methods perform in the full environment (+group etc.)?\n(3.5) The texts on Fig.5 are not evenly spaced, which makes them harder to align with the attention (also some letters have dots above them). Does this figure correspond to (15)? In Fig.5b, a0 focuses on \"gleaming\" and a1 focuses on \"lighting\", which are not present in the entities (compared to the caption). Something is wrong or missing here.\n(3.6) Eventually, the txt2\\pi algorithm only achieves modest improvement over random agents (65% versus 50%), which is not very impressive.\n\n## Update ##\nThank you for the thorough explanations, which are very helpful. It would be better to include them in the paper to clarify potential misunderstandings.\n\nOne more remark is that the goal-doc attention map (i.e., lg) in Fig. 5b focuses on lynx, which does not make sense in an environment with only a beetle and a wolf. This is strange and problematic given that it is an important attention map for many subsequent modules.\n\nI have changed my score accordingly. It also needs to be considered that I'm not very familiar with the field.", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes RTFM, a new model in the field of language-conditioned policy learning. This approach is promising and important in reinforcement learning because of the difficulty to learn policies in new environments. 

Reviewers appreciate the importance of the problem and the effective approach. After the author response which addressed some of the major concerns, reviewers feel more positive about the paper. They comment, though, that presentation could be clearer, and the limitations of using synthetic data should be discussed in depth.

I thank the authors for submitting this paper.
Decision: Accept (Poster)