RTFM: Generalising to New Environment Dynamics via Reading

1 INTRODUCTION . Reinforcement learning ( RL ) has been successful in a variety of areas such as continuous control ( Lillicrap et al. , 2015 ) , dialogue systems ( Li et al. , 2016 ) , and game-playing ( Mnih et al. , 2013 ) . However , RL adoption in real-world problems is limited due to poor sample efficiency and failure to generalise to environments even slightly different from those seen during training . We explore language-conditioned policy learning , where agents use machine reading to discover strategies required to solve a task , thereby leveraging language as a means to generalise to new environments . Prior work on language grounding and language-based RL ( see Luketina et al . ( 2019 ) for a recent survey ) are limited to scenarios in which language specifies the goal for some fixed environment dynamics ( Branavan et al. , 2011 ; Hermann et al. , 2017 ; Bahdanau et al. , 2019 ; Fried et al. , 2018 ; Co-Reyes et al. , 2019 ) , or the dynamics of the environment vary and are presented in language for some fixed goal ( Branavan et al. , 2012 ) . In practice , changes to goals and to environment dynamics tend to occur simultaneously—given some goal , we need to find and interpret relevant information to understand how to achieve the goal . That is , the agent should account for variations in both by selectively reading , thereby generalising to environments with dynamics not seen during training . Our contributions are two-fold . First , we propose a grounded policy learning problem that we call Read to Fight Monsters ( RTFM ) . In RTFM , the agent must jointly reason over a language goal , a document that specifies environment dynamics , and environment observations . In particular , it must identify relevant information in the document to shape its policy and accomplish the goal . To necessitate reading comprehension , we expose the agent to ever changing environment dynamics and corresponding language descriptions such that it can not avoid reading by memorising any particular environment dynamics . We procedurally generate environment dynamics and natural language templated descriptions of dynamics and goals to produced a combinatorially large number of environment dynamics to train and evaluate RTFM . ∗Work done during an internship at Facebook AI Research . Second , we propose txt2π to model the joint reasoning problem in RTFM . We show that txt2π generalises to goals and environment dynamics not seen during training , and outperforms previous language-conditioned models such as language-conditioned CNNs and FiLM ( Perez et al. , 2018 ; Bahdanau et al. , 2019 ) both in terms of sample efficiency and final win-rate on RTFM . Through curriculum learning where we adapt txt2π trained on simpler tasks to more complex tasks , we obtain agents that generalise to tasks with natural language documents that require five hops of reasoning between the goal , document , and environment observations . Our qualitative analyses show that txt2π attends to parts of the document relevant to the goal and environment observations , and that the resulting agents exhibit complex behaviour such as retrieving correct items , engaging correct enemies after acquiring correct items , and avoiding incorrect enemies . Finally , we highlight the complexity of RTFM in scaling to longer documents , richer dynamics , and natural language variations . We show that significant improvement in language-grounded policy learning is needed to solve these problems in the future . 2 RELATED WORK . Language-conditioned policy learning . A growing body of research is learning policies that follow imperative instructions . The granularity of instructions vary from high-level instructions for application control ( Branavan , 2012 ) and games ( Hermann et al. , 2017 ; Bahdanau et al. , 2019 ) to step-by-step navigation ( Fried et al. , 2018 ) . In contrast to learning policies for imperative instructions , Branavan et al . ( 2011 ; 2012 ) ; Narasimhan et al . ( 2018 ) infer a policy for a fixed goal using features extracted from high level strategy descriptions and general information about domain dynamics . Unlike prior work , we study the combination of imperative instructions and descriptions of dynamics . Furthermore , we require that the agent learn to filter out irrelevant information to focus on dynamics relevant to accomplishing the goal . Language grounding . Language grounding refers to interpreting language in a non-linguistic context . Examples of such context include images ( Barnard & Forsyth , 2001 ) , games ( Chen & Mooney , 2008 ; Wang et al. , 2016 ) , robot control ( Kollar et al. , 2010 ; Tellex et al. , 2011 ) , and navigation ( Anderson et al. , 2018 ) . We study language grounding in interactive games similar to Branavan ( 2012 ) ; Hermann et al . ( 2017 ) or Co-Reyes et al . ( 2019 ) , where executable semantics are not provided and the agent must learn through experience . Unlike prior work , we require grounding between an underspecified goal , a document of environment dynamics , and world observations . In addition , we focus on generalisation to not only new goal descriptions but new environments dynamics . 3 READ TO FIGHT MONSTERS . We consider a scenario where the agent must jointly reason over a language goal , relevant environment dynamics specified in a text document , and environment observations . In reading the document , the agent should identify relevant information key to solving the goal in the environment . A successful agent needs to perform this language grounding to generalise to new environments with dynamics not seen during training . To study generalisation via reading , the environment dynamics must differ every episode such that the agent can not avoid reading by memorising a limited set of dynamics . Consequently , we procedurally generate a large number of unique environment dynamics ( e.g . effective ( blessed items , poison monsters ) ) , along with language descriptions of environment dynamics ( e.g . blessed items are effective against poison monsters ) and goals ( e.g . Defeat the order of the forest ) . We couple a large , customisable ontology inspired by rogue-like games such as NetHack or Diablo , with natural language templates to create a combinatorially rich set of environment dynamics to learn from and evaluate on . In RTFM , the agent is given a document of environment dynamics , observations of the environment , and an underspecified goal instruction . Figure 1 illustrates an instance of the game . Concretely , we design a set of dynamics that consists of monsters ( e.g . wolf , goblin ) , teams ( e.g . Order of the Forest ) , element types ( e.g . fire , poison ) , item modifiers ( e.g . fanatical , arcane ) , and items ( e.g . sword , hammer ) . When the player is in the same cell with a monster or weapon , the player picks up the item or engages in combat with the monster . The player can possess one item at a time , and drops existing weapons if they pick up a new weapon . A monster moves towards the player with 60 % probability , and otherwise moves randomly . The dynamics , the agent ’ s inventory , and the underspecified goal are rendered as text . The game world is rendered as a matrix of text in which each cell describes the entity occupying the cell . We use human-written templates for stating which monsters belong to which team , which modifiers are effective against which element , and which team the agent should defeat ( see appendix H for details on collection and G for a list of entities in the game ) . In order to achieve the goal , the agent must cross-reference relevant information in the document and as well as in the observations . During every episode , we subsample a set of groups , monsters , modifiers , and elements to use . We randomly generate group assignments of which monsters belong to which team and which modifier is effective against which element . A document that consists of randomly ordered statements corresponding to this group assignment is presented to the agent . We sample one element , one team , and a monster from that team ( e.g . “ fire goblin ” from “ Order of the forest ” ) to be the target monster . Additionally , we sample one modifier that beats the element and an item to be the item that defeats the target monster ( e.g . “ fanatical sword ” ) . Similarly , we sample an element , a team , and a monster from a different team to be the distractor monster ( e.g . poison bat ) , as well as an item that defeats the distractor monster ( e.g . arcane hammer ) . In order to win the game ( e.g . Figure 1 ) , the agent must 1. identify the target team from the goal ( e.g . Order of the Forest ) 2. identify the monsters that belong to that team ( e.g . goblin , jaguar , and ghost ) 3. identify which monster is in the world ( e.g . goblin ) , and its element ( e.g . fire ) 4. identify the modifiers that are effective against this element ( e.g . fanatical , shimmering ) 5. find which modifier is present ( e.g . fanatical ) , and the item with the modifier ( e.g . sword ) 6. pick up the correct item ( e.g . fanatical sword ) 7. engage the correct monster in combat ( e.g . fire goblin ) . If the agent deviates from this trajectory ( e.g . does not have correct item before engaging in combat , engages with distractor monster ) , it can not defeat the target monster and therefore will lose the game . The agent receives a reward of +1 if it wins the game and -1 otherwise . RTFM presents challenges not found in prior work in that it requires a large number of grounding steps in order to solve a task . In order to perform this grounding , the agent must jointly reason over a language goal and document of dynamics , as well as environment observations . In addition to the environment , the positions of the target and distractor within the document are randomised—the agent can not memorise ordering patterns in order to solve the grounding problems , and must instead identify information relevant to the goal and environment at hand . We split environments into train and eval sets . No assignments of monster-team-modifier-element are shared between train and eval to test whether the agent is able to generalise to new environments with dynamics not seen during training via reading . There are more than 2 million train or eval environments without considering the natural language templates , and 200 million otherwise . With random ordering of templates , the number of unique documents exceeds 15 billion . 4 MODEL . We propose the txt2π model , which builds representations that capture three-way interactions between the goal , document describing environment dynamics , and environment observations . We begin with definition of the Bidirectional Feature-wise Linear Modulation ( FiLM2 ) layer , which forms the core of our model . 4.1 BIDIRECTIONAL FEATURE-WISE LINEAR MODULATION ( FILM2 ) LAYER Feature-wise linear modulation ( FiLM ) , which modulates visual inputs using representations of textual instructions , is an effective method for image captioning ( Perez et al. , 2018 ) and instruction following ( Bahdanau et al. , 2019 ) . In RTFM , the agent must not only filter concepts in the visual domain using language but filter concepts in the text domain using visual observations . To support this , FiLM2 builds codependent representations of text and visual inputs by further incorporating conditional representations of the text given visual observations . Figure 2 shows the FiLM2 layer . We use upper-case bold letters to denote tensors , lower-case bold letters for vectors , and non-bold letters for scalars . Exact dimensions of these variables are shown in Table 4 in appendix B . Let xtext denote a fixed-length dtext-dimensional representation of the text andXvis the representation of visual inputs with height H , width W , and dvis channels . Let Conv denote a convolution layer . Let + and * symbols denote element-wise addition and multiplication operations that broadcast over spatial dimensions . We first modulate visual features using text features : γtext = Wγxtext + bγ ( 1 ) βtext = Wβxtext + bβ ( 2 ) Vvis = ReLU ( ( 1 + γtext ) ∗ Convvis ( Xvis ) + βtext ) ( 3 ) Unlike FiLM , we additionally modulate text features using visual features : Γvis = Convγ ( Xvis ) ( 4 ) Bvis = Convβ ( Xvis ) ( 5 ) Vtext = ReLU ( ( 1 + Γvis ) ∗ ( Wtextxtext + btext ) +Bvis ) ( 6 ) The output of the FiLM2 layer consists of the sum of the modulated features V , as well as a max-pooled summary s over this sum across spatial dimensions . V = Vvis + Vtext ( 7 ) s = MaxPool ( V ) ( 8 ) 4.2 THE TXT2π MODEL We model interactions between observations from the environment , goal , and document using FiLM2 layers . We first encode text inputs using bidirectional LSTMs , then compute summaries using self-attention and conditional summaries using attention . We concatenate text summaries into text features , which , along with visual features , are processed through consecutive FiLM2 layers . In this case of a textual environment , we consider the grid of word embeddings as the visual features for FiLM2 . The final FiLM2 output is further processed by MLPs to compute a policy distribution over actions and a baseline for advantage estimation . Figure 3 shows the txt2π model . Let Eobs denote word embeddings corresponding to the observations from the environment , where Eobs [ : , : , i , j ] represents the embeddings corresponding to the lobs-word string that describes the objects in location ( i , j ) in the grid-world . Let Edoc , Einv , and Egoal respectively denote the embeddings corresponding to the ldoc-word document , the linv-word inventory , and the lgoal-word goal . We first compute a fixed-length summary cgoal of the the goal using a bidirectional LSTM ( Hochreiter & Schmidhuber , 1997 ) followed by self-attention ( Lee et al. , 2017 ; Zhong et al. , 2018 ) . Hgoal = BiLSTMgoal ( Egoal ) ( 9 ) a ′ goal , i = wgoalh ᵀ goal , i + bgoal ( 10 ) agoal = softmax ( a ′ goal ) ( 11 ) cgoal = lgoal∑ i=1 agoal , ihgoal , i ( 12 ) We abbreviate self-attention over the goal as cgoal = selfattn ( Hgoal ) . We similarly compute a summary of the inventory as cinv = selfattn ( BiLSTMinv ( Einv ) ) . Next , we represent the document encoding conditioned on the goal using dot-product attention ( Luong et al. , 2015 ) . Hdoc = BiLSTMgoal-doc ( Edoc ) ( 13 ) a′doc , i = cgoalh ᵀ doc , i ( 14 ) adoc = softmax ( a ′ doc ) ( 15 ) cdoc = ldoc∑ i=1 adoc , ihdoc , i ( 16 ) We abbreviate attention over the document encoding conditioned on the goal summary as cdoc = attend ( Hdoc , cgoal ) . Next , we build the joint representation of the inputs using successive FiLM2 layers . At each layer , the visual input to the FiLM2 layer is the concatenation of the output of the previous layer with positional features . For each cell , the positional feature Xpos consists of the x and y distance from the cell to the agent ’ s position respectively , normalized by the width and height of the grid-world . The text input is the concatenation of the goal summary , the inventory summary , the attention over the document given the goal , and the attention over the document given the previous visual summary . Let [ a ; b ] denote the feature-wise concatenation of a Model Win rate Train Eval 6×6 Eval 10×10 conv 24± 0 25± 1 13± 1 FiLM 49± 1 49± 2 32± 3 no task attn 49± 2 49± 2 35± 6 no vis attn 49± 2 49± 1 40±12 no text mod 49± 1 49± 2 35± 2 txt2π 84±21 83±21 66±22 Table 1 : Final win rate on simplest variant of RTFM . The models are trained on one set of dynamics ( e.g . training set ) and evaluated on another set of dynamics ( e.g . evaluation set ) . “ Train ” and “ Eval ” show final win rates on training and eval environments . and b . For the ith layer , we have R ( i ) = [ V ( i−1 ) ; Xpos ] ( 17 ) T ( i ) = [ cgoal ; cinv ; cdoc ; attend ( BiLSTMvis-doc ( Edoc ) , s ( i−1 ) ) ] ( 18 ) V ( i ) , s ( i ) = FiLM2 ( i ) ( R ( i ) , T ( i ) ) ( 19 ) BiLSTMvis-doc ( Edoc ) is another encoding of the document similar toHgoal , produced using a separate LSTM , such that the document is encoded differently for attention with the visual features and with the goal . For i = 0 , we concatenate the bag-of-words embeddings of the grid with positional features as the initial visual features V ( 0 ) = [ ∑ j Eobs , j ; Xpos ] . We max pool a linear transform of the initial visual features to compute the initial visual summary s ( 0 ) = MaxPool ( WiniV ( 0 ) +bini ) . Let s ( last ) denote visual summary of the last FiLM2 layer . We compute the policy ypolicy and baseline ybaseline as o = ReLU ( Wos ( last ) + bo ) ( 20 ) ypolicy = MLPpolicy ( o ) ( 21 ) ybaseline = MLPbaseline ( o ) ( 22 ) where MLPpolicy and MLPbaseline are 2-layer multi-layer perceptrons with ReLU activation . We train using TorchBeast ( Küttler et al. , 2019 ) , an implementation of IMPALA ( Espeholt et al. , 2018 ) . Please refer to appendix D for details .

Summary: This work proposes a new environment, Read to Fight Monsters (RTFM), and correspondingly a new algorithm, txt2\pi, for solving this problem. The RTFM requires the agent to read a description of the rules (x beats y, etc) and a description of the goal (to eliminate y), and perform the task correctly to win the game. The txt2\pi algorithm uses the newly proposed FiLM^2 module and consists of integrating the visual input (grid world configuration) and the text input (descriptions) to learn a policy and baseline (actor-critic).