RTFM: Generalising to New Environment Dynamics via Reading

1 INTRODUCTION . Reinforcement learning ( RL ) has been successful in a variety of areas such as continuous control ( Lillicrap et al. , 2015 ) , dialogue systems ( Li et al. , 2016 ) , and game-playing ( Mnih et al. , 2013 ) . However , RL adoption in real-world problems is limited due to poor sample efficiency and failure to generalise to environments even slightly different from those seen during training . We explore language-conditioned policy learning , where agents use machine reading to discover strategies required to solve a task , thereby leveraging language as a means to generalise to new environments . Prior work on language grounding and language-based RL ( see Luketina et al . ( 2019 ) for a recent survey ) are limited to scenarios in which language specifies the goal for some fixed environment dynamics ( Branavan et al. , 2011 ; Hermann et al. , 2017 ; Bahdanau et al. , 2019 ; Fried et al. , 2018 ; Co-Reyes et al. , 2019 ) , or the dynamics of the environment vary and are presented in language for some fixed goal ( Branavan et al. , 2012 ) . In practice , changes to goals and to environment dynamics tend to occur simultaneously—given some goal , we need to find and interpret relevant information to understand how to achieve the goal . That is , the agent should account for variations in both by selectively reading , thereby generalising to environments with dynamics not seen during training . Our contributions are two-fold . First , we propose a grounded policy learning problem that we call Read to Fight Monsters ( RTFM ) . In RTFM , the agent must jointly reason over a language goal , a document that specifies environment dynamics , and environment observations . In particular , it must identify relevant information in the document to shape its policy and accomplish the goal . To necessitate reading comprehension , we expose the agent to ever changing environment dynamics and corresponding language descriptions such that it can not avoid reading by memorising any particular environment dynamics . We procedurally generate environment dynamics and natural language templated descriptions of dynamics and goals to produced a combinatorially large number of environment dynamics to train and evaluate RTFM . ∗Work done during an internship at Facebook AI Research . Second , we propose txt2π to model the joint reasoning problem in RTFM . We show that txt2π generalises to goals and environment dynamics not seen during training , and outperforms previous language-conditioned models such as language-conditioned CNNs and FiLM ( Perez et al. , 2018 ; Bahdanau et al. , 2019 ) both in terms of sample efficiency and final win-rate on RTFM . Through curriculum learning where we adapt txt2π trained on simpler tasks to more complex tasks , we obtain agents that generalise to tasks with natural language documents that require five hops of reasoning between the goal , document , and environment observations . Our qualitative analyses show that txt2π attends to parts of the document relevant to the goal and environment observations , and that the resulting agents exhibit complex behaviour such as retrieving correct items , engaging correct enemies after acquiring correct items , and avoiding incorrect enemies . Finally , we highlight the complexity of RTFM in scaling to longer documents , richer dynamics , and natural language variations . We show that significant improvement in language-grounded policy learning is needed to solve these problems in the future . 2 RELATED WORK . Language-conditioned policy learning . A growing body of research is learning policies that follow imperative instructions . The granularity of instructions vary from high-level instructions for application control ( Branavan , 2012 ) and games ( Hermann et al. , 2017 ; Bahdanau et al. , 2019 ) to step-by-step navigation ( Fried et al. , 2018 ) . In contrast to learning policies for imperative instructions , Branavan et al . ( 2011 ; 2012 ) ; Narasimhan et al . ( 2018 ) infer a policy for a fixed goal using features extracted from high level strategy descriptions and general information about domain dynamics . Unlike prior work , we study the combination of imperative instructions and descriptions of dynamics . Furthermore , we require that the agent learn to filter out irrelevant information to focus on dynamics relevant to accomplishing the goal . Language grounding . Language grounding refers to interpreting language in a non-linguistic context . Examples of such context include images ( Barnard & Forsyth , 2001 ) , games ( Chen & Mooney , 2008 ; Wang et al. , 2016 ) , robot control ( Kollar et al. , 2010 ; Tellex et al. , 2011 ) , and navigation ( Anderson et al. , 2018 ) . We study language grounding in interactive games similar to Branavan ( 2012 ) ; Hermann et al . ( 2017 ) or Co-Reyes et al . ( 2019 ) , where executable semantics are not provided and the agent must learn through experience . Unlike prior work , we require grounding between an underspecified goal , a document of environment dynamics , and world observations . In addition , we focus on generalisation to not only new goal descriptions but new environments dynamics . 3 READ TO FIGHT MONSTERS . We consider a scenario where the agent must jointly reason over a language goal , relevant environment dynamics specified in a text document , and environment observations . In reading the document , the agent should identify relevant information key to solving the goal in the environment . A successful agent needs to perform this language grounding to generalise to new environments with dynamics not seen during training . To study generalisation via reading , the environment dynamics must differ every episode such that the agent can not avoid reading by memorising a limited set of dynamics . Consequently , we procedurally generate a large number of unique environment dynamics ( e.g . effective ( blessed items , poison monsters ) ) , along with language descriptions of environment dynamics ( e.g . blessed items are effective against poison monsters ) and goals ( e.g . Defeat the order of the forest ) . We couple a large , customisable ontology inspired by rogue-like games such as NetHack or Diablo , with natural language templates to create a combinatorially rich set of environment dynamics to learn from and evaluate on . In RTFM , the agent is given a document of environment dynamics , observations of the environment , and an underspecified goal instruction . Figure 1 illustrates an instance of the game . Concretely , we design a set of dynamics that consists of monsters ( e.g . wolf , goblin ) , teams ( e.g . Order of the Forest ) , element types ( e.g . fire , poison ) , item modifiers ( e.g . fanatical , arcane ) , and items ( e.g . sword , hammer ) . When the player is in the same cell with a monster or weapon , the player picks up the item or engages in combat with the monster . The player can possess one item at a time , and drops existing weapons if they pick up a new weapon . A monster moves towards the player with 60 % probability , and otherwise moves randomly . The dynamics , the agent ’ s inventory , and the underspecified goal are rendered as text . The game world is rendered as a matrix of text in which each cell describes the entity occupying the cell . We use human-written templates for stating which monsters belong to which team , which modifiers are effective against which element , and which team the agent should defeat ( see appendix H for details on collection and G for a list of entities in the game ) . In order to achieve the goal , the agent must cross-reference relevant information in the document and as well as in the observations . During every episode , we subsample a set of groups , monsters , modifiers , and elements to use . We randomly generate group assignments of which monsters belong to which team and which modifier is effective against which element . A document that consists of randomly ordered statements corresponding to this group assignment is presented to the agent . We sample one element , one team , and a monster from that team ( e.g . “ fire goblin ” from “ Order of the forest ” ) to be the target monster . Additionally , we sample one modifier that beats the element and an item to be the item that defeats the target monster ( e.g . “ fanatical sword ” ) . Similarly , we sample an element , a team , and a monster from a different team to be the distractor monster ( e.g . poison bat ) , as well as an item that defeats the distractor monster ( e.g . arcane hammer ) . In order to win the game ( e.g . Figure 1 ) , the agent must 1. identify the target team from the goal ( e.g . Order of the Forest ) 2. identify the monsters that belong to that team ( e.g . goblin , jaguar , and ghost ) 3. identify which monster is in the world ( e.g . goblin ) , and its element ( e.g . fire ) 4. identify the modifiers that are effective against this element ( e.g . fanatical , shimmering ) 5. find which modifier is present ( e.g . fanatical ) , and the item with the modifier ( e.g . sword ) 6. pick up the correct item ( e.g . fanatical sword ) 7. engage the correct monster in combat ( e.g . fire goblin ) . If the agent deviates from this trajectory ( e.g . does not have correct item before engaging in combat , engages with distractor monster ) , it can not defeat the target monster and therefore will lose the game . The agent receives a reward of +1 if it wins the game and -1 otherwise . RTFM presents challenges not found in prior work in that it requires a large number of grounding steps in order to solve a task . In order to perform this grounding , the agent must jointly reason over a language goal and document of dynamics , as well as environment observations . In addition to the environment , the positions of the target and distractor within the document are randomised—the agent can not memorise ordering patterns in order to solve the grounding problems , and must instead identify information relevant to the goal and environment at hand . We split environments into train and eval sets . No assignments of monster-team-modifier-element are shared between train and eval to test whether the agent is able to generalise to new environments with dynamics not seen during training via reading . There are more than 2 million train or eval environments without considering the natural language templates , and 200 million otherwise . With random ordering of templates , the number of unique documents exceeds 15 billion . 4 MODEL . We propose the txt2π model , which builds representations that capture three-way interactions between the goal , document describing environment dynamics , and environment observations . We begin with definition of the Bidirectional Feature-wise Linear Modulation ( FiLM2 ) layer , which forms the core of our model . 4.1 BIDIRECTIONAL FEATURE-WISE LINEAR MODULATION ( FILM2 ) LAYER Feature-wise linear modulation ( FiLM ) , which modulates visual inputs using representations of textual instructions , is an effective method for image captioning ( Perez et al. , 2018 ) and instruction following ( Bahdanau et al. , 2019 ) . In RTFM , the agent must not only filter concepts in the visual domain using language but filter concepts in the text domain using visual observations . To support this , FiLM2 builds codependent representations of text and visual inputs by further incorporating conditional representations of the text given visual observations . Figure 2 shows the FiLM2 layer . We use upper-case bold letters to denote tensors , lower-case bold letters for vectors , and non-bold letters for scalars . Exact dimensions of these variables are shown in Table 4 in appendix B . Let xtext denote a fixed-length dtext-dimensional representation of the text andXvis the representation of visual inputs with height H , width W , and dvis channels . Let Conv denote a convolution layer . Let + and * symbols denote element-wise addition and multiplication operations that broadcast over spatial dimensions . We first modulate visual features using text features : γtext = Wγxtext + bγ ( 1 ) βtext = Wβxtext + bβ ( 2 ) Vvis = ReLU ( ( 1 + γtext ) ∗ Convvis ( Xvis ) + βtext ) ( 3 ) Unlike FiLM , we additionally modulate text features using visual features : Γvis = Convγ ( Xvis ) ( 4 ) Bvis = Convβ ( Xvis ) ( 5 ) Vtext = ReLU ( ( 1 + Γvis ) ∗ ( Wtextxtext + btext ) +Bvis ) ( 6 ) The output of the FiLM2 layer consists of the sum of the modulated features V , as well as a max-pooled summary s over this sum across spatial dimensions . V = Vvis + Vtext ( 7 ) s = MaxPool ( V ) ( 8 ) 4.2 THE TXT2π MODEL We model interactions between observations from the environment , goal , and document using FiLM2 layers . We first encode text inputs using bidirectional LSTMs , then compute summaries using self-attention and conditional summaries using attention . We concatenate text summaries into text features , which , along with visual features , are processed through consecutive FiLM2 layers . In this case of a textual environment , we consider the grid of word embeddings as the visual features for FiLM2 . The final FiLM2 output is further processed by MLPs to compute a policy distribution over actions and a baseline for advantage estimation . Figure 3 shows the txt2π model . Let Eobs denote word embeddings corresponding to the observations from the environment , where Eobs [ : , : , i , j ] represents the embeddings corresponding to the lobs-word string that describes the objects in location ( i , j ) in the grid-world . Let Edoc , Einv , and Egoal respectively denote the embeddings corresponding to the ldoc-word document , the linv-word inventory , and the lgoal-word goal . We first compute a fixed-length summary cgoal of the the goal using a bidirectional LSTM ( Hochreiter & Schmidhuber , 1997 ) followed by self-attention ( Lee et al. , 2017 ; Zhong et al. , 2018 ) . Hgoal = BiLSTMgoal ( Egoal ) ( 9 ) a ′ goal , i = wgoalh ᵀ goal , i + bgoal ( 10 ) agoal = softmax ( a ′ goal ) ( 11 ) cgoal = lgoal∑ i=1 agoal , ihgoal , i ( 12 ) We abbreviate self-attention over the goal as cgoal = selfattn ( Hgoal ) . We similarly compute a summary of the inventory as cinv = selfattn ( BiLSTMinv ( Einv ) ) . Next , we represent the document encoding conditioned on the goal using dot-product attention ( Luong et al. , 2015 ) . Hdoc = BiLSTMgoal-doc ( Edoc ) ( 13 ) a′doc , i = cgoalh ᵀ doc , i ( 14 ) adoc = softmax ( a ′ doc ) ( 15 ) cdoc = ldoc∑ i=1 adoc , ihdoc , i ( 16 ) We abbreviate attention over the document encoding conditioned on the goal summary as cdoc = attend ( Hdoc , cgoal ) . Next , we build the joint representation of the inputs using successive FiLM2 layers . At each layer , the visual input to the FiLM2 layer is the concatenation of the output of the previous layer with positional features . For each cell , the positional feature Xpos consists of the x and y distance from the cell to the agent ’ s position respectively , normalized by the width and height of the grid-world . The text input is the concatenation of the goal summary , the inventory summary , the attention over the document given the goal , and the attention over the document given the previous visual summary . Let [ a ; b ] denote the feature-wise concatenation of a Model Win rate Train Eval 6×6 Eval 10×10 conv 24± 0 25± 1 13± 1 FiLM 49± 1 49± 2 32± 3 no task attn 49± 2 49± 2 35± 6 no vis attn 49± 2 49± 1 40±12 no text mod 49± 1 49± 2 35± 2 txt2π 84±21 83±21 66±22 Table 1 : Final win rate on simplest variant of RTFM . The models are trained on one set of dynamics ( e.g . training set ) and evaluated on another set of dynamics ( e.g . evaluation set ) . “ Train ” and “ Eval ” show final win rates on training and eval environments . and b . For the ith layer , we have R ( i ) = [ V ( i−1 ) ; Xpos ] ( 17 ) T ( i ) = [ cgoal ; cinv ; cdoc ; attend ( BiLSTMvis-doc ( Edoc ) , s ( i−1 ) ) ] ( 18 ) V ( i ) , s ( i ) = FiLM2 ( i ) ( R ( i ) , T ( i ) ) ( 19 ) BiLSTMvis-doc ( Edoc ) is another encoding of the document similar toHgoal , produced using a separate LSTM , such that the document is encoded differently for attention with the visual features and with the goal . For i = 0 , we concatenate the bag-of-words embeddings of the grid with positional features as the initial visual features V ( 0 ) = [ ∑ j Eobs , j ; Xpos ] . We max pool a linear transform of the initial visual features to compute the initial visual summary s ( 0 ) = MaxPool ( WiniV ( 0 ) +bini ) . Let s ( last ) denote visual summary of the last FiLM2 layer . We compute the policy ypolicy and baseline ybaseline as o = ReLU ( Wos ( last ) + bo ) ( 20 ) ypolicy = MLPpolicy ( o ) ( 21 ) ybaseline = MLPbaseline ( o ) ( 22 ) where MLPpolicy and MLPbaseline are 2-layer multi-layer perceptrons with ReLU activation . We train using TorchBeast ( Küttler et al. , 2019 ) , an implementation of IMPALA ( Espeholt et al. , 2018 ) . Please refer to appendix D for details .

Summary: This paper constructs a new game that requires combining visual reasoning with text understanding to win.  The authors propose a new model txt2π, based on a new layer called FiLM², which combines text and visual features in a way that allows visual features to be encoded with knowledge of the text features (as in the FiLM layer from previous work), as well as text to be encoded with knowledge of the visual features. The model is trained to play the game using IMPALA.  Ablation studies show that the FiLM² layer leads to a substantial improvement, and also shows the necessity of curriculum learning.  Performance is still below human performance suggesting this is a promising area for future work.