Title: RTFM: GENERALISING TO NOVEL ENVIRONMENT DYNAMICS VIA READING

Abstract: Obtaining policies that can generalise to new environments in reinforcement learning is challenging. In this work, we demonstrate that language understanding via a reading policy learner is a promising vehicle for generalisation to new environments. We propose a grounded policy learning problem, Read to Fight Monsters (RTFM), in which the agent must jointly reason over a language goal, relevant dynamics described in a document, and environment observations. We procedurally generate environment dynamics and corresponding language descriptions of the dynamics, such that agents must read to understand new environment dynamics instead of memorising any particular information. In addition, we propose txt2π, a model that captures three-way interactions between the goal, document, and observations. On RTFM, txt2π generalises to new environments with dynamics not seen during training via reading. Furthermore, our model outperforms baselines such as FiLM and language-conditioned CNNs on RTFM. Through curriculum learning, txt2π produces policies that excel on complex RTFM tasks requiring several reasoning and coreference steps.

1. INTRODUCTION:

Reinforcement learning (RL) has been successful in a variety of areas such as continuous control (Lillicrap et al., 2015) , dialogue systems (Li et al., 2016) , and game-playing (Mnih et al., 2013) . However, RL adoption in real-world problems is limited due to poor sample efficiency and failure to generalise to environments even slightly different from those seen during training. We explore language-conditioned policy learning, where agents use machine reading to discover strategies required to solve a task, thereby leveraging language as a means to generalise to new environments.

Prior work on language grounding and language-based RL (see Luketina et al. (2019) for a recent survey) are limited to scenarios in which language specifies the goal for some fixed environment dynamics (Branavan et al., 2011; Hermann et al., 2017; Bahdanau et al., 2019; Fried et al., 2018; Co-Reyes et al., 2019) , or the dynamics of the environment vary and are presented in language for some fixed goal (Branavan et al., 2012) . In practice, changes to goals and to environment dynamics tend to occur simultaneously-given some goal, we need to find and interpret relevant information to understand how to achieve the goal. That is, the agent should account for variations in both by selectively reading, thereby generalising to environments with dynamics not seen during training.

Our contributions are two-fold. First, we propose a grounded policy learning problem that we call Read to Fight Monsters (RTFM). In RTFM, the agent must jointly reason over a language goal, a document that specifies environment dynamics, and environment observations. In particular, it must identify relevant information in the document to shape its policy and accomplish the goal. To necessitate reading comprehension, we expose the agent to ever changing environment dynamics and corresponding language descriptions such that it cannot avoid reading by memorising any particular environment dynamics. We procedurally generate environment dynamics and natural language templated descriptions of dynamics and goals to produced a combinatorially large number of environment dynamics to train and evaluate RTFM.

Second, we propose txt2π to model the joint reasoning problem in RTFM. We show that txt2π generalises to goals and environment dynamics not seen during training, and outperforms previous language-conditioned models such as language-conditioned CNNs and FiLM (Perez et al., 2018; Bahdanau et al., 2019) both in terms of sample efficiency and final win-rate on RTFM. Through curriculum learning where we adapt txt2π trained on simpler tasks to more complex tasks, we obtain agents that generalise to tasks with natural language documents that require five hops of reasoning between the goal, document, and environment observations. Our qualitative analyses show that txt2π attends to parts of the document relevant to the goal and environment observations, and that the resulting agents exhibit complex behaviour such as retrieving correct items, engaging correct enemies after acquiring correct items, and avoiding incorrect enemies. Finally, we highlight the complexity of RTFM in scaling to longer documents, richer dynamics, and natural language variations. We show that significant improvement in language-grounded policy learning is needed to solve these problems in the future.

2. RELATED WORK:

Language-conditioned policy learning. A growing body of research is learning policies that follow imperative instructions. The granularity of instructions vary from high-level instructions for application control (Branavan, 2012) and games (Hermann et al., 2017; Bahdanau et al., 2019) to step-by-step navigation (Fried et al., 2018) . In contrast to learning policies for imperative instructions, Branavan et al. (2011; 2012) ; Narasimhan et al. (2018) infer a policy for a fixed goal using features extracted from high level strategy descriptions and general information about domain dynamics. Unlike prior work, we study the combination of imperative instructions and descriptions of dynamics. Furthermore, we require that the agent learn to filter out irrelevant information to focus on dynamics relevant to accomplishing the goal.

Language grounding. Language grounding refers to interpreting language in a non-linguistic context. Examples of such context include images (Barnard & Forsyth, 2001) , games (Chen & Mooney, 2008; Wang et al., 2016) , robot control (Kollar et al., 2010; Tellex et al., 2011) , and navigation (Anderson et al., 2018) . We study language grounding in interactive games similar to Branavan (2012) ; Hermann et al. (2017) or Co-Reyes et al. (2019) , where executable semantics are not provided and the agent must learn through experience. Unlike prior work, we require grounding between an underspecified goal, a document of environment dynamics, and world observations. In addition, we focus on generalisation to not only new goal descriptions but new environments dynamics.

3. READ TO FIGHT MONSTERS:

We consider a scenario where the agent must jointly reason over a language goal, relevant environment dynamics specified in a text document, and environment observations. In reading the document, the agent should identify relevant information key to solving the goal in the environment. A successful agent needs to perform this language grounding to generalise to new environments with dynamics not seen during training.

To study generalisation via reading, the environment dynamics must differ every episode such that the agent cannot avoid reading by memorising a limited set of dynamics. Consequently, we procedurally generate a large number of unique environment dynamics (e.g. effective(blessed items, poison monsters)), along with language descriptions of environment dynamics (e.g. blessed items are effective against poison monsters) and goals (e.g. Defeat the order of the forest). We couple a large, customisable ontology inspired by rogue-like games such as NetHack or Diablo, with natural language templates to create a combinatorially rich set of environment dynamics to learn from and evaluate on.

In RTFM, the agent is given a document of environment dynamics, observations of the environment, and an underspecified goal instruction. Figure 1 illustrates an instance of the game. Concretely, we design a set of dynamics that consists of monsters (e.g. wolf, goblin), teams (e.g. Order of the Forest), element types (e.g. fire, poison), item modifiers (e.g. fanatical, arcane), and items (e.g. sword, hammer). When the player is in the same cell with a monster or weapon, the player picks up the item or engages in combat with the monster. The player can possess one item at a time, and drops existing weapons if they pick up a new weapon. A monster moves towards the player with 60% probability, and otherwise moves randomly. The dynamics, the agent's inventory, and the underspecified goal are rendered as text. The game world is rendered as a matrix of text in which each cell describes the entity occupying the cell. We use human-written templates for stating which monsters belong to which team, which modifiers are effective against which element, and which team the agent should defeat (see appendix H for details on collection and G for a list of entities in the game). In order to achieve the goal, the agent must cross-reference relevant information in the document and as well as in the observations.

During every episode, we subsample a set of groups, monsters, modifiers, and elements to use. We randomly generate group assignments of which monsters belong to which team and which modifier is effective against which element. A document that consists of randomly ordered statements corresponding to this group assignment is presented to the agent. We sample one element, one team, and a monster from that team (e.g. "fire goblin" from "Order of the forest") to be the target monster.

Additionally, we sample one modifier that beats the element and an item to be the item that defeats the target monster (e.g. "fanatical sword"). Similarly, we sample an element, a team, and a monster from a different team to be the distractor monster (e.g. poison bat), as well as an item that defeats the distractor monster (e.g. arcane hammer).

In order to win the game (e.g. Figure 1 ), the agent must 1. identify the target team from the goal (e.g. Order of the Forest)

2. identify the monsters that belong to that team (e.g. goblin, jaguar, and ghost)

3. identify which monster is in the world (e.g. goblin), and its element (e.g. fire)

4. identify the modifiers that are effective against this element (e.g. fanatical, shimmering)

5. find which modifier is present (e.g. fanatical), and the item with the modifier (e.g. sword)

6. pick up the correct item (e.g. fanatical sword) 7. engage the correct monster in combat (e.g. fire goblin).

If the agent deviates from this trajectory (e.g. does not have correct item before engaging in combat, engages with distractor monster), it cannot defeat the target monster and therefore will lose the game. The agent receives a reward of +1 if it wins the game and -1 otherwise.

RTFM presents challenges not found in prior work in that it requires a large number of grounding steps in order to solve a task. In order to perform this grounding, the agent must jointly reason over a language goal and document of dynamics, as well as environment observations. In addition to the environment, the positions of the target and distractor within the document are randomised-the agent cannot memorise ordering patterns in order to solve the grounding problems, and must instead identify information relevant to the goal and environment at hand.

We split environments into train and eval sets. No assignments of monster-team-modifier-element are shared between train and eval to test whether the agent is able to generalise to new environments with dynamics not seen during training via reading. There are more than 2 million train or eval environments without considering the natural language templates, and 200 million otherwise. With random ordering of templates, the number of unique documents exceeds 15 billion.

4. MODEL:

We propose the txt2π model, which builds representations that capture three-way interactions between the goal, document describing environment dynamics, and environment observations. We begin with definition of the Bidirectional Feature-wise Linear Modulation (FiLM 2 ) layer, which forms the core of our model.

4.1. BIDIRECTIONAL FEATURE-WISE LINEAR MODULATION (FILM 2 ) LAYER:



Text features:



Visual features:

Linear < l a t e x i t s h a 1 _ b a s e 6 4 = " b w 8 7

x d X k n 4 0 R Q u 1 2 V W W A L P t H / 7 s = " > A A A B / n i c b V D L S s N A F J 3 4 r P U V F V d u g k V w V Z I q 6 L L g x o W L C v Y B T Q i T 6 U 0 7 d G Y S Z i Z C C Q V / x Y 0 L R d z

6 H e 7 8 G y d t F t p 6 Y O B w z r 0 z Z 0 6 U M q q 0 6 3 5 b K 6 t r 6 x u b l a 3 q 9 s 7 u 3 r 5 9

c N h R S S Y J t E n C E t m L s A J G B b Q 1 1 Q x 6 q Q T M I w b d a H x T + N 1 H k I o m 4 k F P U g g 4 H g o a U 4 K 1 k U L 7 2 O d Y j y T P 7 8 w F W E 5 D P w K N Q 7 v m 1 t 0 Z n G X i l a S G S r R C + 8 s f J C T j I D R h W K m + 5 6 Y 6 y L H U l D C Y V v 1 M Q Y r J G A + h b 6 j A H F S Q z + J P n T O j D J w 4 k e Y I 7 c z U 3 x s 5 5 k p N e G Q m i 7 B q 0 S v E / 7 x + p u P r I K c i z T Q I M n 8 o z p i j E 6 f o w h l Q C U S z i S G Y S G q y O m S E J S b a N F Y 1 J X i L X 1 4 m n U b d u 6 g 3 7 i 9 r T b e s o 4 J O 0 C k 6 R x 6 6 Q k 1 0 i 1 q o j Q j K 0 T N 6 R W / W k / V i v V

s f 8 9 E V q 9 w 5 Q n 9 g f f 4 A 1 4 O V / g = = < / l a t e x i t > Linear < l a t e x i t s h a 1 _ b a s e 6 4 = " b w 8 7

x d X k n 4 0 R Q u 1 2 V W W A L P t H / 7 s = " > A A A B / n i c b V D L S s N A F J 3 4 r P U V F V d u g k V w V Z I q 6 L L g x o W L C v Y B T Q i T 6 U 0 7 d G Y S Z i Z C C Q V / x Y 0 L R d z

6 H e 7 8 G y d t F t p 6 Y O B w z r 0 z Z 0 6 U M q q 0 6 3 5 b K 6 t r 6 x u b l a 3 q 9 s 7 u 3 r 5 9

c N h R S S Y J t E n C E t m L s A J G B b Q 1 1 Q x 6 q Q T M I w b d a H x T + N 1 H k I o m 4 k F P U g g 4 H g o a U 4 K 1 k U L 7 2 O d Y j y T P 7 8 w F W E 5 D P w K N Q 7 v m 1 t 0 Z n G X i l a S G S r R C + 8 s f J C T j I D R h W K m + 5 6 Y 6 y L H U l D C Y V v 1 M Q Y r J G A + h b 6 j A H F S Q z + J P n T O j D J w 4 k e Y I 7 c z U 3 x s 5 5 k p N e G Q m i 7 B q 0 S v E / 7 x + p u P r I K c i z T Q I M n 8 o z p i j E 6 f o w h l Q C U S z i S G Y S G q y O m S E J S b a N F Y 1 J X i L X 1 4 m n U b d u 6 g 3 7 i 9 r T b e s o 4 J O 0 C k 6 R x 6 6 Q k 1 0 i 1 q o j Q j K 0 T N 6 R W / W k / V i v V s f 8 9 E V q 9 w 5 Q n 9 g f f 4 A 1 4 O V / g = = < / l a t e x i t > Lineartext < l a t e x i t s h a 1 _ b a s e 6 4 = " m v M s i 3 R f z m g e L z X S v J w X 5 8 f x v j I = " > A A A C B n i c b V D L S s N A F J 3 U V 6 2 v q E s R g k V w V Z I q 6 L L g x o W L C v Y B b Q i T 6 a Q d O p O E m R u x h K z c + C t u X C j i 1 m 9 w 5 9 8 4 a S N o 6 4 G B M + f c O 3 P v 8 W P O F N j 2 l 1 F a W l 5 Z X S u v V z Y 2 t 7 Z 3 z N 2 9 t o o S S W i L R D y S X R 8 r y l l I W 8 C A 0 2 4 s K R Y + p x 1 / f J n 7 n T s q F Y v C W 5 j E 1 B V 4 G L K A E Q x a 8 s z D v s A w k i K 9 1 g 9 g m X k / d 6 D 3 k H l m 1 a 7 Z U 1 i L x C l I F R V o e u Z n f x C R R N A Q C M d K 9 R w 7 B j f F E h j h N K v 0 E 0 V j T M Z 4 S H u a h l h Q 5 a b T N T L r W C s D K 4 i k P i F Y U / V 3 R 4 q F U h P h 6 8 p 8 S D X v 5 e J / X i + B 4 M J N W R g n Q E M y + y h I u A W R l W d i D Z i k B P h E E 0 w k 0 7 N a Z I Q l J q C T q + g Q n P m V F 0 m 7 X n N O a / W b s 2 r D L u I o o w N 0 h E 6 Q g 8 5 R A 1 2 h J m o h g h 7 Q E 3 p B r 8 a j 8 W y 8 G e + z 0 p J R 9 O y j P z A + v g H g m 5 n 4 < / l a t e x i t > Lineartext < l a t e x i t s h a 1 _ b a s e 6 4 = " m v M s i 3 R f z m g e L z X S v J w X 5 8 f x v j I = " > A A A C B n i c b V D L S s N A F J 3 U V 6 2 v q E s R g k V w V Z I q 6 L L g x o W L C v Y B b Q i T 6 a Q d O p O E m R u x h K z c + C t u X C j i 1 m 9 w 5 9 8 4 a S N o 6 4 G B M + f c O 3 P v 8 W P O F N j 2 l 1 F a W l 5 Z X S u v V z Y 2 t 7 Z 3 z N 2 9 t o o S S W i L R D y S X R 8 r y l l I W 8 C A 0 2 4 s K R Y + p x 1 / f J n 7 n T s q F Y v C W 5 j E 1 B V 4 G L K A E Q x a 8 s z D v s A w k i K 9 1 g 9 g m X k / d 6 D 3 k H l m 1 a 7 Z U 1 i L x C l I F R V o e u Z n f x C R R N A Q C M d K 9 R w 7 B j f F E h j h N K v 0 E 0 V j T M Z 4 S H u a h l h Q 5 a b T N T L r W C s D K 4 i k P i F Y U / V 3 R 4 q F U h P h 6 8 p 8 S D X v 5 e J / X i + B 4 M J N W R g n Q E M y + y h I u A W R l W d i D Z i k B P h E E 0 w k 0 7 N a Z I Q l J q C T q + g Q n P m V F 0 m 7 X n N O a / W b s 2 r D L u I o o w N 0 h E 6 Q g 8 5 R A 1 2 h J m o h g h 7 Q E 3 p B r 8 a j 8 W y 8 G e + z 0 p J R 9 O y j P z A + v g H g m 5 n 4 < / l a t e x i t > Conv < l a t e x i t s h a 1 _ b a s e 6 4 = " D 4 M n J A P 3 9 B L q p g C 1 Q G X e A 8 c a F 9 o = " > A A A B / X i c b V D L S g M x F M 3 4 r P U 1 P n Z u g k V w V W a q o M t C N y 4 r 2 A d 0 h i G T p m 1 o H k O S K d S h + C t u X C j i 1 v 9 w 5 9 + Y a W e h r Q c C h 3 P u 5 Z 6 c O G F U G 8 / 7 d t b W N z a 3 t k s 7 5 d 2 9 / Y N D 9 + i 4 r W W q M G l h y a T q x k g T R g V p G W o Y 6 S a K I B 4 z 0 o n H j d z v T I j S V I o H M 0 1 I y N F Q 0 A H F y F g p c k 8 D j s x I 8 a w h x W Q W B U P E O Y r c i l f 1 5 o C r x C 9 I B R R o R u 5 X 0 J c 4 5 U Q Y z J D W P d 9 L T J g h Z S h m Z F Y O U k 0 S h M d o S H q W C s S J D r N 5 + h m 8 s E o f D q S y T x g 4 V 3 9 v Z I h r P e W x n c y z 6 m U v F / / z e q k Z 3 I Y Z F U l q i M C L Q 4 O U Q S N h X g X s U 0 W w Y V N L E F b U Z o V 4 h B T C x h Z W t i X 4 y 1 9 e J e 1 a 1 b + q 1 u 6 v K 3 W v q K M E z s A 5 u A Q + u A F 1 c A e a o A U w e A T P 4 B W 8 O U / O i / P u f C x G 1 5 x i 5 w T 8 g f P 5 A x w P l Z Y = < / l a t e x i t > Conv < l a t e x i t s h a 1 _ b a s e 6 4 = " D 4 M n J A P 3 9 B L q p g C 1 Q G X e A 8 c a F 9 o = " > A A A B / X i c b V D L S g M x F M 3 4 r P U 1 P n Z u g k V w V W a q o M t C N y 4 r 2 A d 0 h i G T p m 1 o H k O S K d S h + C t u X C j i 1 v 9 w 5 9 + Y a W e h r Q c C h 3 P u 5 Z 6 c O G F U G 8 / 7 d t b W N z a 3 t k s 7 5 d 2 9 / Y N D 9 + i 4 r W W q M G l h y a T q x k g T R g V p G W o Y 6 S a K I B 4 z 0 o n H j d z v T I j S V I o H M 0 1 I y N F Q 0 A H F y F g p c k 8 D j s x I 8 a w h x W Q W B U P E O Y r c i l f 1 5 o C r x C 9 I B R R o R u 5 X 0 J c 4 5 U Q Y z J D W P d 9 L T J g h Z S h m Z F Y O U k 0 S h M d o S H q W C s S J D r N 5 + h m 8 s E o f D q S y T x g 4 V 3 9 v Z I h r P e W x n c y z 6 m U v F / / z e q k Z 3 I Y Z F U l q i M C L Q 4 O U Q S N h X g X s U 0 W w Y V N L E F b U Z o V 4 h B T C x h Z W t i X 4 y 1 9 e J e 1 a 1 b + q 1 u 6 v K 3 W v q K M E z s A 5 u A Q + u A F 1 c A e a o A U w e A T P 4 B W 8 O U / O i / P u f C x G 1 5 x i 5 w T 8 g f P 5 A x w P l Z Y = < / l a t e x i t > Linear < l a t e x i t s h a 1 _ b a s e 6 4 = " J u U F x 7 3 b V K g A f a + d d 6 h w z A I L 8 i o = " > A A A B / 3 i c b V D L S g M x F M 3 U V 6 2 v U c G N m 2 A R X J W Z K u i y 4 M a F i w r 2 A Z 1 h u J O m b W g y M y Q Z o Y x d + C t u X C j i 1 t 9 w 5 9 + Y a W e h r Q c C h 3 P u T U 5 O m H C m t O N 8 W 6 W V 1 b X 1 j f J m Z W t 7 Z 3 f P 3 j 9 o q z i V h L Z I z G P Z D U F R z i L a 0 k x z 2 k 0 k B R F y 2 g n H 1 7 n f e a B S s T i 6 1 5 O E + g K G E R s w A t p I g X 3 k C d A j K b J b c w H I a e A N Q Q g I 7 K p T c 2 b A y 8 Q t S B U V a A b 2 l 9 e P S S p o p A k H p X q u k 2 g / A 6 k Z 4 X R a 8 V J F E y B j G N K e o R E I q v x s l n + K T 4 3 S x 4 N Y m h N p P F N / b 2 Q g l J q I 0 E z m a d W i l 4 v / e b 1 U D 6 7 8 j E V J q m l E 5 g 8 N U o 5 1 j P M y c J 9 J S j S f G A J E M p M V k x F I I N p U V j E l u I t f X i b t e s 0 9 r 9 X v L q o N p 6 i j j I 7 R C T p D L r p E D X S D m q i F C H p E z + g V v V l P 1 o v 1 b n 3 M R 0 t W s X O I / s D 6 / A G h b p Z v < / l a t e x i t > Linear < l a t e x i t s h a 1 _ b a s e 6 4 = " J u U F x 7 3 b V K g A f a + d d 6 h w z A I L 8 i o = " > A A A B / 3 i c b V D L S g M x F M 3 U V 6 2 v U c G N m 2 A R X J W Z K u i y 4 M a F i w r 2 A Z 1 h u J O m b W g y M y Q Z o Y x d + C t u X C j i 1 t 9 w 5 9 + Y a W e h r Q c C h 3 P u T U 5 O m H C m t O N 8 W 6 W V 1 b X 1 j f J m Z W t 7 Z 3 f P 3 j 9 o q z i V h L Z I z G P Z D U F R z i L a 0 k x z 2 k 0 k B R F y 2 g n H 1 7 n f e a B S s T i 6 1 5 O E + g K G E R s w A t p I g X 3 k C d A j K b J b c w H I a e A N Q Q g I 7 K p T c 2 b A y 8 Q t S B U V a A b 2 l 9 e P S S p o p A k H p X q u k 2 g / A 6 k Z 4 X R a 8 V J F E y B j G N K e o R E I q v x s l n + K T 4 3 S x 4 N Y m h N p P F N / b 2 Q g l J q I 0 E z m a d W i l 4 v / e b 1 U D 6 7 8 j E V J q m l E 5 g 8 N U o 5 1 j P M y c J 9 J S j S f G A J E M p M V k x F I I N p U V j E l u I t f X i b t e s 0 9 r 9 X v L q o N p 6 i j j I 7 R C T p D L r p E D X S D m q i F C H p E z + g V v V l P 1 o v 1 b n 3 M R 0 t W s X O I / s D 6 / A G h b p Z v < / l a t e x i t > Conv < l a t e x i t s h a 1 _ b a s e 6 4 = " m o 8 R u D z V S H 6 e E s M H V 5 w 2 8 p j D V M U = " > A A A B / H i c b V D L S s N A F J 3 U V 6 2 v a J d u g k V w V Z I q 6 L L Q j c s K 9 g F N C J P p p B 0 6 j z A z K Y R Q f 8 W N C 0 X c + i H u / B s n b R b a e m D g c M 6 9 3 D M n S i h R 2 n W / r c r W 9 s 7 u X n W / d n B 4 d H x i n 5 7 1 l U g l w j 0 k q J D D C C p M C c c 9 T T T F w 0 R i y C K K B 9 G s U / i D O Z a K C P 6 o s w Q H D E 4 4 i Q m C 2 k i h X f c Z 1 F P J 8 o 7 g 8 0 X o R 1 j D 0 G 6 4 T X c J Z 5 N 4 J W m A E t 3 Q / v L H A q U M c 4 0 o V G r k u Y k O c i g 1 Q R Q v a n 6 q c A L R D E 7 w y F A O G V Z B v g y / c C 6 N M n Z i I c 3 j 2 l m q v z d y y J T K W G Q m i 6 h q 3 S v E / 7 x R q u O 7 I C c 8 S T X m a H U o T q m j h V M 0 4 Y y J x E j T z B C I J D F Z H T S F E i J t + q q Z E r z 1 L 2 + S f q v p X T d b D z e N t l v W U Q X n 4 A J c A Q / c g j a 4 B 1 3 Q A w h k 4 B m 8 g j f r y X q x 3 q 2 P 1 W j F K n f q 4 A + s z x 9 S 7 Z U l < / l a t e x i t > Conv < l a t e x i t s h a 1 _ b a s e 6 4 = " m o 8 R u D z V S H 6 e E s M H V 5 w 2 8 p j D V M U = " > A A A B / H i c b V D L S s N A F J 3 U V 6 2 v a J d u g k V w V Z I q 6 L L Q j c s K 9 g F N C J P p p B 0 6 j z A z K Y R Q f 8 W N C 0 X c + i H u / B s n b R b a e m D g c M 6 9 3 D M n S i h R 2 n W / r c r W 9 s 7 u X n W / d n B 4 d H x i n 5 7 1 l U g l w j 0 k q J D D C C p M C c c 9 T T T F w 0 R i y C K K B 9 G s U / i D O Z a K C P 6 o s w Q H D E 4 4 i Q m C 2 k i h X f c Z 1 F P J 8 o 7 g 8 0 X o R 1 j D 0 G 6 4 T X c J Z 5 N 4 J W m A E t 3 Q / v L H A q U M c 4 0 o V G r k u Y k O c i g 1 Q R Q v a n 6 q c A L R D E 7 w y F A O G V Z B v g y / c C 6 N M n Z i I c 3 j 2 l m q v z d y y J T K W G Q m i 6 h q 3 S v E / 7 x R q u O 7 I C c 8 S T X m a H U o T q m j h V M 0 4 Y y J x E j T z B C I J D F Z H T S F E i J t + q q Z E r z 1 L 2 + S f q v p X T d b D z e N t l v W U Q X n 4 A J c A Q / c g j a 4 B 1 3 Q A w h k 4 B m 8 g j f r y X q x 3 q 2 P 1 W j F K n f q 4 A + s z x 9 S 7 Z U l < / l a t e x i t > Convvis < l a t e x i t s h a 1 _ b a s e 6 4 = " K W C u G L C U k F x r X 9 m L g f 6 + 9 l H Z h m 0 = " > A A A C A 3 i c b V D L S s N A F J 3 U V 6 2 v q D v d B I v g q i R V 0 G W h G 5 c V 7 A P a E C b T S T t 0 H m F m U i g h 4 M Z f c e N C E b f + h D v / x k k b Q V s P D J w 5 5 1 7 u v S e M K V H a d b + s 0 t r 6 x u Z W e b u y s 7 u 3 f 2 A f H n W U S C T C b S S o k L 0 Q K k w J x 2 1 N N M W 9 W G L I Q o q 7 4 a S Z + 9 0 p l o o I f q 9 n M f Y Z H H E S E Q S 1 k Q L 7 Z M C g H k u W N g W f Z s H P b 0 p U F t h V t + b O 4 a w S r y B V U K A V 2 J + D o U A J w 1 w j C p X q e 2 6 s / R R K T R D F W W W Q K B x D N I E j 3 D e U Q 4 a V n 8 5 v y J x z o w y d S E j z u H b m 6 u + O F D K l Z i w 0 l f m O a t n L x f + 8 f q K j G z 8 l P E 4 0 5 m g x K E q o o 4 W T B + I M i c R I 0 5 k h E E l i d n X Q G E q I t I m t Y k L w l k 9 e J Z 1 6 z b u s 1 e + u q g 2 3 i K M M T s E Z u A A e u A Y N c A t a o A 0 Q e A B P 4 A W 8 W o / W s / V m v S 9 K S 1 b R c w z + w P r 4 B n c / m K I = < / l a t e x i t > Convvis < l a t e x i t s h a 1 _ b a s e 6 4 = " K W C u G L C U k F x r X 9 m L g f 6 + 9 l H Z h m 0 = " > A A A C A 3 i c b V D L S s N A F J 3 U V 6 2 v q D v d B I v g q i R V 0 G W h G 5 c V 7 A P a E C b T S T t 0 H m F m U i g h 4 M Z f c e N C E b f + h D v / x k k b Q V s P D J w 5 5 1 7 u v S e M K V H a d b + s 0 t r 6 x u Z W e b u y s 7 u 3 f 2 A f H n W U S C T C b S S o k L 0 Q K k w J x 2 1 N N M W 9 W G L I Q o q 7 4 a S Z + 9 0 p l o o I f q 9 n M f Y Z H H E S E Q S 1 k Q L 7 Z M C g H k u W N g W f Z s H P b 0 p U F t h V t + b O 4 a w S r y B V U K A V 2 J + D o U A J w 1 w j C p X q e 2 6 s / R R K T R D F W W W Q K B x D N I E j 3 D e U Q 4 a V n 8 5 v y J x z o w y d S E j z u H b m 6 u + O F D K l Z i w 0 l f m O a t n L x f + 8 f q K j G z 8 l P E 4 0 5 m g x K E q o o 4 W T B + I M i c R I 0 5 k h E E l i d n X Q G E q I t I m t Y k L w l k 9 e J Z 1 6 z b u s 1 e + u q g 2 3 i K M M T s E Z u A A e u A Y N c A t a o A 0 Q e A B P 4 A W 8 W o / W s / V m v S 9 K S 1 b R c w z + w P r 4 B n c / m K I = < /

l a t e x i t > + ⇤ < l a t e x i t s h a 1 _ b a s e 6 4 = " c 3 W u z S b i d r m R 8 d 8 z m k N y n g B B Q c o = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B Z B P J S k C n o s e P H Y g q 2 F N p T N d t K u 3 W z C 7 k Y o o b / A i w d F v P q T v P l v 3 L Y 5 a O u D g c d 7 M 8 z M C x L B t X H d b 6 e w t r 6 x u V X c L u 3 s 7 u 0 f l A + P 2 j p O F c M W i 0 W s O g H V K L j E l u F G Y C d R S K N A 4 E M w v p 3 5 D 0 + o N I / l v Z k k 6 E d 0 K H n I G T V W a l 7 0 y x W 3 6 s 5 B V o m X k w r k a P T L X 7 1 B z N I I p W G C a t 3 1 3 M T 4 G V W G M 4 H T U i / V m F A 2 p k P s W i p p h N r P 5 o d O y Z l V B i S M l S 1 p y F z 9 P Z H R S O t J F N j O i J q R X v Z m 4 n 9 e N z X h j Z 9 x m a Q G J V s s C l N B T E x m X 5 M B V 8 i M m F h C m e L 2 V s J G V F F m b D Y l G 4 K 3 / P I q a d e q 3 m W 1 1 r y q 1 N 0 8 j i K c w C m c g w f X U I c 7 a E A L G C A 8 w y u 8 O Y / O i / P u f C x a C 0 4 + c w x / 4 H z + A G v l j K A = < / l a t e x i t > ⇤ < l a t e x i t s h a 1 _ b a s e 6 4 = " c 3 W u z S b i d r m R 8 d 8 z m k N y n g B B Q c o = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B Z B P J S k C n o s e P H Y g q 2 F N p T N d t K u 3 W z C 7 k Y o o b / A i w d F v P q T v P l v 3 L Y 5 a O u D g c d 7 M 8 z M C x L B t X H d b 6 e w t r 6 x u V X c L u 3 s 7 u 0 f l A + P 2 j p O F c M W i 0 W s O g H V K L j E l u F G Y C d R S K N A 4 E M w v p 3 5 D 0 + o N I / l v Z k k 6 E d 0 K H n I G T V W a l 7 0 y x W 3 6 s 5 B V o m X k w r k a P T L X 7 1 B z N I I p W G C a t 3 1 3 M T 4 G V W G M 4 H T U i / V m F A 2 p k P s W i p p h N r P 5 o d O y Z l V B i S M l S 1 p y F z 9 P Z H R S O t J F N j O i J q R X v Z m 4 n 9 e N z X h j Z 9 x m a Q G J V s s C l N B T E x m X 5 M B V 8 i M m F h C m e L 2 V s J G V F F m b D Y l G 4 K 3 / P I q a d e q 3 m W 1 1 r y q 1 N 0 8 j i K c w C m c g w f X U I c 7 a E A L G C A 8 w y u 8 O Y / O i / P u f C x a C 0 4 + c w x / 4 H z + A G v l j K A = < / l a t e x i t > + 1 ReLU + 1 ⇤ < l a t e x i t s h a 1 _ b a s e 6 4 = " c 3 W u z S b i d r m R 8 d 8 z m k N y n g B B Q c o = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B Z B P J S k C n o s e P H Y g q 2 F N p T

N d t K u 3 W z C 7 k Y o o b / A i w d F v P q T v P l v 3 L Y 5 a O u D g c d 7 M 8 z M C x L B t X H d b 6 e w t r 6 x u V X c L u 3 s 7 u 0 f l A + P 2 j p O F c M W i 0 W s O g H V K L j E l u F G Y C d R S K N A 4 E M w v p 3 5 D 0 + o N I / l v Z k k 6 E d 0 K H n I G T V W a l 7 0 y x W 3 6 s 5 B V o m X k w r k a P T L X 7 1 B z N I I p W G C a t 3 1 3 M T 4 G V W G M 4 H T U i / V m F A 2 p k P s W i p p h N r P 5 o d O y Z l V B i S M l S 1 p y F z 9 P Z H R S O t J F N j O i J q R X v Z m 4 n 9 e N z X h j Z 9 x m a Q G J V s s C l N B T E x m X 5 M B V 8 i M m F h C m e L 2 V s J G V F F m b D Y

l G 4 K 3 / P I q a d e q 3 m W 1 1 r y q 1 N 0 8 j i K c w C m c g w f X U I c 7 a E A L G C A 8 w y u 8 O Y / O i / P u f C x a C 0 4 + c w x / 4 H z + A G v l j K A = < / l a t e x i t > ⇤ < l a t e x i t s h a 1 _ b a s e 6 4 = " c 3 W u z S b i d r m R 8 d 8 z m k N y n g B B Q c o = " > A A A B 6 H i c b V B N S 8 N A E J 3 U r 1 q / q h 6 9 L B Z B P J S k C n o s e P H Y g q 2 F N p T

N d t K u 3 W z C 7 k Y o o b / A i w d F v P q T v P l v 3 L Y 5 a O u D g c d 7 M 8 z M C x L B t X H d b 6 e w t r 6 x u V X c L u 3 s 7 u 0 f l A + P 2 j p O F c M W i 0 W s O g H V K L j E l u F G Y C d R S K N A 4 E M w v p 3 5 D 0 + o N I / l v Z k k 6 E d 0 K H n I G T V W a l 7 0 y x W 3 6 s 5 B V o m X k w r k a P T L X 7 1 B z N I I p W G C a t 3 1 3 M T 4 G V W G M 4 H T U i / V m F A 2 p k P s W i p p h N r P 5 o d O y Z l V B i S M l S 1 p y F z 9 P Z H R S O t J F N j O i J q R X v Z m 4 n 9 e N z X h j Z 9 x m a Q G J V s s C l N B T E x m X 5 M B V 8 i M m F h C m e L 2 V s J G V F F m b D Y

l G 4 K 3 / P I q a d e q 3 m W 1 1 r y q 1 N 0 8 j i K c w C m c g w f X U I c 7 a E A L G C A 8 w y u 8 O Y / O i / P u f C x a C 0 4 + c w x / 4 H z + A G v l j K A = < / l a t e x i t > Feature-wise linear modulation (FiLM), which modulates visual inputs using representations of textual instructions, is an effective method for image captioning (Perez et al., 2018) and instruction following (Bahdanau et al., 2019) .

In RTFM, the agent must not only filter concepts in the visual domain using language but filter concepts in the text domain using visual observations. To support this, FiLM 2 builds codependent representations of text and visual inputs by further incorporating conditional representations of the text given visual observations. Figure 2 shows the FiLM 2 layer.

We use upper-case bold letters to denote tensors, lower-case bold letters for vectors, and non-bold letters for scalars. Exact dimensions of these variables are shown in Let + and * symbols denote element-wise addition and multiplication operations that broadcast over spatial dimensions. We first modulate visual features using text features:

EQUATION

Unlike FiLM, we additionally modulate text features using visual features: The output of the FiLM 2 layer consists of the sum of the modulated features V , as well as a max-pooled summary s over this sum across spatial dimensions.

EQUATION

V = V vis + V text (7) s = MaxPool(V ) (8) 4.2 THE T X T 2π MODEL

We model interactions between observations from the environment, goal, and document using FiLM 2 layers. We first encode text inputs using bidirectional LSTMs, then compute summaries using self-attention and conditional summaries using attention. We concatenate text summaries into text features, which, along with visual features, are processed through consecutive FiLM 2 layers. In this case of a textual environment, we consider the grid of word embeddings as the visual features for FiLM 2 . The final FiLM 2 output is further processed by MLPs to compute a policy distribution over actions and a baseline for advantage estimation. Figure 3 shows the txt2π model.

Let E obs denote word embeddings corresponding to the observations from the environment, where E obs [:, :, i, j] represents the embeddings corresponding to the l obs -word string that describes the objects in location (i, j) in the grid-world. Let E doc , E inv , and E goal respectively denote the embeddings corresponding to the l doc -word document, the l inv -word inventory, and the l goal -word goal.

We first compute a fixed-length summary c goal of the the goal using a bidirectional LSTM (Hochreiter & Schmidhuber, 1997) followed by self-attention (Lee et al., 2017; Zhong et al., 2018) .

EQUATION

EQUATION

We abbreviate self-attention over the goal as c goal = selfattn(H goal ). We similarly compute a summary of the inventory as c inv = selfattn(BiLSTM inv (E inv )). Next, we represent the document encoding conditioned on the goal using dot-product attention (Luong et al., 2015) .

EQUATION

EQUATION

We abbreviate attention over the document encoding conditioned on the goal summary as c doc = attend(H doc , c goal ). Next, we build the joint representation of the inputs using successive FiLM 2 layers. At each layer, the visual input to the FiLM 2 layer is the concatenation of the output of the previous layer with positional features. For each cell, the positional feature X pos consists of the x and y distance from the cell to the agent's position respectively, normalized by the width and height of the grid-world. The text input is the concatenation of the goal summary, the inventory summary, the attention over the document given the goal, and the attention over the document given the previous visual summary. Let [a; b] 

EQUATION

EQUATION

EQUATION

BiLSTM vis-doc (E doc ) is another encoding of the document similar to H goal , produced using a separate LSTM, such that the document is encoded differently for attention with the visual features and with the goal. For i = 0, we concatenate the bag-of-words embeddings of the grid with positional features as the initial visual features V (0) = [ j E obs,j ; X pos ]. We max pool a linear transform of the initial visual features to compute the initial visual summary s (0) = MaxPool(W ini V (0) + b ini ).

Let s (last) denote visual summary of the last FiLM 2 layer. We compute the policy y policy and baseline y baseline as

EQUATION

where MLP policy and MLP baseline are 2-layer multi-layer perceptrons with ReLU activation. We train using TorchBeast (Küttler et al., 2019) , an implementation of IMPALA (Espeholt et al., 2018) . Please refer to appendix D for details.

5. EXPERIMENTS:

We consider variants of RTFM by varying the size of the grid-world (6 × 6 vs 10 × 10), allowing many-to-one group assignments to make disambiguation more difficult (group), allowing dynamic, moving monsters that hunt down the player (dyna), and using natural language templated documents (nl). In the absence of many-to-one assignments, the agent does not need to perform steps 3 and 5 in section 3 as there is no need to disambiguate among many assignees, making it easier to identify relevant information.

We compare txt2π to the FiLM model by Bahdanau et al. (2019) and a language-conditioned residual CNN model. We train on one set of dynamics (e.g. group assignments of monsters and modifiers) and evaluated on a held-out set of dynamics. We also study three variants of txt2π.

In no task attn, the document attention conditioned on the goal utterance (equation 16) is removed and the goal instead represented through self-attention and concatenated with the rest of the text features. In no vis attn, we do not attend over the document given the visual output of the previous layer (equation 18), and the document is instead represented through self-attention. In no text mod, text modulation using visual features (equation 6) is removed. Please see appendix C for model details on our model and baselines, and appendix D for training details.

5.1. COMPARISON TO BASELINES AND ABLATIONS:

We compare txt2π to baselines and ablated variants on a simplified variant of RTFM in which there are one-to-one group assignments (no group), stationary monsters (no dyna), and no natural language templated descriptions (no nl). Figure 4 shows that compared to baselines and ablated variants, txt2π is more sample efficient and converges to higher performance. Moreover, no ablated variant is able to solve the tasks-it is the combination of ablated features that enables txt2π to win consistently. Qualitatively, the ablated variants converge to locally optimum policies in which the agent often picks up a random item and then attacks the correct monster, resulting in a ∼ 50% win rate. Table 1 shows that all models, with the exception of the CNN baseline, generalise to new evaluation environments with dynamics and world configurations not seen during training, with txt2π outperforming FiLM and the CNN model.

We find similar results for txt2π, its ablated variants, and baselines on a separate, language-based rock-paper-scissors task in which the agent needs to deduce cyclic dependencies (which type beats which other type) through reading in order to acquire the correct item and defeat a monster. We observe that the performance of reading models transfer from training environments to new environments with unseen types and unseen dependencies. Compared to ablated variants and baselines, txt2π is more sample efficient and achieves higher performance on both training and new environment dynamics. When transferring to new environments, txt2π remains more sample efficient than the other models. Details on these experiments are found in appendix E. Due to the long sequence of co-references the agent must perform in order to solve the full RTFM (10 × 10 with moving monsters, many-to-one group assignments, and natural language templated documents) we design a curriculum to facilitate policy learning by starting with simpler variants of RTFM.

5.2. CURRICULUM LEARNING FOR COMPLEX ENVIRONMENTS:

We start with the simplest variant (no group, no dyna, no nl) and then add in an additional dimension of complexity. We repeatedly add more complexity until we obtain 10×10 worlds with moving monsters, many-to-one group assignments and natural language templated descriptions. The performance across the curriculum is shown in Weights are normalised across words (e.g. horizontally). Darker means higher attention weight.

(see Figure 13 in appendix F for training curves of each stage). We see that curriculum learning is crucial to making progress on RTFM, and that initial policy training (first row of Table 2 ) with additional complexities in any of the dimensions result in significantly worse performance. We take each of the 5 runs after training through the whole curriculum and evaluate them on dynamics not seen during training. Table 3 shows variants of the last stage of the curriculum in which the model was trained on 6 × 6 versions of the full RTFM and in which the model was trained on 10 × 10 versions of the full RTFM. We see that models trained on smaller worlds generalise to bigger worlds. Despite curriculum learning, however, performance of the final model trail that of human players, who can consistently solve RTFM. This highlights the difficulties of the RTFM problem and suggests that there is significant room for improvement in developing better language grounded policy learners.

Attention maps. Figure 5 shows attention conditioned on the goal and on observation summaries produced by intermediate FiLM 2 layers. Goal-conditioned attention consistently locates the clause that contains the team the agent is supposed to attack. Intermediate layer attentions focus on regions near modifiers and monsters, particularly those that are present in the observations. These results suggests that attention mechanisms in txt2π help identify relevant information in the document.

Analysis of trajectories and failure modes. We examine trajectories from well-performing policies (80% win rate) as well as poorly-performing policies (50% win rate) on the full RTFM. We find that well-performing policies exhibit a number of consistent behaviours such as identifying the correct item to pick up to fight the target monster, avoiding distractors, and engaging target monsters after acquiring the correct item. In contrast, the poorly-performing policies occasionally pick up the wrong item, causing the agent to lose when engaging with a monster. In addition, it occasionally gets stuck in evading monsters indefinitely, causing the agent to lose when the time runs out. Replays of both policies can be found in GIFs in the supplementary materials1 .

6. CONCLUSION:

We proposed RTFM, a grounded policy learning problem in which the agent must jointly reason over a language goal, relevant dynamics specified in a document, and environment observations. In order to study RTFM, we procedurally generated a combinatorially large number of environment dynamics such that the model cannot memorise a set of environment dynamics and must instead generalise via reading. We proposed txt2π, a model that captures three-way interactions between the goal, document, and observations, and that generalises to new environments with dynamics not seen during training. txt2π outperforms baselines such as FiLM and language-conditioned CNNs. Through curriculum learning, txt2π performs well on complex RTFM tasks that require several reasoning and coreference steps with natural language templated goals and descriptions of the dynamics. Our work suggests that language understanding via reading is a promising way to learn policies that generalise to new environments. Despite curriculum learning, our best models trail performance of human players, suggesting that there is ample room for improvement in grounded policy learning on complex RTFM problems. In addition to jointly learning policies based on external documentation and language goals, we are interested in exploring how to use supporting evidence in external documentation to reason about plans (Andreas et al., 2018) and induce hierarchical policies (Hu et al., 2019; Jiang et al., 2019) .

A PLAYTHROUGH EXAMPLES:

These figures shows key snapshots from a trained policy on randomly sampled environments. does not yet have "soldier's knife", which beats the target. In 11 and 13, the agent obtains "soldier's knife" while evading monsters. In 14, the agent defeats the target and wins. Like txt2π, the CNN baseline consists of 5 layers of convolutions with channels of 16, 32, 64, 64, and 64. There are residual connections from the 3rd layer to the 5th layer. The input to each layer consists of the output of the previous layer, concatenated with positional features.

The input to the network is the concatenation of the observations V (0) and text representations. The text representations consist of self-attention over bidirectional LSTM-encoded goal, document, and inventory. These attention outputs are replicated over the dimensions of the grid and concatenated feature-wise with the observation embeddings in each cell. Figure 8 illustrates the CNN baseline.

C.3 FILM BASELINE:

The FiLM baseline encodes text in the same fashion as the CNN model. However, instead of using convolutional layers, each layer is a FiLM layer from Bahdanau et al. (2019) . Note that in our case, the language representation is a self-attention over the LSTM states instead of a concatenation of terminal LSTM states. In addition to the main RTFM tasks, we also study a simpler formulation called Rock-paper-scissors that has a fixed goal. In Rock-paper-scissors, the agent must interpret a document that describes the environment dynamics in order to solve the task. Given an set of characters (e.g. a-z), we sample 3 characters and set up a rock-paper-scissors-like dependency graph between the characters (e.g. "a beats b, b beats c, c beats a"). We then spawn a monster in the world with a randomly assigned type (e.g. "b goblin"), as well as an item corresponding to each type (e.g. "a", "b", and "c"). The attributes of the agent, monster, and items are set up such that the player must obtain the correct item and then engage the monster in order to win. Any other sequence of actions (e.g. engaging the monster without the correct weapon) results in a loss. The winning policy should then be to first identify the type of monster present, then cross-reference the document to find which item defeats that type, then pick up the item, and finally engage the monster in combat. Figure 9 shows an instance of Rock-paper-scissors.

Scenario:

Reading models generalise to new environments. We split environment dynamics by permuting 3-character dependency graphs from an alphabet, which we randomly split into training and held-out sets. This corresponds to the "permutations" setting in Table 5 . We train models on the 10 × 10 worlds from the training set and evaluate them on both seen and not seen during training. The left of Figure 10 shows the performance of models on worlds of varying sizes with training environment dynamics. In this case, the dynamics (e.g. dependency graphs) were seen during training. For 9 × 9 and 11 × 11 worlds, the world configuration not seen during training.

For 10 × 10 worlds, there is a 5% chance that the initial frame was seen during training.2 Figure 10 shows the performance on held-out environments not seen during training. We see that all models generalise to environments not seen during training, both when the world configuration is not seen (left) and when the environment dynamics are not seen (right).

Reading models generalise to new concepts.

In addition to splitting via permutations, we devise two additional ways of splitting environment dynamics by introducing new edges and nodes into the held-out set. Table 5 shows the three different settings. For each, we study the transfer behaviour of models on new environments. Figure 11 shows the learning curve when training a model on the held-out environments directly and when transferring the model trained on train environments to held-out environments. We observe that all models are significantly more sample-efficient when transferring from training environments, despite the introduction of new edges and new nodes.

txt2π is more sample-efficient and learns better policies. In Figure 10 , we see that the FiLM model outperforms the CNN model on both training environment dynamics and held-out environment dynamics. txt2π further outperforms FiLM, and does so more consistently in that the final performance has less variance. This behaviour is also observed in the in Figure 11 . When training on the held-out set without transferring, txt2π is more sample efficient than FiLM and the CNN model, and achieves higher win-rate. When transferring to the held-out set, txt2π remains more sample efficient than the other models. 

H LANGUAGE TEMPLATES:

We collect human-written natural language templates for the goal and the dynamics. The goal statements in RTFM describe which team the agent should defeat. We collect 12 language templates for goal statements. The document of environment dynamics consists of two types of statements. The first type describes which monsters are assigned to with team. The second type describes which modifiers, which describe items, are effective against which element types, which are associated with monsters. We collection 10 language templates for each type of statements. The entire document is composed from statements, which are randomly shuffled. We randomly sample a template for each statement, which we fill with the monsters and team for the first type and modifiers and element for the second type.

:

Trajectories by txt2π on RTFM can be found at https://gofile.io/?c=9k7ZLk

There are 24360 unique grid configurations given a particular dependency graph, 4060 unique dependency graphs in the training set, and 50 million frames seen during training. After training, the model finishes an episode in approximately 10 frames. Hence the probability of seeing a redundant initial frame is 5e7/10 24360 * 4060 = 5%.

Figures and Tables:

Figure1: RTFM requires jointly reasoning over the goal, a document describing environment dynamics, and environment observations. This figure shows key snapshots from a trained policy on one randomly sampled environment. Frame 1 shows the initial world. In 4, the agent approaches "fanatical sword", which beats the target "fire goblin". In 5, the agent acquires the sword. In 10, the agent evades the distractor "poison bat" while chasing the target. In 11, the agent engages the target and defeats it, thereby winning the episode. Sprites are used for visualisation -the agent observes cell content in text (shown in white). More examples are in appendix A.

Figure 2: The FiLM 2 layer.

Figure 3: txt2π models interactions between the goal, document, and observations.

Figure 5: txt2π attention on the full RTFM. These include the document attention conditioned on the goal (top) as well as those conditioned on summaries produced by intermediate FiLM 2 layers. Weights are normalised across words (e.g. horizontally). Darker means higher attention weight.

You should use arcane and mysterious items to beat lightning monsters. Panther, warg, and wolf are from the Order of the Forest. Blessed and Grandmaster's items beat poison monsters. Cold is not good against fanatical and shimmering weapons. The Star Alliance team is made up of beetle, jackal, and shaman. Imp, jaguar, and lynx make up the Rebel Enclave. Gleaming and Soldier's weapons beat fire monsters.

Figure6: The initial world is shown in 1. In 4, the agent avoids the target "lightning shaman" because it does not yet have "arcane spear", which beats the target. In 7 and 8, the agent is cornered by monsters. In 9, the agent is forced to engage in combat and loses.

Figure 8: The convolutional network baseline. The FiLM baseline has the same structure, but with convolutional layers replaced by FiLM layers.

Figure 10: Performance on the Rock-paper-scissors task across models. Left shows final performance on environments whose goals and dynamics were seen during training. Right shows performance on the environments whose goals and dynamics were not seen during training.

Figure 11: Learning curve while transferring to the development environments. Win rates of individual runs are shown in light colours. Average win rates are shown in bold, dark lines.

Table 4 in appendix B. Let x text denote a fixed-length d text -dimensional representation of the text and X vis the representation of visual inputs with height H, width W , and d vis channels. Let Conv denote a convolution layer.

Table content: <table/>

Curriculum training results. We keep 5 randomly initialised models through the entire curriculum. A cell in row i and column j shows transfer from the best-performing setting in the previous stage (bold in row i -1) to the new setting in column j. Each cell shows final mean and standard deviation of win rate on the training environments. Each experiment trains for 50 million frames, except for the initial stage (first row, 100 million instead). For the last stage (row 4), we also transfer to a 10 × 10 + dyna + group + nl variant and obtain 61 ± 18 win rate.

Table content: <table><tr><td>Transfer</td><td>Transfer to</td><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>from</td><td>6 × 6</td><td>6 × 6 dyna</td><td>6 × 6 groups</td><td>6 × 6 nl</td><td>6 × 6 dyna groups</td><td>6 × 6 group nl</td><td>6 × 6 dyna nl</td><td>6 × 6 dyna group nl</td></tr><tr><td>random +6 × 6 +dyna +group</td><td>84 ± 20</td><td colspan="2">26 ± 7 85 ± 9 82 ± 19 25 ± 3</td><td colspan="2">45 ± 6 78 ± 24 64 ± 12 23 ± 2 77 ± 10</td><td colspan="3">25 ± 3 52 ± 13 53 ± 18 40 ± 8 23 ± 2 23 ± 2 65 ± 16 43 ± 4 65 ± 17</td></tr></table>

Win rate when evaluating on new dynamics and world configurations for txt2π on the full RTFM problem.

Table content: <table><tr><td>Train</td><td>Eval</td><td>Win rate</td><td/></tr><tr><td>env</td><td>env</td><td>Train</td><td>Eval</td></tr><tr><td>6 × 6 10 × 10</td><td>6 × 6 10 × 10 10 × 10</td><td>65 ± 17 61 ± 18</td><td>55 ± 22 55 ± 27 43 ± 13</td></tr></table>

The entities present are soldier's axe, shimmering axe, fire shaman, and poison wolf.

Table content: <table><tr><td>gl</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>a0</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>a1</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>a2</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>a3</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>a4</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>you</td><td>should</td><td>use</td><td>grandmasters</td><td>,</td><td>shimmering</td><td colspan="2">items</td><td colspan="2">to</td><td colspan="2">beat</td><td colspan="2">fire</td><td colspan="2">monsters</td><td colspan="2">.</td><td colspan="2">the</td><td colspan="2">shaman</td><td colspan="2">,</td><td colspan="2">zombie</td><td colspan="2">,</td><td colspan="2">ghost</td><td>monsters</td><td>are</td><td>rebel</td><td>enclave</td><td>.</td><td>poison</td><td>is</td><td>weak</td><td>against</td><td>fanatical</td><td>,</td><td>gleaming</td><td>.</td><td colspan="2">lightning</td><td colspan="2">is</td><td colspan="2">defeated</td><td colspan="2">by</td><td colspan="2">blessed</td><td colspan="2">,</td><td colspan="2">mysterious</td><td colspan="2">.</td><td colspan="2">wolf</td><td colspan="2">,</td><td colspan="2">bat</td><td>,</td><td>panther</td><td>are</td><td>order</td><td>of</td><td>the</td><td>forest</td><td>.</td><td>cold</td><td>is</td><td>not</td><td>good</td><td>against</td><td>arcane</td><td>,</td><td>soldiers</td><td>.</td><td>star</td><td>alliance</td><td>team</td><td>is</td><td>made</td><td>up</td><td>of</td><td>goblin</td><td>,</td><td>imp</td><td>,</td><td>jaguar</td><td>.</td><td>pad</td></tr><tr><td colspan="74">(a) The entities present are shimmering morning star, mysterious spear, fire jaguar, and lightning ghost.</td></tr><tr><td>gl</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>a0</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>a1</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>a2</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>a3</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>a4</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>lightning</td><td>is</td><td>defeated</td><td>by</td><td>blessed</td><td colspan="2">,</td><td colspan="2">gleaming</td><td colspan="2">.</td><td colspan="2">poison</td><td colspan="2">is</td><td colspan="2">defeated</td><td colspan="2">by</td><td colspan="2">grandmasters</td><td colspan="2">,</td><td colspan="2">soldiers</td><td colspan="2">.</td><td colspan="2">cold</td><td colspan="2">is</td><td>weak</td><td>against</td><td>arcane</td><td>,</td><td>mysterious</td><td>.</td><td>the</td><td>zombie</td><td>,</td><td>ghost</td><td>,</td><td colspan="2">wolf</td><td colspan="2">monsters</td><td colspan="2">are</td><td colspan="2">star</td><td colspan="2">alliance</td><td colspan="2">.</td><td colspan="2">shaman</td><td colspan="2">,</td><td colspan="2">imp</td><td colspan="2">,</td><td colspan="2">bat</td><td colspan="2">are</td><td>on</td><td>the</td><td>rebel</td><td>enclave</td><td>team</td><td>.</td><td>the</td><td>goblin</td><td>,</td><td>jaguar</td><td>,</td><td>panther</td><td>monsters</td><td>are</td><td>order</td><td>of</td><td>the</td><td>forest</td><td>.</td><td>fire</td><td>is</td><td>not</td><td>good</td><td>against</td><td>fanatical</td><td>,</td><td>shimmering</td><td>.</td><td>pad</td></tr><tr><td/><td/><td/><td/><td/><td colspan="5">(b)</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr></table>

B VARIABLE DIMENSIONSLet x text ∈ R dtext denote a fixed-length d text -dimensional representation of the text and X vis ∈ R dvis×H×W denote the representation of visual inputs with The txt2π used in our experiments consists of 5 consecutive FiLM 2 layers, each with 3x3 convolutions and padding and stride sizes of 1. The txt2π layers have channels of 16, 32, 64, 64, and 64, with residual connections from the 3rd layer to the 5th layer. The Goal-doc LSTM (see Figure3) shares weight with the Goal LSTM. The Inventory and Goal LSTMs have a hidden dimension of size 10, whereas the Vis-doc LSTM has a dimension of 100. We use a word embedding dimension of 30.

Table content: <table><tr><td colspan="3">C MODEL DETAILS</td><td/></tr><tr><td>C.1 T X T 2π</td><td/><td/><td/></tr><tr><td colspan="4">Variable d text -dim text representation d vis -dim visual representation with height H, width W , d vis channels Environment observations embeddings Hyperparameters. C.2 CNN WITH RESIDUAL CONNECTIONS l obs -word string that describes the objects in location (i, j) in the grid-world l doc -word document embeddings Inventory BiLSTM Inventory Positional features cat Selfattn l inv -word inventory embeddings l goal -word goal embeddings Conv BiLSTM Goal Selfattn Goal cat cat</td><td>Symbol x text X vis E obs E obs [:, :, i, j] l obs × d emb Dimension d text d vis × H × W l obs × d emb × H × W E doc Policy MLP policy l doc × d emb E inv l inv × d emb E goal l goal × d emb Conv Conv Linear cat …</td></tr><tr><td>Document</td><td>Doc BiLSTM</td><td>Selfattn</td><td colspan="2">Table 4: Variable dimensions</td></tr><tr><td>Visual features</td><td/><td/><td/><td>Baseline MLP</td><td>baseline</td></tr></table>

Statistics of the three variations of the Rock-paper-scissors task

Table content: <table><tr><td/><td/><td/><td/><td># graphs</td><td/><td/><td># edges</td><td/><td/><td># nodes</td></tr><tr><td/><td/><td/><td colspan="8">train dev unseen train dev % new train dev % new</td></tr><tr><td colspan="3">permutation</td><td>30</td><td>30</td><td>y</td><td>20</td><td>20</td><td>n</td><td>60</td><td>60</td><td>n</td></tr><tr><td colspan="2">new edge</td><td/><td>20</td><td>20</td><td>y</td><td>48</td><td>36</td><td>y</td><td>17</td><td>13</td><td>n</td></tr><tr><td colspan="3">new edge+nodes</td><td>60</td><td>60</td><td>y</td><td>20</td><td>20</td><td>y</td><td>5</td><td>5</td><td>y</td></tr><tr><td/><td>1.0</td><td colspan="3">seen environment dynamics</td><td/><td/><td colspan="4">new environment dynamics</td></tr><tr><td/><td>0.8</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>win rate</td><td>0.4 0.6</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.0 0.2</td><td>9x9</td><td colspan="2">10x10 world size</td><td>11x11 conv film film^2</td><td/><td>9x9</td><td colspan="2">10x10 world size</td><td>11x11 conv film film^2</td></tr></table>

Figure 13: Curriculum learning results for txt2π on RTFM. Win rates of individual runs are shown in light colours. Average win rates are shown in bold, dark lines.G ENTITIES AND MODIFIERSBelow is a list of entities and modifiers contained in RTFM: Monsters: wolf, jaguar, panther, goblin, bat, imp, shaman, ghost, zombie Weapons: sword, axe, morningstar, polearm, knife, katana, cutlass, spear Elements: cold, fire, lightning, poison Modifiers: Grandmaster's, blessed, shimmering, gleaming, fanatical, mysterious, Soldier's, arcane Teams: Star Alliance, Order of the Forest, Rebel Enclave

Table content: <table><tr><td colspan="11">F CURRICULUM LEARNING TRAINING CURVES</td><td/><td/><td/></tr><tr><td/><td>1.0</td><td/><td/><td/><td>rand</td><td/><td/><td/><td/><td/><td/><td colspan="2">rand +6x6</td></tr><tr><td/><td>0.8</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>win rate</td><td>0.4 0.6</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.2</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td colspan="2">0.0 0.0 1.0</td><td>0.2</td><td colspan="3">0.4 rand +6x6 +dyna 0.6 frames</td><td>0.8</td><td>1.0 1e8</td><td>0</td><td>1</td><td colspan="3">2 rand +6x6 +dyna +groups 3 frames</td><td>4</td><td>5 1e7</td></tr><tr><td/><td>0.8</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>win rate</td><td>0.4 0.6</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.0 0.2</td><td>0</td><td>1 6 10</td><td>2</td><td>frames move 6 extra 6</td><td>3</td><td>4 nl 6 extra,nl 6</td><td>5 1e7</td><td>0</td><td colspan="2">1 move,nl 6 extra,move 6</td><td>2</td><td>3 extra,move,nl 6 frames extra,move,nl 10</td><td>4</td><td>1e7 5</td></tr></table>

Peer review (iclr2020):
First of all, I should acknowledge that this is a last-minute emergency review. Second, I should also acknowledge that I haven’t actively followed the specific research topic handled in this paper, although I work quite a lot in reinforcement learning and dialogue systems. The last paper I read on the topic was Branavan et al’s “Learning to win by reading manuals in monte-carlo framework” (ACL 2011). This paper is about language-conditioned reinforcement learning, where the agent is required to perform machine reading of documents to learn policies to solve a task in new environments. The contribution of this paper is seemingly two fold: first, a new benchmark test suite is proposed (named “read to fight monsters” funny acronym), and second, an improved neural model (named “txt2pi”) which extends FiLM proposed for related tasks. Various experiments on txt2pi are conducted to show how the model performs against baselines, as well as it behaves under different learning settings. Overall, this paper makes a good contribution to the research on deep models for language-conditioned reinforcement learning, but I see a number of things I miss in this paper. The first one is about the test suite RTFM. The authors claim that it poses a new challenge that the agent has to understand the language specification of the goal, the environment dynamics, and the observations *all together* (compared to prior work that misses one or two), I am not confident that it is not challenging enough. - The main concern I have is that they are all machine-generated, and soon or later, this kind of test suite could be beaten-to-death. A more ideal test suite would be the texts collected by humans, aiming for more diversity, ambiguity, and sometimes incorrectness. - The RTFM would combinatorily generate a large number of environments, but still, they all look pretty simple. It would be interesting to have “partially specified” documents, such as “lightning monsters are known to be weak against grandmaster’s or soldier’s weapons, but it is not specifically known”. It would be interesting to have partially observable environments, where the agent has to do some information gathering in the environment to behave optimally. The second one is about txt2pi model proposed in the paper. - txt2pi adequately extends FiLM to handle multiple sources of information, but what is the lesson being learned? I find the discussions in page 15 very enlightening. Could you move the content to the main text and defer the curriculum learning details to the appendix? In summary, this paper proposes an interesting test suite for language-conditioned reinforcement learning, but I wish it is more realistic in the language data and more complex in the RL tasks. --- I have updated my rating based on the reviewer response. Thank you for clarifying some of my concerns. I am still a bit reserved because of the use of synthetic corpus. However, at the same time, I think there could be a lot of follow-up papers based on this work.