Graph Constrained Reinforcement Learning for Natural Language Action Spaces
Authors: Prithviraj Ammanabrolu
Matthew Hausknecht
Venue:  (2020)
Keywords: natural language generation
deep reinforcement learning
knowledge graphs
interactive fiction

Abstract: Interactive Fiction games are text-based simulations in which an agent interacts with the world purely through natural language. They are ideal environments for studying how to extend reinforcement learning agents to meet the challenges of natural language understanding, partial observability, and action generation in combinatorially-large text-based action spaces. We present KG-A2C, an agent that builds a dynamic knowledge graph while exploring and generates actions using a template-based action space. We contend that the dual uses of the knowledge graph to reason about game state and to constrain natural language generation are the keys to scalable exploration of combinatorially large natural language actions. Results across a wide variety of IF games show that KG-A2C outperforms current IF agents despite the exponential increase in action space size.

Published as a conference paper at ICLR 2020
GRAPH CONSTRAINED REINFORCEMENT LEARNING FOR NATURAL LANGUAGE ACTION SPACES
Prithviraj Ammanabrolu Georgia Institute of Technology ra j. ammanabrol uê gaggatthhhh edu
Matthew Hausknecht
Microsoft Research
matthew. haus knecht wtw.e/eemcccccooooooooooooooo
ABSTRACT
Interactive Fiction games are text-based simulations which an agent interacts with the world purely through natural language. They are ideal environments for studying how to extend reinforcement learning agents to meet the challenges of natural language understanding, partial observability, and action generation in combinatorially--arge text-based action spaces. We present KG-A2d] an agent that builds a dynamic knowledge graph while exploring and generates actions us- ing template-based action space. We contend that the dual uses of the knowledge graph reason about game state and to constrain natural language generation are the keys to scalable exploration of combinatorially large natural language actions. Results across a wide variety of IF games show that KG-A2C outperforms current IF agents despite the exponential increase in action space size.
INTRODUCTION
Natural language communication has long been considered defining characteristic of human in- telligence. We are motivated by the question of how learning agents can understand and generate contextually relevant natural language in service of achieving goal. In pursuit of this objective we study Interactive Fiction (IF) games, or text-adventures: simulations in which an agent interacts with the world purely through natural descriptions and commands. To progress in these games, an agent must generate natural language actions that are coherent, contextually relevant, and able to effect the desired change in the world.
and "talking" the world using textual
Complicating the problem of generating contextually relevant language in these games is the issue of partial observability: the fact that the agent never has access the true underlying world state. IF games are structured as puzzles and often consist of an complex. interconnected web of distinct locations, objects, and characters. The agent needs to thus reason about the complexities of such a world solely through the textual descriptions that it receives, descriptions that are often incomplete. Further, an agent must be able to perform commonsense reasoning IF games assume that human players possess prior commonsense and thematic knowledge- c.g. knowing that swords can kill trolls or that trolls live in dark places. Knowledge graphs provide us with an intuitive way of rep- resenting these partially observable worlds. Prior works have shown how using knowledge graphs aid in the twin issues of partial observability (Ammanabrolu & Riedl] 2019a) and commonsense reasoning (Ammanabrolu & Ried] 2019b), but do not use them in the context of generating natural language.
To gain sense for the challenges surrounding natural language generation, we need first un- derstand how large this space really is. In order to solve solve a popular IF game such as Zork] it's necessary generate actions consisting of up to five-words from a relatively modest vocab- ulary of 697 words recognized by Zork's parser. Even this modestly sized vocabulary leads to (6979) 1.64 x 10 possible actions at every step a dauntingly-large combinatorially. sized action space for learning agent to explore. In order to reduce the size of this space while maintain- ing expressiveness, Hausknecht et al. 2019a) propose the use of template-actions in which the agent first selects template (e.g. |pu]]
then fills in the blanks using vocabulary words. There are 237 templates in Zorkl, each with up to two blanks, yielding a template-action space of size
Coode available htttpp://ggthhu/ com/ jaajaaaabroll//k--aaa

---

Published as a conference paper at ICLR 2020
((337 x 6972) 1.15 x 10 This space is six orders of magnitude smaller than the word-based space, but still six orders of magnitude larger than the action spaces used by previous text-based agents (Narasimhan et al.) 2015 Zahavy al.] 2018). We demonstrate how these templates provide the structure required to further constrain our action space via our knowledge graph-and make the argument that the combination of these approaches allows to generate meaningful natural language commands.
Our contributions are as follows: We introduce an novel agent that utilizes both knowledge graph based state space and template based action space and show how to train such an agent. We then conduct an empirical study evaluating our agent across a diverse set of IF games followed by an ablation analysis studying the effectiveness of various components of our algorithm as well as its overall generalizability. Remarkably we show that our agent achieves state-of-the on a large proportion of the games despite the exponential increase in action space size.
performance
2 RELATED WORK
We examine prior work in three broad categories: text-based game playing agents and frameworks as well as knowledge graphs used for natural language generation and game playing agents.
LSTM-DQN (Narasimhan et al. 2015), considers verb-noun actions up to two-words in length. Separate Q- Value estimates are produced for each possible verb and object, and the action consists of pairing the maximally valued verb combined with the maximally valued object. The DRRN algorithm for proice-based games (He et al. 2 Zelinka 2018) estimates Q- V-Vuless for a particular action from a particular state. Fuudd et al. (211)) use Word2Vec (Mikolov et al.] 2013) to aid in extracting affordances for items in these games and use this information to produce relevant action verbs. iZahavy et al. (2018) reduce the combinatorialll-sizze action space into a discrete form using a walkthrough of the game and introduce the Action Elimination DQN, which learns to eliminate
actions unlikely to cause a world change.
Côté et al. (2018) introduce Tex(World, framework for procedurally generating parser-based games, allowing user to control the difficulty of a generated game Yuan et al. (2019) intro- duce the concept of interactive
in the form of DAit-modelimg QA tasks in
TextWorld. Urbanek et al.) (2019) introduce Light, a dataset of crowdsourced rext-adventure game dialogs focusing on giving collaborative agents the ability to generate contextually relevant dialog and emotes. Hausknecht et al. (2019a) have open-sourced Jeriche an optimized interface for play- ing human- made IF games- formalizing this task. They further provide a comparative study of various types of agents their set of games, testing the performance of heuristic based agents such as NAIL (Hausknecht et al., 2019b) and various reinforcement learning agents are benchmarked. We use Jericho and the tools that provides to develop our agents.
Knowledge graphs have been shown to be useful representations for a variety of tasks surround- ing natural language generation and interactive fiction. Ghazvininejad et al. (017)) and Guan et al. 2018) effectively use knowledge graph representations to improve neural conversational and story ending prediction models respectively. (Ammanabrolu et al. (2019) explore procedural content gen- eration in text-adventure knowledge graphs to ground generative systems trained produce quest content. From the perspec- tive of text- gext-game playing agent and most in line with the spirit of our work, Ammanabrolu & Riedl (2019a) present the Knowledge Graph DQN or KG-DQN, an approach where knowledge graph built during exploration is used as state representation for deep reinforcement learning based agent. Ammanabrolu & Ried] (2019b) further expand on this work, exploring methods of transfer- ring control policies in rext-games using knowledge graphs to seed agent with useful common- sense knowledge and to transfer knowledge between different games within a domain. Both of these works, however, identify discrete set of actions required to play the game beforchand and so do not fully tackle the issue of the combinatorial action space.
at constructing a quest for given game world, and use
2

---

Published as a conference paper at ICLR 2020
3 STATE AND ACTION SPACES
Formally, IF games are partially observable Markov decision processes (POMDP), represented as a 7-tupic of (S, A, O, 7) representing the set of environment states, mosty deterministic conditional transition probabilities between states, the vocabulary or words used to compose text commands, observations returned by the game, observation conditional probabilities, reward func- tion, and the discount factor respectively (Côté et al.] 2018 Hausknechh et al., 2019a). To deal with the resulting twin challenges of partial observability and combinatorial actions, we use a knowledge graph based state space and template-based action space each described in detail below.
Knowledge Graph State Space. Building on Ammanabrolu & Riedl (2019a), we use a knowledge graph as a state representation that is learnt during exploration. The knowledge graph is stored as set of 3-tuples of (subject, relation, object). These triples are extracted from the observations using Stanford's Open Information Extraction (OpenIE) (Angeli et al. 2015). Human-made IF games often contain relatively complex structure information that OpenIE is not designed to parse and so we add additional rules to ensure that we are parsing the relevant information.
Updated after every action, the knowledge graph helps the agent form map of the world that it is exploring, in addition to retaining information that it has learned such as the affordances associated with object, the properties of character, current inventory, etc. Nodes relating such informa- tion are shown on the basis of their relation to the agent which is presented on the graph using a "you" node (see example in Fig. 2a).
Ammanabrolu & Riedl (2019a) build a knowledge graph in a similar manner but restrict themselves single domain. In contrast, we test methods on much more diverse of games defined in the Jericho framework (Hausknecht et al. 2019a). These games are each structured differently- covering wider variety of genres-and and so be able extract the same information from all of them in general manner, we relax many of the rules found in Ammanabrolu & Riedl (2019)). To aid in the generalizability of graph building, we introduce the concept of interactive objects -iemesss-ttems items that an agent is able to directly interact with in the surrounding environment. These items are directly linked to the "you" node, indicating that the agent can interact with them, and the node for the current room, showing their relative position. All other triples built from the graph are extracted by OpenIE. Further details regarding knowledge graph updates are found in Appendix B. An example of graph built using these rules is seen in Fig. 2a
Template Action Space. Templates are subroutines used by the game's parser to interpret the player's action. They consist of interchangeable verbs phrases (V optionally followed by prepo- sitional phrases (VP PP), e.g. ([carry/ hoodd /take] (at/ /against/ poruiiss/oo//onie /onto _). where the verbs and prepositions within are aliases. As shown in Figure 2b actions may be constructed from templates by filling in the template's 's blanks using words in the game's 's vocabulary. Templates and vocabulary words are programmatically accessible through the Jericho framework and are thus available for every IF game. Further details about how we prioritize interchangeable verbs and prepositions are available in Appendix|B.2
and ([drop/throw throw /discadd//dd put)
4 KNOWLEDGE GRAPH ADVANTAGE ACTOR CRITIC
Combining the knowledge-graph state space with the template action space, Knowledge Graph Ad- vantage Actor Critic or KG-A2C, is an on-policy reinforcement learning agent that collects experi- ence from many parallel environments. We first discuss the architecture of KG-A2C, then detail the training algorithm. As seen in Fig. [ KG-A2C's architecture can broadly be described in terms of encoding a state representation and then using this encoded representation to decode an action. We describe each of these processes below.
Input Representation. The input representation network is broadly divided into three parts: an observation encoder, a score encoder, and the knowledge graph. At every step observation con- sisting of several components is received: o, = (Otaracy Ogame Otane at-1) corresponding to the room description, game feedback, inventory, and previous action. and total score R, The room de- scription Or is a textual description of the agent's 's location, obtained by executing the command "look." " The game feedback or is the simulators response the agent's previous action and con-
3

---

Published as a conference paper at ICLR 2020
Graph Mask
Action Decoder
Critic
Template
Decoder GFUUT
Decoder GRUb
Decoder OUUg
Linear
Input Representation
Multi-headed Graph Attention
Greph Embeddings
Encoder GRU
Encoder GRU
Enooder GRU
Encoder GRU
Knowledge Graph
Hoom descriation
Game lessback
Inventory
Previous Action
Binery Score Encoding
Updato
Total Score
Plales
Observation
Figure 1:
The full KG-A2C architecture. Solid lines represent computation flow along which the gradient can be back-propagated.
sists of narrative and flavor text. The inventory Ot.... and previous action d:-1 components inform the agent about the contents its inventory and the last action taken respectively.
The observation encoder processes each component of o using separate GRU encoder. As not given the vocabulary that is comprised of, we use subword cikenizationn-secififca using the unigram subword tokenization method described in Kudo & Richardson (2018). This method predicts the most likely sequence of subword tokens for a given input using a unigram language model which, in our case, trained on dataset of human playthroughs of IF games and contains a total vocabulary of size 8000. For each of the GRUs, we pass in the final hidden state of the GRU step to initialize the hidden state step We concatenate each the encoded components and use linear layer to combine them into the final encoded observation Oç.
At each step, we update our knowledge graph Ge using O, as described in Sec. and then embedded into single vector ge. Bollowing/Ammanahhee networks or GAT's (Veličković al. 2018) with an attention mechanism similar to that described in Bahdanau et al. 2014h. Node features is the number of nodes and F' the number of features in each node, consist the average subword embeddings of the entity and of the relations for all incoming edges using unigram language model. Self-attention is then used after a learnable linear transformation W E 2F applied to all the node features. Attention coefficients aij then computed by softmaxing i E A with V being the neighborhood in which we compute the attention coefficients and consists of all edges in G,,
& Ried] (2019a) we use Graph Attention
computed as H (h1. h2...... bN). hı IR., where N
Cij Leaky W(; h;))
expleis)
ai ΣLEN Cxeyer)(ea)) cxp(ex)
(2)

---

Published as a conference paper at ICLR 2020
Templates
Objects
Jein were of
a trophy case
m
poweur
strange geftic lettering
sword
sword
quantity water
Jewels
Jewels
Moodon dbor
living room
turn
Teorraine
machine
machine
open
glass bottle
case
nailed shut
take
of the room
large oriental
doorway
gothic
gothic
push
all
Living Room You are living room. There doorway the cast, wooden door with strange gothic lettering which appears
from
troll
Troll
west, nailed shut, trophy case, large oriental rug in the center of the room, Above the trophy case hangs an elvish sword of great antiquity. A battery- powered brass lantern on the trophy case. You are carrying:
lantem
grue
grue
door
door
A glass bottle
The glass bottle contains:
A quantity of water.
(b) Visualization of the action decoding process using (a) The extracted knowledge graph for the correspond- templates and objects. Objects consist of the entire ing state. Bolded words in the observation indicate in- game input vocabulary. Greyed out words indicate ob- teractive objects.
jects masked out by the knowledge graph.
Figure 2: overall example of the knowledge graph building and subsequent action decoding process for given state in Zorkl, illustrating the use of interactive objects and the graph mask.
where 2F is a learnable parameter. The final knowledge graph embedding vector ge is computed as:
gt J(W, k=1
(k) oig
w())))) by)
JEN
where k refers to the parameters of the hth independent attention mechanism, W, and by the weights and biases of the output linear layer, and b represents concatenation. The final component of state embedding vector is a binary encoding c of the total score obtained so far in the game-giving the agent a sense for how far it has progressed in the game even when it is not collecting reward. The state embedding vector is then calculated s, ge @ O ① C-.
Action Decoder. The state embedding vector S; is then used to sequentially construct an action by first predicting template and then picking the objects fill into the template using series of Decoder GRUs This gives rise to template policy тт and a policy for each object TD, Architecture wise, at every decoding step all previously predicted parts of the action are encoded and passed along with S; through an attention layer which learns to attend over these representations coniraaiioon-coiiiiiiinnnn every predicted object on all the previously predicted objects and template. All the object decoder GRUs share parameters while the template decoder GRU, remains separate.
To effectively constrain the space of template-actions, we introduce the concept of graph mask, leveraging our knowledge graph at that timestep G, to streamline the object decoding process. For- mally, the graph mask m, = (o o E G, A o E V). consists of all the entities found within the knowledge graph G2 and vocabulary V and is applied to the outputs of the object decoder GRUs- restricting them to predict objects in the mask. Generally, in an IF game, it is impossible to interact with an object that you never seen or that are not in your inventory and so the mask lets ex- plore the action space more efficiently. To account for cases where this assumption does not hold, i.. when object that the agent has never interacted with before must be referenced in order to progress in the game, we randomly add objects o E V to m, with a probability Pm. An example of the graph-constrained action decoding process is illustrated in Fig. 2h
5

---

Published as a conference paper at ICLR 2020
4.1 TRAINING
We adapt the Advantage Actor Critic (A2C) method (Mnih et al. 2016) to train our network, using multiple workers to gather experiences from the simulator, making several significant changes along the way-as described below.
Using a template-action space there are millions of possible actions at each step. Most of these actions do not make sense, are ungrammatical, etc. and an even fewer number of them actually cause the agent effect change the world, Without any sense for which actions present valid interactions with the world, the combinatorial action space becomes prohibitively large for effective exploration.
Valid Actions.
We thus the concept of valid actions, actions that can change the world in particular state. These actions can usually be recognized through the game feedback, with responses like "Nothing happens" or "That phrase is not recognized." In practice, use the valid action detection algorithm provided by Jericho. Formally, Valid(s;) = [o, aI...NN and from this we can construct the corresponding set of valid templates (eatia($1) = {ro, TI TI...TN) {ou, 0100000 } which consists of all objects in the graph mask as defined in Sec. 4 This lets us introduce two cross-entropy loss terms to aid the action decoding process. The template loss given particular state and current network parameters is applied to the decoder GRU, Similarly, the object loss is applied across the decoder GRUo and calculated by summing cross-entropy loss from all the object decoding steps.
follow Hausknecht et al. (2019a) and
We further define set of valid objects (s)
1
Cr(S;, aci
Un, (ogxx( + (1 - yr. vr.)(1 - (ogTT(T.(s..)
(4)
N
M
Cols;, (;;),a;;))==)) =
(oo (oggo,((olls)) + +1 Vo.) (ogTo, (oils:)))
(5)
M
1=1
Ti (salid(St)
yo: = = {
o, E (valid(se) else
Vr. 0
else
Updates. A2C training starts with calculating the advantage of taking an action in state A(s,, at), defined as the value of taking action Q( ae) compared the average value of taking all possible valid actions in that state (st):
A(st, ae) Q(,t, ax) V(st)
(6)
(st, 0z) = E(r, + +VV(
(7)
(st) is predicted by the critic as shown in Fig. and Tt is the reward received at step t.
The action decoder or actor is then updated according the gradient:
(oolo
Σ
logma, (0,1st, r.... (,
(8)
updating the template policy and object policies TO, based on the fact that each step in the action decoding process is conditioned on all the previously decoded portions. The critic updated with respect to the gradient:
2 Vo(Q(st, a; 02) - V(s;;
bringing the critic's 's prediction of the value of being in a state closer to its true underlying value. We further add entropy loss over the valid actions, designed to prevent the agent from prematurely converging on a trajectory.
Lu(St, 01; 0,)= aEV(s.)
Σ P(als))logP(alss)
(10)

---

Published as a conference paper at ICLR 2020
5 EXPERIMENTAL RESULTS
The KG-A2C is tested suite of Jericho supported games and is compared to strong, established baselines. Additionally, as encouraged by Hausknecht al. (2019)), we present the set handicaps used by agents: (I) Jericho's ability identify valid actions and (2) the Load, Save handicap in order to acquire Otanse and Ot using the look and inventory commands without changing the game state. Hyperparameters are provided in Appendix [
Template DQN Baseline. We compare KG-A2C against Template-DON a strong baseline also utilizing the template based action space. TDQN (Hausknecht et al.] 2019a) is extension of LSTM-DQN (Narasimhan et al. 2015) to template-based action spaces. This is accomplished using three output heads: one for estimating the Q-Values over templates Q u) u)vu E T and two for estimating Q-Values (st,, 01)) )1 (( 02)Vo; E O over vocabulary to fill in the blanks of the tem- plate. The final executed action is constructed by greedily sampling from the predicted Q-values. importantly. TDQN uses the same set of handicaps as KG-A2C allowing a fair comparison between these two algorithms.
Table shows how KG-A2C fares across a diverse set of games supported by seachho-estting the agent's s ability to generalize different genres, game structures, reward functions, and state-action spaces. KG-A2C matches or outperforms TDQN on 23 out of the 28 games that we test on. Our agent is thus shown be capable of extracting knowledge graph that can sufficiently constrain template based action space enable effective exploration in a broad range of games.
ABLATION STUDY
In order to understand the contributions of different components of KG-A2C's architec- ture, we ablate KG-A2C's knowledge graph, template-action space, and valid-action loss. These ablations are performed on Zork f and result the following agents:
Game
T
[V]
TDQN
KGA2C
MaxRew
296
acorncourt
151
343
1.6
0.3
30
advent!
189
786
36
36
350
adventureland
anchor
260
2257
0
100
awakea
159
505
50
A2C removes all components of KG-A2C's knowledge graph. In particular, the state em- bedding vector is now computed St. Or and the graph mask is not used to constrain ac- tion decoding.
balances
156
452
4.8
10
51
deephome
173
760
300
detective
197
344
207.9
360
dragon
177
1049
-5.3
25
enchanter
inhumane
141
409
0.7
300
jewel
161
657
90
kam
161
657
1.2
90
KG-A2C-no-gal remove's 's the Graph Attention network, but retains the graph masking compo- nents. The knowledge graph is still constructed usual but the agent the same state bedding vector as A2C.
library
173
510
6.3
14.3
30
ludicorp
187
503
17.8
150
moonlit
166
669
U
omniquest
pentari
155
472
17.4
50.7
70
snacktime
201
468
9.7
50
288
1013
5.8
400
KG-A2C-no-mask ablates the graph mask for purposes of action decoding. The knowledge graph is constructed as usual and the agent re- tains graph attention.
spellbrikr
333
844
18.7
21.3
600
spirit
169
0.6
temple
175
622
7.9
7.6
35
zork1
237
697
9.9
34
350
zork3
214
564
.1
186
607
4.9
100
On Zorkl as shown in Figure 3 we observe
similar asymptotic performance between the all of the ablations- all reach approximately points. This level of performance corresponds to a local optima where the agent collects the majority of available rewards without fighting the troll. Several other authors also report scores at this threshold (Jain et al., 2019; Zahavy et al. (2188). In terms of learning speed, the methods which have access to either the graph attention or the graph mask converge slightly faster than pure A2C which has neither.
Table 1: Raw scores comparing KG-A2C to TDQN across wide set of games supported by Jericho. Advent starts at score of 36.
A map of Zorkl with annotated rewards can he found in Appendix along with transcript of KG-A2C playing this game.

---

Published as a conference paper at ICLR 2020
Agent
Mask GAT
A2C
KG-A2C-no-gat
KG-A2C-no-mask
KG-A2C-full KG-A2C-unsup
KOHATE
ROAAD ADD no
Figure 3: Ablation results on Zorkl, averaged across 5 independent runs.
To further understand these differences we performed larger study across the full set of games comparing KG-A2C-full with KG-A2C-no-mask. The results in Table KG-A2C-full outper- forms KG-A2C-no-mash on 10 games and is outperformed by KG-A2C-no-masd on 6. From this larger study we thus conclude the graph mask and knowledge graph are broadly useful components.
We perform two final ablations to study the importance of the supervised valid-action loss and the template action space:
KG-AeC-unsuperviised order to understand the importance training with valid-actions, KG- A is allowed access the list of valid actions the valid-action-losses Lr and Lo are disabled and Lg now based on the full action set. Thus, the agent must explore the template when trained for the same number of steps all the
action space manually, KG-A2C-unsupervisse other agents, fails to achieve any score. We can infer that the valid action auxiliary loss remains an important part of the overall algorithm, and access the knowledge graph alone is not yet sufficient for removing this auxiliary loss.
KG-A2C-seq discards the template action space and instead decodes actions word by word up to maximum of four words. A supervised cross-entropy-batt -bsee valid action loss Lvana is now calculated by selecting random valid action at,sa E Valid( st) and using each token in it as a target label. As this action space is orders of magnitude larger than template actions, we use ceacher-forcing to enable more effective exploration while training the and the decoded action otherwise. All other components remain the same as in the full KG-A2C.
ar,m with a probability Pvalid = 0.5
KG-A2C-seq reaches relatively low asymptotic performance of 8 points. This agent, using a action space consisting of the full vocabulary, performs significantly worse than the rest of the agents even when given the handicaps of teacher forcing and being allowed to train for significantly longer- indicating that the template based action space is also necessary for effective exploration.
CONCLUSION
Tabula rasa reinforcement learning offers an intuitive paradigm for exploring goal driven, contextu- ally aware natural language generation. The sheer size of the natural language action space, however, has proven to be out of the reach of existing algorithms. In this paper we introduced KG-A2C, novel learning agent that demonstrates the feasibility of scaling reinforcement learning towards nat- ural language actions spaces with hundreds of millions of actions. The key insight to being able to efficiently explore such large spaces the combination of a soowledge-graph-bee a template-bass action space. The knowledge graph serves as a means for the agent to understand its surroundings, accumulate information about the game, and disambiguate similar textual obser- vations while the template-based action space lends a measure of structure that enables us to exploit that same knowledge graph for language generation. Together they constrain the vast space of possi- ble actions into the compact space of sensible ones. A suite of experiments across diverse set of 28 human-made IF games shows wide improvement over TDQN, the current ttae--ofthhe-s template- based agent. Finally, ablation study replicates state-of-te sof-the-art performance on Zorkl even though KG-A2C is using an action space six orders of magnitude larger than previous seunt--indicating the overall efficacy of our combined state-action space.
state space and
8

---

Published as a conference paper at ICLR 2020
REFERENCES
Prithviraj Ammanabrolu and Mark O. Riedl. Playing text-adventure games with graph-based deep reinforcement learning. In Proceedings 2019 Annual Conference of the North American Chap- ter of the Association for Computationall Linguistics: Human Language Technologies, NAACL- HLL 2019, 2019a.
Prithviraj Ammanabrolu and Mark O. Riedl. Transfer in deep reinforcement learning using knowl- edge graphs. CoRR, Tet///908..65555 2019b.
Prithviraj Ammanabrolu, William Broniec, Alex Mueller, Jeremy Paul, and Mark O. Riedl. Toward automated quest generation in text-adventure games. CoRR, abs/ 1909.06283,,,, 2019.
Gabor Angeli, Johnson Premkumar, Melvin Jose, and Christopher D. Manning. Leveraging Lin-
guistic Structure For Open Domain Information Extraction. In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Confer- ence on Natural Language Processing (Volume Long Papers), 2015.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning align and translate. arXiv: 1409.0473, 2014.
Marc-Alexandre Côté, Ákos Kádár, Xingdi Yuan, Ben Kybartas, Tavian Barnes, Emery Fine, James Moore, Matthew Hausknecht, Layla EI Asri, Mahmoud Adada, Wendy Tay, and Adam Trischler. Textworld: A learning environment for text-based games. CoRR, abs/1806. 1532, 2018.
Nancy Fulda, Daniel Ricks, Ben Murdoch, and David Wingate. What can you do with rock? affordance extraction via word embeddings. In IJCAI, pp. 1039-1045, 2017. doi: 10.24963/ijcai. 2017/144.
Marjan Ghazvininejad, Chris Brockett, Ming- Wei Chang, William B. Dolan, Jianfeng Gao, Wen tau Yih, and Michel Galley, A knowledge-groune neural conversation model. In AAAI, 2017.
Jian Guan, Yansen Wang, and Minlie Huang. Story Ending Generation with Incremental Encoding and Commonsense Knowledge. 1808 2018.
Matthew Hausknecht, Prithviraj Ammanabrolu, AlxanAle Côté, and Xingdi Yuan. Interactive fiction games: A colossal adventure. CoRR, abs/1909.05333 2019a.
Matthew J. Hausknecht, Ricky Loynd, Greg Yang, Adith Swaminathan, and Jason D. Williams. NAIL: A general interactive fiction agent. CoRR, abs/a abs//199,. 2019b.
Ji He, Jianshu Chen, Xiaodong He, Jianfeng Gao, Lihong Li, Li Deng, and Mari Ostendorf. Deep reinforcement learning with a natural language action space. In ACL, 2016.
Vishal Jain, William Fedus, Hugo Larochelle, Doina Precup, and Mare G. Bellemare. Algorithmic improvements for deep reinforcement learning applied to interactive fiction, 2019.
Taku Kudo and John Richardson. Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text processing. CoRR, abs/1808.062266, 2018
Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word represen- tations in vector space. CORR, abs/13011.37 2013.
Volodymyr Mnih, Adria Puigdomenech Badia, Mehdi Mirza, Alex Graves, Timothy Lillicrap, Tim Harley, David Silver, and Koray Kavukcuoglu. Asynchronous methods for deep reinforcement learning. In International conference on machine learning, pp. 1928-1937, 2016.
Karthik Narasimhan, Tejas D. Kulkarni, and Regina Barzilay. Language understanding for text- based games using deep reinforcement learning. In EMNLP,, pp. 1-11, 2015.
Jack Urbanek, Angela Fan, Siddharth Karamcheti, Saachi Jain, Samuel Humeau, Emily Dinan, Tim Rocktschel, Douwe Kiela, Arthur Szlam, and Jason Weston. Learning to speak and act in a fantasy text adventure game. CoRR, abs/ ths/11900 1101109033.44 2019.

---

Published as a conference paper at ICLR 2020
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liô, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations (ICLR), 2018.
Xusen Yin and Jonathan May. Comprehensible context-driven game playing. CoRR, abs/1905.02265, 2019.
Xingdi Yuan, Marc-Alexandre Côté, Jie Fu, Zhouhan Lin, Christopher Pal, Yoshua Bengio, and Adam Trischler. Interactive language learning by question answering. In EMNLP, 2019.
Tom Zahavy, Matan Haroush, Nadav Merlis, Daniel Mankowitz, and Shic Mannor. Learn what not to learn: Action elimination with deep reinforcement learning. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Infor- mation Processing Systems 31, pp. 3562-3573. Curran Associates, Inc., 2018.
Mikulás Zelinka. Using reinforcement learning learn how play text-based games. CoRR, Abs//100,01199, 2018.
10

---

Published as a conference paper at ICLR 2020
A
ABLATION RESULTS
Game
(
V
KGA2C-Full
KGAZC-unmasked
MaxRew
905
82
296
acorncourt
151
343
0.3
0.3
30
advent
189
786
36
36
350
adventureland
156
398
100
anchor
260
2257
0
0
100
awaken
159
505
0
50
balances
156
452
10
10
51
deephome
173
760
1
29.2
300
detective
197
344
207.9
141
360
dragon
177
1049
0
-22
25
enchanter
290
722
12.1
7.6
400
inhumane
141
409
3
10.2
300
jewel
161
657
1.8
1.3
90
karn
161
657
0
0
90
library
173
510
14.3
9.6
30
ludicorp
187
503
17.8
17.9
150
moonlit
166
669
0
omniquest
207
460
3
5.4
50
pentari
155
472
50.7
50.4
70
snacktime
201
468
0
0
50
sorcerer
288
1013
5.8
16.8
400
spellbrkr
333
844
21.3
30.1
600
spirit
169
1112
1.3
1.3
250
temple
175
622
7.6
6.4
35
zenon
149
401
3,9
3.1
350
zorkl
237
697
34
27
350
zork3
214
564
.1
.1
7
ztuu
186
607
9.2
100
Table 2: Ablations
B IMPLEMENTATION DETAILS
B. KNOWLEDGE GRAPH UPDATE RULES
Candidate interactive objects are identified by performing part-of- tagging on the current observation, identifying singular and proper nouns well as adjectives, and are then filtered by checking if they can be examined using the command ezamine OBJ. Only the interactive objects not found in the inventory are linked to the node corresponding to the current room and the inven- tory items are linked to the "you" node. The only other rule applied uses the navigational actions performed by the agent to infer the relative positions of rooms, e,g. (kitchen, down, cellar) when the agent performs go down when the kitchen to move to the cellar.
B.2 TEMPLATE PREPROCESSING
Templates are processed by selecting a single verb and preposition from the aliases. For the sake of agent explainability, we pick the verb and preposition that are most likely to be used by hu- mans when playing games. This done by assessing token frequencies from a dataset of hu- man playthroughs such those given in ClubFloyd] This dataset consists of 425 unique play sessions and 273,469 state-action pairs. The examples given earlier, (lcarry/holl hold/Lake] hal/agnimst /nn onto would then be converted lake.
and
(Idrop/ hifrop//hrow// throw///w//// htwhh////////rrr//////r/////// didddiiddddpp and put
html #clubfloyd

---

Published as a conference paper at ICLR 2020
C EXPERIMENT DETAILS
Episodes are terminated after 100 valid steps or game over/vietory. Agents that decode invalid actions often wouldn't t make very far into the game, and we only count valid-actions against the hundred step limit. All agents are trained individually on each game and then evaluated on that game. All A2C based agents are trained using data collected from parallel environments. TDQN was trained using single environment. Hyperparameters for all agents were tuned on the game of Zork/ and held constant across all other games. Final reported scores are an average over runs of each algorithm.
WYYY
4.0
201
1000
A0e
100
100 10 Libn 1AUO
Figure 4: Learning curves for KGA2C-full. Shaded regions indicate standard deviations.
12

---

Published as a conference paper at ICLR 2020
ZORK I
Legend
The Great Underground Empire
Cag
Legend
Kitchen 10
Start here
gellar
Painting +4
Figure 5: A map the world of ZorkI with some initial rewards annotated. The blue arrow indicates connection between the left and right maps, corresponding to the overworld and the dungeon.
D ZORK1
Zork/ was identified by Hausknecht et al. (2019a) to be one of the most difficult games their suite and the subject of much prior work (Zahavy et al. 2018 Yin & May 2019). Zork] is of the earliest IF games and a dungeon-cravler-2 player must explore a vast labyrinth while fighting off enemies and complete puzzles in order to collect treasures. It features relatively sparse reward for collecting a treasure or moving along the right path to one, and stochasticity in terms of random enemy movements.
To understand how humans progress in Zorkl, a group of 10 human flayers-familiar with IF games- were asked play Zork/ for the first time (with no access to walkthroughs). Half of the players reached a game score of around 40 before dying to the first beatable NPC, a troll, mostly due to neglecting collect a weapon to fight it with beforehand. Three of the remaining players died to hidden traps even before reaching this point, achieving scores between 5 and 15. The final two players made it significantly past the troll gaining scores of around 70.
The following transcript of KG-A2C playing Zorkl, shows top predicted probabilities for templates and objects.
Obos Dese: Wost of House You are standing in an open field west of white house, with a boarded fronz door, yhanded, Pewdbauks West of House You are atanding snall mailbox Tomplace probss open CBT: 0.339, cast: 0.271, sounh: 6.215, west: 0.094, nozth: 0.021, qo around cac: 0.013, blow out 2Bd: 0.011, theoww 0B3 au c3T: 0.01, mustimest: 0.009
There is snell Thux bere. Tavs You boarded front
field
white house,
ob jeject probas windews 0.7, anail; 0.259 object boarded, snall, Aclions wesl.
door,
nai house,
Score: D
foress, with in all diswet Inv: are enocyhanded. Foadback: Foreat This forest, with trees directions. east, there appeers to be sunlight.
the there apgeara sualight
Template peubes CBT: 0.49, souths 0.304, wauls 0.116, waals 0.04, nucths 0.015, asound OBJ: 0.011, 0.005, northwest: 0.006, blow 0.006 Object probes windows 0.7, small: 6.3 Intesactive oblectas fosest, all, Lzees
Act eest
Gbet neec: Forest Path This is a path winding ng th through dinly lit forest, The path heeds northagut's here, One part icularly large sowe with soow low boanches stands the edge of the path, Ines You anplyhanded. path vinding through dimly torest. pata heads northsouth here One particnn cu anly. large tree wich some low branches stands at the edge the path.
Feedback: Foress Path
13

---

Published as a conference paper at ICLR 2020
Templace probs: up: 0,386, opan cBJ: 3,35,, east: C,123, west: 0.05, nerch: 0.023, go acound QBJ: 2.014, blow out CBI: 0.01, nozthwest:: 0.039, throw oed at 0B3: 0.008
Object pross: vindew: 0,701, small: 0.295 Interact.ive dafectt: tree, path, branches, Corest, large, all
Action: up
Score: 0
0bs: Dese: Tree You about 10 fest abeve the cround nestled
large branches. The nearest beanch above you is aboow your eeach, Bealde you un the branch is a anall birds nest. Ia the birds neat La a lares egg encrusted with precious jevala, asparently seaveraed by a childless sorabird. The eaq is cevored with tine gold inlay, and ornancated in lapis lazuli and notherotpeart. Unlike nost cezs, this closed with delinate look claap.
way appeazs exicumlly fragile. aratvhanded. Feadback: Up Iree You are about feet above the ground nestled anong some large branches. The nearest brench above you is above your reach, Boside you on the branch is small birds neat, In the birda neat is large wau enczusted with precious javela, appacently scavenged by childleas songbird. The netherofpearl Unlike nost eigh, this one is hinged and closed wish a delicate looking claap, The egg appears exirenely fragile.
You
covared
fine geld inlay,
ornanented in
lazell and
Template probs: weat: 0.63, open QBJ: 0.18, do around CBJ: 0.074, east: 0.064, north: 0.063, south: 0.C52, 083: 0,041, pat
OBT: 0.C23,
0,014, clase Olds 0.005
ob ject probas snall: 0.213, window: 0.184, house: c.116, wgg: 0.068, whites 0.067, pach: 0.062, branch: 0.042, kitche: 0,024, nest: 0,013, greund: 0.009
Intssact lvv oblectss neal, large, eyg, ground, beanches, all, branch, Lowe Actions open
Score:
Obs:
Dese: Up Tree You aze about 10 fest above the ground nestled anong some lazge branches. The neazeat branch above you above your reach, Reside you the branch is snall
nest. In hirds is a large egy encrusted with precious juvela, agparently scavenged by a childleas sunghird. The agg La eevered with fine gold inlay, and crcananced in lapis lazul; and notherospearl, Balike noat eeta, this is hinged and olosed with delinaee looking claap. The egg appeats extrenely fragile. You hear in the distance di thipping of song bird. lovs You is already open.
saptyhanied. You in the distance
song bird. Foodback: Template probes ueat: 0.347, agen Old: 0,182, go aroand ond: 0.085, mest: 0.077, north: 0,073, south: 0.058, Lake 0BJ: 0,045, put duan OBT: 0.C25, down: 0,018, clese OBJ: 0.013
chirping
Object pross: snall: 0.213, windows 0,183, house: C,117, eggt 0.068, whites 0.057, pasht 0.062, brancht 0,042, kitche: 0.024, nest: 0,018, ground: 0.009
Ivve objectas nest, lasye, egg, ground, beanchees, all, branch, Lewn Act ons take egg
Score:
Obet nese: Up a Tree You are sbous 10 feet above the ground nestled anong some large branches, The nearest branch aboie
is above your seach. Bexide you the branch is snall birds next. Tave You carrying A jswelencrusted egg Feedback: Taken,
Template probes down 0,412, open ORd: 0,181, go arourd ond: 0,034, eests 0.065, north: 0,063, souths 0.053, Lake OBJ: 0,042, pas down 0BJ: 0.025, down: 0.014, close OBd: 0.01
object proas: emall: 0.213, windows 0,184, nouse: 0,116, egg: 9.068, whites 0.00%, past: 0.062, brancat 0.042, kitcher 0.024, nest: 0,018, grourds 0.009
Intesactive object neat, lazge, egg, ground, bearches, all, branch, Lzme
Action: down
Score:
0b5: Desa: Forest Path This is a path winding through a dinly Lit torest, The path heads northeouta here. One part voulaau y large cree with some low branches stands at the edge of the path, Tave You are empt shanyhenddd.
Feedback: Forest Path
Template probst south: 0,3,,, open OBJ: 0.358, east: 0.123, west: 0,36, norths 0,023, go around 0B3: 0,014, blow out ond: 6.01, northeest: : 0.009, throw old at ONJ: 0.008
Object proas: windew: 0,701, small: 0.295 Interactive oojects:
path, branches, forest, large, all
Action: south
Score:
0bs: Dese: North Bouse You are facing the
side white house. There
here, and all the
windove are boarded up. To the norta narzow path winde througt. the trees. Invs are enptyhandes.
Peedbacks Norch of House You are facing the north side of white house. There ia no door here, and a
the windows are bearded up. To the north a nacrex path winds tarough the trces.
Template probs: west: 0.502, open 0821 0.171, 0.0%, 0.147, north: 0.399, south: 0.035, throw OBJ at 032: 0,012, go around
0,012, object proas: windew: 0.124, small: 0.243, trap: 0.014, kitche: 0.009, rug: 0.003
and: 0,005
Interactive oajecte: path, windexs, narrow, white, house, all
Action: west.
Score:
Obs: Dese: Hest of Bouss You are standing in an open rield west of white house, with boarded fronz door. There is emall mailbox here. ra9: You emptyhanded. Fredback:
Template probs: open CBT: 0.538, south: 0.261, east: 6.121, west: 0,353, north: 0,015, go around oBC: 0,011, throw 220 at OBJ: 0.007, blow ouz cas: 0.00s, nesthwest: 0.005
Object proas: vindew: 0.7, small: 0.3 Interactive decetts: hoarded, anail, front, deor, white, mailbox, heuse, all Action: open white
Score: 5
Obs: Dese: Hest of Bouss You standing
open field wost white house, with boarded front door. There is a small moil here. Iav: are emptyhanded. Feedback: cant now to get in trom here. Templata probs: open car: 0.339, east: 0.271, sourh: 0.215, west: 0.094, nosth: 0.021, go around oac: 0.013, blow out 083: c.011, throw 0BJ at car: 0.01, northwest: 0.009
jejtct probes windews 0.7, amail: 0.259 Interactive cofecta: hoarded, snall, front, door, whita, mailbox, heuse, all Action: west
Score: 5
14

---

Published as a conference paper at ICLR 2020
oba Dese: Poreat This ia forest, with trees in all dizectionn. to the east, there appears be sualight.
Inv: You are enpeyhanded. Feedback: Forast This 1o a tozest, with trees in all dicections. To the east,
there appcors to be cunligat.
Template probs: open CBT: 0.49, south: 0.304, throw 020 at OBJ: 0.005, noxtawest: 0.006, blaw out oBd: 0.006 object press: windew: 0.7, snsll: 0.3 Interactive dafecta: forest, all, treas
0,116, weats 0.04, north: 0.015, go azound CBJ: 0.011,
Action: east
Boare:
Obs: Dese: Pozest Fath This is a path winding througa dinly 1it forest. The path hoads northzouta here. cae part iculesty large with low beauches stands the edge the path, laes You unplyhanded. pach vinding through dimly One particularly Large tree wish sone lox branches stands at the odge of the path.
Feadback: Porest Path This
forest. The sath heads northsouth here
Template prubes south: 0,386,
OBJ: 0.358, seat; 6.123, west: 0.06, north: 0.023, around OBJ: 0.014, 0.01, northwest: 0.C03, throx
blow
0BJ: 0.005
Object probes windows 0,701, snall; 0.298
Interactive objectt: Lrew, path, branches, Corest, large, all
Action: opan all
Buore:
Obst Dese: Ferest Path This is a path winding through a dinly 1it ferest. The path hoads northseuth here, Caa part icularly large Law with now low beanchee stands the edge of the path, Ines You new enplyhanded. Feadback:
clear what yeure reforring
Templata probst open CBT: 0,331, souch: 0.259, east: C.24, west: 0.037, nerth: 0.031, go around OBJ: 0.013, blow out oBd: 0.011, northawst: 3.01, Lhrow 0Bd as 0BJ; 0.009
jbjectt probas windews 0.701, small: 0.295 Interact objests: tree, path, branches, Forest, large, Actions waut.
Score: 5
Obaz Deses Poseal This dinly Cuceal, with
all around, Lavs You shenyle You hear in the distance the chieping of song bird. Feedback: Forest This dinly 1it forest, with large trees all around. You bear in the distance the chi chirping of a song bird.
Templabe probes southiz 0,38, open GBd; 0,281, wauls 0.174, waals 0.072, norths 0.034, around 0BJ: 0,015, out
6.013, northwest: 0,012, throw OBd at oRds 0.008 Object probes windews 0.7, snell: 6,289 Intesadi ive dbjecta Iosest, lazge, all, Liowa
Act ons south
Score:
Obes neees Glearing You are in a ssa'l I clearing in a well narked forest path that exterds to the east and west Iivs You are enplyhended. Feedback: Cleasing You are in snall cleaning in well narked foowst pach that extende so the east
west, Template probss open CRT: 0.45), east: 0,364, wests 0,164, north: 0,008, throw GNJ at ORd: 0.005
dbjject probos windows 0.592, small: 0,405, kitcle: 0.002
Interactive cojects: torest, path,
Boares
Ohhs
neses Canyon View You
the of the Great Canyon its west well. Pron here there is nervelous view or the canyon and pacts of the Frigid River upetrean. Across the canysn, the wolls ot the Mhite Blitts join the nighty renparte of the Flathead Mountains to the east. Following the Canyon upstreem to the north, Aragain nn Fally nay be seen, complate with rainbow. The nighty Frigid River floss out fron grest dark cavern, To is possible to clinb
imense torest, stretcaing for miles azound. erptyhanced, Peedbacks Canyon Viex You are at the top the Great Canyon on isa west wall, Pron here there 18 a narvelous view of the canyon and parts at the Frigid River upstrean, Acress the canyor, the walls ot the White clifts join the mighty ramparts of the Platheed Mountaine upstrean the nerth, Aragein Falls may be soon, complete with rainbow. The nighty Frigid River floms out zrom a great cark cavern. To the west and south can ze seen an immence corest, stretching for miles around. A path leads northwest. It is possible to climb down into the canyon from here. Template probs: west: 0.444 take 082: 0,206, open 032: 0.147, blaw out 0B3: 0.118, turn 0a2: 0.027, cast: 0,026, downs 9. 000, go around 083: 0.005, pach 083: 0.004
path leade northvest,
canyon from here. Iev:
the east lowlowing the Canyon
object pross: wines 0, 0,693, snall: 0.307
Interactive cofects: path, walls, srigid, rivor, rainbow, wall, all Action: west
Score:
Obst Desc: Forest This is a dinly lis torest, with large trees all around. Iav: You are empryhanded. Feedback: all around. Template probs: south: 0,496, open OBJ: 0.201, east: 0.105, west: 0,353, north: 0.020, go around oac: 0.018, northvest: 0.015, blew out OBJ: 0, 014, throw GBd at 0B3: 0,007, down: 0.005
Forest is dimly lit forast, with large
object prohs: windew: 6.699, small: 0.301 Interactive cofects: torest, large, all, Action: south Scorc: 5
Obs: Dese: Porest This dinly forest, with large trees all around. lav: You exocyhanded. Feadback: Storntoszed sxces block your way.
Template probs: ezst: 0,722, north: 0.051, west: 0.001, open OB3: 0,061, south: 0.048, blew out 0B3: 0,011, nerthveat; 0.01, around OBJ: 0.007, throw OBJ at 0B3: 0.005
object pross: windew: 0.7, smail: 0.3
Interactive cofects: storm, tarest, tzees, large, all Action: east
Scorc: 5
Obs: Dese: Forest This dinly 1i: fosest, with large trees all around. lav: You are expcyhanded. Feadback: The rank undergrewch provents castward novenent. Template proba: south: 6,434, open OBJ: 0.339, eaat: 0.076, weats 0.065, noxth: 0.024, around dac; 0.02, nerthweat: 0.012, biew
0.011, throw 0BJ 083: 0.909 Object proas: vindow: 70701, small: 0.295
15

---

Published as a conference paper at ICLR 2020
Interactive dbfecta: fosest, lazge, all, trees Action: Sump
Scorc: 5
Obs: Dese: Forest This dinly 14: cosest, wita lazge trees all acound. lnv: You are expeyhanded. Feodback: Kanesscsecs !!!! You hear in the distance the chirping of a song inzzd.
Template probs: south: 0,438, open OBJ: 0.292, east; c.098, weat: 0,075, noxth: 0.02, around OBJ: 0.019, nerthwest: 0.013, biew
0,012, threw 0BJ 083: 0.01
db juct prober windows 0.7, amall: 0.259 Interactivee obfecta: Zorest, lazge, all, trees Action: south
Buore:
0bs: Dese: Porest This is a dinly 11: torest, with lorge trees all around. lav: You are erpeyhanded. Feodback: BLorntossed Lrees block your
9.07, OBJ: 0.067, south: 0.043, noxthwast: 0.012, blew out CBJ: 0.011, go asound OBJ: 0.007, throw ond as OBd: 0.006 jejt probas window: 0.7, amall; 0.3 Interactive cofects: acozn, forest, trees, lazge, all
Template probs:
0,694,
0.091,
Actions
Boore:
Obos Deses Cleas ing You are in anall cleacing in well nacked Curwst pach Chat extends to the wast and wes You anptyhanded. Feedback: cleazing
Templata probs: weat: O, 037,, aast: 0.211, open 0BJ: 0.171, noxth: 0,026, thzow CBJ at 0B3: 0.015, south: 0,013, blow out 0BJ: 0,011, go around 083; 0,006, northweat: 0.C05
jbjectt probas windews 0.699, small: 0.301
Interact objests: forest, peth, Actions theow path at forwsi Score: 5
in snall clear lnn well macked Curest path Chat extends Lo Lie wast and Inve You aze anptyhanded. Foedbacks You dont have the way. Template probes open ChT: 0.377, wess: 0,541, eeads 0.149, south: 0,057, norths 0.022, throw ORJ at ORd: 0,018, blow out OBJ: 0,011, acound 0B3: 0,011, northweals 0.c08 Object pross: windews 0.7, snall: 0.289 Interact ive objectas forest, path, all Actions west.
Obaz Deses Cless ing
Score: 5
Obas Desas Behind House You ase behind the while house. à path leada into the Foreat co the east. one corner of the house there is a sma 11 window whi ch is elight ly ajar. tres You are eharydadded, Feedbacks behind the whice house. path leads into the forwxt to the wast. In one porner of snall window waich slight.y agar.
Behind House You
house there
Template probss open CRT: 0.953, west: 0,036, south: 6.006
dbjectt probos small; 0,895, windows 0.2 Interactive oojects: path, windew, snall, white, open
sll
Soares
Ohhs neees Rehl nd House You behiid the white house. à path 1 Teads into the forest to the east. In ccrner ot the hause there is small window which is elightly aiar. Irv: You are empsyhandes, Foedback:
The window is closed,
Template probes open CRT: 0,943, west: 0,047
Object proas: emall: 0.88, window: 0.12
Interactive oojecte: path, windww, snall, white, 300se, all Act onn open peth
Score: 5
Obs: Deso: Bchind House You are behind white Jouse. A path leads into the sorest so the cast. one ocener at the house there is a snall window which is slightly aiar, irvi You are emptyhanded. Feedbask: You must tell ne how to do that to a way.
0,048, Object pross: vindew: 0,58, emails 9.41%, kitcae: 0,002
Template probs:
CBT: 0.899,
0.044, south: 0.004
Interactivv oojeste: path, wi ndeww, snall slall, white, house, all Action: open window
Score:
0bs: Desc: Behind House You are behind the white house, A path leads into the forast =o the cast. In one octner at the house there is a snall windox which is open. Inv: You are emptytanced, Feedback: With acfort, you open the vindow far enouth allow entry.
Template probs: sest: 0.997
object prosst smail: 0.951, vincow: 0.049
Interactive cofects: path, windon, snall, waite, house, all Action: west
Score:
0bs: Desa: Kitchen You are in the kitchen oz the white house. & table seens to have been used receatly for the dark staircase proparation of food. passage leads the snall vindox which open. the table is an elaneated dack chinney leads dewn brown sack, sneliing of hot peppers. A bottle is sitting on the table. The glass nottle contains A quantity of water Inv: You aze amptyhanded, Foedback: Kitchen You are in the kitchen of the white house. recently
leading upvard.
the east
proparation of food. dazk staircase can be seea lesding upward. dack chinney leads dow and to tae east is small window which is open. on the table is an eloneated brown sack, snslling of hot poppers. A bottle is sitting on the table. The glass bottle contains A quantity of wcter Templace probes
table
leads the
cBI: 0.47, 0,012, east: 0.008, close CBJ: 0.co5, chrow 0Bd at OBJ: 0.005
0.351, tarn 083: 0.061, alew
0.052, downs 0.023, oper. 06J:
object pross: windew: O, 682,, small: 0.317
Interactive dla datectas xitchen, window, passage, staircase, glass, sack, wates, small, sable, chimney, bot:le, quantity, browe, all Action: west.
16

---

Published as a conference paper at ICLR 2020
Bcore: 15
0bs:
Dese: Living Roon. You are in the living zoon, Ihere is doerway to the east, a wooden door with strange which appears
gethic lettering the in the center the roen, Abova the trophy case hangs an aivish aword great aatiquisy. à
nailed shat, :rophy case, and large oriental
battarypowerdd brase lantern i2 or the trephy case. inv: You are cmgryhanded. Foedback: Living Roon You are in the living roen. There is doorway to the east, wooden door with strange gothie lettering to the west, which appears to rasled shut, roun, Abovw the Laophy case banga an wlvish sourd of great antiquaty. bellasypeweeed brass lantern La on the trophy case.
case, and lorge oriental rug in the etater
Template probs: take car: 0,456 west: 0.241, tern en CBJ: 0.107, open OBJ: 0.053, dowe: 0.05, nlew cat OBJ: 0B3: 0.0:1, waals 0.009, north: 0.005
0,034, close 0BJ: 0.015, Chrow
object preha: vindew: 0,685, snall: 0.315
Interactive cafects: civish, strange, traphy, hrass, wooden, rug, oriental, lettering, antiquity, sword, gethic, veat, dous, large, lantert, case, all Actions take brass
Scare:
Obs: Dese: Living Room. You are in the living zoon. There is doorway to the east, a wooden door with strange appears Lo nailed shat, crophy case, and large usiental rug in the centez of the zoon, Above the Lrophy case hanga an elvish award of geeat antiquisy. lav: You are carzying A beass lantera Foodback: taken.
gethic letincing
the
Template poubes open CBI: 0.48, push oBd: 0.138, wests 0.058, Lake oad; 0.067, Luen on CBJ: 0.059, throw DB3 at 5B3: 0,031, put down 0BJ: 0,025, put CBJ in oad: 0.018, put 08d OBJ: 0.016, down: 0.015 Object prohet windows D, 685, small: 0.315
Interact live abjectas wlvishi sizange, Lrophy, braes, wouden, rug, us inntall lelleeing, iguily, aword, gethie, veat, dooz, large, lanterc, case, all Action turn brass
Boores 15
Obbs neses Livingg Boter You are in the living room There is doorway to the east, wooden door with strange ccophy case, and large usienial zug in the centez of the roon, Abova the Lrophy case hanga an alvish: aword of gueat antiquisy. lav: You are varrying A bress lantern (providing light] Peedback: The brass Lanterr is now on. CBI: 0.438, push GBJ: 0.141, seal: 0,088, Lake OBJ: 0_081, Luze DBJ: 0.361, Uhrew CaJ at bRd: 0,036, put down OKJ: 3,025, pur CBJ in oRd: 0,019, down O, 0,0,8, pot ORJ on 03d: 0.018
gathic letiieznn the
shich appears Lo nailed
Template poubez
Object probes windews 0,685, small; 0.315
Invesadiiee objectas brass, strange, Leuplyy, elviah, sug, ociental, light, lettering, anciquity, saurd, gothie west, door, large, wooden, lantern, case, al Actions take trephy
Boore: 15
Beses Living Boter You are in the living roon, There is duorway to the west, wooden door with strrange gothic lettering to the vest, which appeare to nailed shat, trophy case, and large orsental rug in the center of the roon, Above the trophy case hangs en elvish avord of great intiiuitty Tav: You are varsying A bress lantern (providing light) Peedback: The Leophy case is socurely fesiened to the well. Template probs: open GBI: 0,43%, push OBJ: 0.141, west: 0,00%, take OBJ: 0,082, ture 0BJ: 0,361, torow Car at bids 0,035, put down OKT: 3.024, downs 0.019, put ond or oRd: 0,018, por and oWd: 0.018 Ob jest probes windews D, 6884, small: 0.315
Obes
Interactive onfecte: strange, trophy, case, antiquity, Lancern, lettering, eword, gothic, wooden, welly elvish brass, west, oriental, door, large, light, all Action: push sase
Score: 15
0bs: Desa: Living Roon You
wooden with strange gothic lettering the vest, which appeare to led shat, srophy case, and large orsental rug in the center of the roon, Above the trophy case hangs an elvish avord of great ant intiigu igy. Tave You are carrying A bracs lantern (providing light) Feedbsck: Puching the trophy case has no etfect.
living
There doorway the
Template probs: open cad: 0,407, push OBJ: 9.145, take OR3: 0.39,
0.085, turn oBT: 0.062, throw ORd at 083: 0.027, put down 0BJ: 0.024, down: 0.022, put CBJ on OBJ: 0.015, put OBJ in caz: 0.019 object pross: windew: o, 686, smali: 0.314
Interactive ive objecte: elvish, strange, trophy, brass, wooden, rug, orientel, case, letsering, antiquity, sword, gothic, west, doer, largo, lastern, light, al: Action: push lenter
Score: 15
0bs: Desa: Living Roon. You vest, living
appears to Ihere doorway the shat, srophy case,
wooden with strange
gothic lettering in the conter of the roen, Abovs the trophy case hangs an sivist sword of great antiquity. Iaz: You are carrying A brats lantern (providing light) Feedback: Puening the brase lantern ieat notably helptu.. CBI: 0.403, push 0BJ: 0.164, west: 0.093, take
large oriental rug
Template probs: at 0B3: 0.020, gut dewn 0BJ: 0.025, put OBJ on 083: 0.02, somn: 0.019, put 0BJ in 0B3: 0.019 object prosst window: O, 685, snallr 0.315
0.039,
0BJ: 0.362, threw CaJ
Interactive cofects: brase, strange, trophy, elvist, rug, oriental, light, lettorine, anciquity, sword, gothic vest, door, large, woeden, lentera, case, al: Action: open wooden
Scorc: 15
Obs: Dese: Living Roon, You
wooden door with strange gethic letterira to the west, which appears to nailed shat, srophy case, and 12rge orsental zug in the center of the roen, Above the trophy case hangs an civist oword ot great antiquity, lav: You are carzying A brass lantern (providirg light) Peadback: The door cannot be opened,
living
There doorway the
Template probs: open CBI: 0.438, push OBd: 0.102, at 083: 0.025, put dewn 0BJ: 3,024, down: 0.02, put 083 in CBJ: 0.C18, put 0BJ on 0B3: 0.010 object prohs: viadew: 0.685, small: 0.315
0,081, west: 0.08,
0.062, throw 0B3
Interactive cafects: civich, strange, trophy, brase, Nooden, rug, oriental, case, letsoring, antiquity, sword, gothin, west, does, lazgs, lastern, light, al: Action: opan brass
Scorc: 15
Obs: Dese: Living Roon, You gothic lottering to the west, which appears to nailed shat, trophy case, and i8rgo orsental zug
living There doorway the woodan doos with strange
17

---

Published as a conference paper at ICLR 2020
in the centez of the zoon. Abova the trophy case hanga an alvish aword of great antiquity. lav: You are carzying a beass lantera (previding light) Peadback: You nust tell me how to do that to brass lantern. Template probs: open CBJ: 0,503, push OBJ: 0.129, wost: t.l,, take 0BJ: 0.053, tuzn on 083: 0.051, throN OBJ at down 082: 0.329, pat 063 in OBT: 0.021, put bad 0B3: 0.02, down: 0.009
cad: 0,042,
object preas: vindew: 0,677, small: 0.323
Interactive cofects: krass, strange, trophy, eivist, rug, oriental, Hight, letterine, anciquity, sword, gothic veat, doox, large, wooden, lantern, case, al: Action: push rug
Boore:
Obs: Dese: Living Room. You are in the living zoon. There is doerway to the east, woodon doos with strange appears
-eophy case, and cleaed Lrap your feet. Abova the trophy case tangs an alvish aword of great inciyy Inv: You are carsying brass lantern (providing 1ight) Peodback: With a great effert, the req is noved to one side of the zoom, zevealing the dusby cover of a closed Loap door.
letiecang the weat,
nailed
Template probs: open CBT: 0.582, push OBJ; 0.092, downs 0.076, take
0.065, puz down CBJ: 0.025, zero DB3: 0,025, weats 0.023, close C3I: 0.021, throw DBJ at CBJ: 0.019, put OBJ in DBJ: 3.016
dbject probas window: 0.674, snall: 0.326
Interactive dafecta: dasty, strange, trosby, covar, case, antiquity, lantern, lettering, sword, aothic, woodan elvish, brees, seat, rug, Lrap, dopr, light, al: Action: west
Scorc: 15
Obe:
Dese: Living Roon, You
There doorway the eaat, wooden doos with strange gethic lettering to the west, which appears to nailed shat, crophy case, and clased trap dooz at your feel. Abuve the Leophy case Langs an elvish sword of great ant iquley. Tav: You are varoying A beess lanteza (providing Light) Peedback: The dooz is nailed shut.
the living
Terplate probez open ChT: 0.679, push ORds 0.025, dowrs 0.057,
0.046, put down old: 0,022, corn 0B3: 0,019, waatz 0.017, close CaT: 0.015, theuw DDJ at CBJ: 0.015, put oBJ in OBJ: 2.015
Object probss windews 0.672, snall: 0.328 Irteract l've oblests: elvish, strange, trophy, brass, case, lectoring, intiguity, word, gothic, west, door, Leap, woudees lantern, light, all
Actions down
Segre: 15
Desos Livinng Roon You are
liwing room, There doorway to the east, weoden door with strange gethic lettering to the west, which agpears to led shat, a crophy case, and closed trap donr at
your foet. Above the Leophy case Langa an elvish sword of geeas and iquicy. Tav: You are carsying A beass lantern (providing light) Peedbacks The trap door is clased.
Template probes open cad: 0.63,
0,05, put down oad: 0.045, push ORT: 0.04, weats 0.023, Luzn on 3B3: 0.02, close OBJ: 0.C18, Lhrow 0Bd as OBJ: 0.018, pas Cad in 083: 0.015 object probs: vi vinee 0.657, small:: 0,342
0.687, take
Interact obs elvish, strange, trophy, brass, taxe, lettwring, ant iquiiyyy word, gothic, west, trap, door, woosen, lantern, light, sll Act on: open sword
Suore: 15
neses livinng You
doorway to the east, woaden door with sange gethic letinzing Lo the west, shich appears Lo nailed shat, crophy case, and closed Lrap dour at your teet, Above the trophy case bangs an elvish aword or grest antiquity. Inv: You are carrying & bress tell me how to do that to axord.
liwing room,
lantern (providing light) Peedback: You
Template probs: open CBT: 0.625, dewn: 0,075, take cad: 0.053, push OBJ: 0.03, pet come OB3: 0.049, wect: 0,028, turn on OBJ: 8.021, throw OBd at CB3: 9.02, clese OBJ: 0.018, pat OBJ in 036: 0.018
objeect pross: wi windows 0.859, small: 0,34 Interactive cafects: elvich, strange, trophy, brase, case, lettering, antiquity, sword, gothic, vest, trep, wooder, lantern, light, all Action: open cese
Score: 15
Obs:
Desa: Living Roon, You are the living roon. There is docrway to the cast, woodon doos with strange gethic lettering to the vest, which appears to be nailed shut, a trophy case, and closed trap coor at your feet, Above the trophy case hangs an elvish aword of grest ant iguity. Thv: You are carrying A bress lantern (providing 1ight) Peodback: opened.
Template probs: open CBT: 0.61, down: 0,081, take 0B3: 0.036, guen cad: 0,053, put coMe OB3: 0.349, weet: 0,029, turn on Ond: 0,025, throw ond at ca3: 8.02, put and in ORTs 0.019, close ond: 0.019
object prohs: wiadew: 03639, small: 0.341
Interactive oojects: elvich, strange, trophy, brase, case, lettering, antiquity, sword, gothic, trep, wooden, lantern, 1ight, Action: open docr
Score: 15
Obs: Desa: Living Roon, You are the living room. Taere is docrway to the cast, wooden dooc with strange gethic lettering to the vest, which sppears to be led shut, a tropty case, and closed trap door at your feet. Above the trophy case banes an sivish sword of great antiquity. Inv: You are carzying & brass wooden Template probs: open CB2: 0.552, down: 0,085, take 0ad: 0.062, put sown 0831
lantern (providing light) Feodback: Which door you
trap door?
pust OB3: 0.055, weet: 0.04, turn on 083: 0.026, throw 0BJ at oes: 0.026, put 080 in CBJ: 0,024, close cac: 0.021
object proas: vindew: 0.66, small; 0.33 Interactive dafects: elvish, strange, trophy, brass, cass, lettering, antiquity, sword, acthic, vest, trap, dcos, wooden, lantern, 1ight, zi1 Action: open all
Score: 15
0bs:
Desa: Living Roon, You are the living room. Inere is doorway to the east, wooden door with strange esthic lotterina to the west, which appears to be nailed shat, a traphy case, and clesed trap dooz at your feet. Anove the trephy case bangs an eivish sword of great antiquity. Inv: You are carzying & brass jeats with tapen".
maltiple Template probs: open CBT: 0.529, dewn: 0.086, put dovn QBJ: 0.555, take OB3: 063,, push cad: 0.059, west: 0,046, throw 05J at CBJ: 0.029, zuro on 0B3: 0.028, put CBJ in 083: 0.026, close OBJ: 0.021
lantera (providing light) Peedbacks
jbject proba: windew: 0.661, small: 0.335
Interactive cofects: elvish, strange, trophy, hrass, cass, lettering, antiquity, sword, acthio, trap, wooden, lantern, 1ight, zi1
18

---

Published as a conference paper at ICLR 2020
Action: down
acorc: 15
Dese: Living Roon, You
There doosway the wooden door with strange gethic letzerina to the west, which appears to nailed shat, zzopty case, and clased trap dooz at your reet. Above the trephy cass tangs an alvich aword of grest antiquity. Inv: You are carrying A beess lantezn (providing light) Peedback: The teap deor is clesed. 0.089, take car: 0.063, put
the living
o8d: 0.059, push cad: 0,049, west: 0,036, Luzn sn CBI: 0.024, Lheow DBJ as CBJ: 0.024, put C&J in 083: 0.021, sloow OBJ: 0.021 object prehs: window: 0.658, snall: 0.341
Template probs: open CBT: 0.569,
Interactive cofects: civish, strange, trophy, brass, case, lettering, antiquity, sword, gethic, vest, trop, dous, wouden, lantern, light, all Action: shake aothic
acorc: 15
Obs:
Dese: Living Roon, You
the living zoon, There doorway the east, wooden doos with strange gethic letineiin to the west, which agpears to neiled shat, crophy case, and closed trap dour at your feet. Above the Leophy case Langa an elvish sword of great ant iquicy. lav: You are carsying A beass lantern (providing light) Peodback: You dont have the weoden door.
Templace prubez open cBT: 0.62, push
0.085, Lake cBJ: 0.055, dew:: 0,053, wese 0,031, put dewn OBJ: 0,026, tuin on CBJ: n.025, theow DBJ as CBJ: 0.02, put 2B3 CBJ: 0.C13, close oaC: 0.014
Object prohs: windew: 0.668, small: 0.332
Intesact seceet wlvish, sizange, Lrophy, brass, wade, lettwring, and iquily, soord gothie, west, dour, trap, wooden, lantern, light, all Action: opan xest.
Suore: 15
neses livinng You
living room, There doorway to the east, woaden door with stnge gethic letiezang Lo the west, which appears Lo nailed shat, .rophy case, and cloaed Lrap dour at your feet. Abova the trophy case Langa an alvish sword of great ant incicyy Iav: You are carzying A beass lantern (providing light) Peedbbakk: The door cannot be opened.
Template probez open CBT: 0.648, push OBJ: 0.089, down 0.053, Lake
0.042, weat: 0.025, 0,025, put down. 0B3: 0.024, throw Cad at OB2: 0.021, put 0BJ in CBI: 0.02, close OBJ: 0.013
OBJ:
objectt probs: windows O. 0.665, small: 0,335
Interadiive ols jeclss wlvish, stzange, Lrophy, brass, caxe, lettering, antiqully, sword, gothie, weat, dous, trep, wooder, lantern, light, ictions open wnoden
Auore: 15
Beses Living
is doorway the west, wooden door with strange gethic letiezang Lo the west, which appears Lo nailed shat, crophy case, and cloaed Lrap doos at your feet. Above the trophy case hangs an elvish award of grest ant inutyyyy Tave You are carrying A bress Lantern (providing Light) Peetbarks The door cannet be opened.
You
the living room,
Template probs: open CBT: 0,63',, push OBJ: 0,001, down: 0.058, take OBJ: 0.05, put cowe OBJ: 0.324, weet: 0,024, turn on ond: 8.024, close old: 0,019, throw ond at ond: 0,017, put and in owT: 0.016
dbjject probos windowe 0.871, small: 0.325 Interactive oojecte: eivish, strange, trophy, brase, case, lettering, antiquity, sword, gothic, west, trep, wooden, lantern, light, Actions push worr
Score: 15
Obs: Desa: Living Roon, You are the living room. Iaere is docrway to the east, wooden dooc with strange gethic lettering to the vest, which appears to be not nei shut, cropty case, and closed trap door at your feet. Above the trophy case hangs an elvish award of great ant igy.ty. Tav: You are carrying A bress lantern (providing light) Peedback: Pushing the sword doesat sean to work.
Template probs: open CaT: 0.59), push OBd: 0.085, downs 0,073, take OBJ: 0.061, turr or OB3: 0,025, pat down DMd: 0,024, weets 0.824, close 092: 0.02, throw ORd at ond: 0,019, put ONd in oRd: 0.017
object pross: windew: 0.614, small: 0.326 Interactive onjecti: elvish, strange, trophy, brase, case, lettering, antiquity, sword, gothic, door, trap, wooden, lantern, light, z11 Action: open strang
Score: 15
Obs: Desa: Living Roon, You are the liwing room, Taere is docrway to the east, wooden door with strange gethic lettening to the vest, which agpears to be net
nailed led shit, crophy case, and closed trap door at your reet. Anove the trophy case tançs an sivish sword of great antiquity. Inv: You are carzying & brass lantern (providing Light) Feedback:
opened.
Template probs: open CRT: 0.614, push ORd: 0.085, dowr: 0.065, take
0.059, west: 0.024, turn on ONJ: 0,026, put dowr. 0B3: 0.021, close cad: 0.017, throw 0BJ at CBJ: 0.017, put CaJ in 083: 0.016
Object pross: windew: 0,676, smallr 0,323 Interactive cofects: elvish, strange, trophy, brass, cass, lettering, antiquity, sword, acthic, west, trap, sooden, lantern, 1ight, zil Action: down
Scorc: 15
0bs:
Deso: Living Roon You are the living room, There doorway to the east, wooden door with strange gethic letzering to the west, which appears to be nailed shat, a tropty case, and a clesed trap dooz at your reet, Azove the trophy case bangs an eivich aword or great antiquity. Inv: You are carzying & brass lantern (providing 1ight) Peadbacks
is elesed.
Tomplate probs: open CBT: 0.665, dean: 0.068, pesa cad: 0.052, take OBJ: 0.0c4, puz down cBd: 0,036, wost: 0,026, turn on OBJ: 0.02, throw OBJ at 085: 0.010, put 083 in 0B3: 0.017, close 0ac: 0.015 object prohs: vindew: 0.661, small: 0.335
Interactive cofects: civish, strange, trophy, brass, cass, lettering, antiquity, sword, gothic, west, trap, dcor, woosen, lantern, 1ight, sll Action: opan trap
Score: 15
Obs: Dese: Living Roon. You ara the living roon. There is doorway to the east, wooden doos with strange gethic letzerirg to the west, which appears to be nailed shat, a tropty case, and a req lying beride an open tzap door. Above the tropby case hangs an elvish awerd of great anciquity. Iav: You are carzying A brass lantern ipreviding light) Zeadback: dear roluctantly opens to reveal rickety staircasa desconding inzo darknoss.
19

---

Published as a conference paper at ICLR 2020
Templaca proba: downs 0,876, open 6B3: 0.046, take caT: 0.013, east: 0,013, close OBJ: 0.009, pat down c3T: 0.007, north: 0.006
Object pross: vindew: 0.604, small: 0.387, kitcto: 0,005, trap: 0.003 Interactivv daeetaa elvish, strange, trophy, braas, zug, staizoase, case, lettering, anciquity, suord, gothic west, does, trap, goodon, lantern, light, al: Action: dom
acore: 15
Obes: Dusas la pitch black, You are likely La be walen by grve- Iove You are eastying brass lanieza
Peodback: You have neved lnto a dark place. It is pitch black, You are likely to be eaten by a grue.
Template probs: take car: 0.142, zurn on OBJ: 0.16, open CBJ: 0.122, dewn: 0,117, clase OB3: 0,073, pat down 0.065, pash 0B3; 0,056, throw oBd 283: 0,054, souths 0.02 Object probs: laatert 0,076, letter: 0.073, vindow: 0.051, doozs 0.059, small: 0.032, gothie: 0.049, aword:
DBJ: 0.065,
0,049, ruz: 0,C3T, cropay: 0.335, west: 0.031
ive ela sotaa brass, dark, lantern, black, all,
Actions doan
Scare:
Obs: Dese: It is pitch black. You are likely to be gaten by grue. Inv: You aze cazzying A brass lenzern sbraight the infernel regions.
Peedhattkk The chesn probably
Templaca proba: Luzn on CBI: 0.167, cake OBJ: 0.132, open 0B3: 0.125, downs 0.124, close CBT: 0,076, push caI: 0.067, put down cBd: 0.066, throw 0B3 at 2Bd: 0,056, vest: 0,05, south: 0.024 jbjeett probes lanters 0.077, lellees 0.071, vindows 0.05, doors 0.059, small; 0.05), yothies 0.35, swards
0,049, rug: 0.C37, czophys 0.335, west: 0.031
Interactive cafecta: hrass, chasm, lantern, black, all, grus Actionn Lutn on lantse
Score: 40
Obes:
Dwee: Cellar You ase in dark and damp wellar with a nascuw passagenay leading north, and craulway to the south. on the west is the botton of a steap metal ramp which is unclinbabl.. There is a cuspiciausleodinn
individual Folding large bag, leaning againas one wall. Iee is arned with deadly Ley asoend the camp,
stileite. Invs You catrying brass lantern (providing light) Pawtbaak: but it is impossible, and you slids back down.
Template probss downs 0.246, open ond: 0,205, take ond: 0.122, close ands 0.1, push ORT: 0.082, turn or OlJ: 0,068, put down OBJ: 0,033, soutas 0,025, aaals 0.019, Cheow and at 0B3: 0.016 Object pross: vest: 0.08, snalls gethies 0,041, strangs 0.035, lanter: 0.034
windows 0,013, doors 0,353, rugt 0,057, letters 0.047, woodens 0,045,
Invesactive dbjecta; light, samp, large, beass, sciletio, ccaulvay, dack, nacsoa, lan.wen, bay, suspicioas, netal, all, wall, passagexay
Actions take dask
Boore:
Obes Beses Cellar You arw in dark and damp cellar with narcow passagmayy leading north, and crailway the south, On the vest the boston ot a steep metal ramp which is unaLimbable. There is
suspictous ooking individual, holding large bag, leaning agairs: one wall. Bee is arned with deadly stilette. Inve You are catrying A breas lantern (providing light) Powtbacks You cand be seriaus.
Template probs: down: 0.289,
oRd: 0.043, 092: 0,201, take 0ad: 0,103, clase 0BJ: 0.091, turn OB3: 0,066, pusa C3J: 0.022, south: 0.02), wests 0.018
0.065, put
Ob jeject probes west: 0,081, small; 0.076, windous 3.055, doocs 0.068, rugt 0.057, letlmes 0.048, woodens 0.045, gothict 0,042, strangg 0,336, larter: 0,035
Irteract ivve oolecta: light, ramp, large, brase, stiletto crawlway, dark, narrow, langern, bag, suspictous, netal, all, well, passagoxay Action: put out passeg Score: 40
Obst Desa: Cellar You are in cark and damp cel lar with narzow passagexey leading norch, and crawiway to the south, On the west is the bott of a steep metal reup which is undl inbable. There is suspiciouriosiim
insividual, bolding large bag, leaning agains; one wall. Be is armed with ceadly stiletto. The thief, finding nothing of value, left disgusted, lavs You are carrying brass lantern providing light) The thief, rirding noching of value, lest disguszed. Feedback: Pashing the chute ient notably heiptul. Someone carrying a large bag is casually leaning against one at the walls hers, Be does not speak, but st is clear from his aspect that the bag will be taker only over his dead body. Template probs: open CBI: 0.211, take 0BJ: 9.205, pust
0.188, south: 0.132, west: 0.105, close 032: 0,051, turn on OBJ: 0,026, go around CBJ: 0.02, easts 0.01%, sowns 0,018
object pross: snall: 0.112, window: 0.103, weats 0.075, doors 0.063, rugs 0.054, wooders 0.044, letter: 0.04, gethic: 0,037, stzong: 0.032, kizche: 0.031 Interactive oojects: light, ramp, large, beg, suspics netal, all, wall, passegexay Action: west Score: 40
stiletto, crawiway, wells, sark, naTEON, thiet, lanterr, chute
Obs: Desa: Collar You are in cark and camp collar with a narvow pascageкоy isading north, and crawiway the south, On the west is the botton at a steep metal remp which ie inbaibbable. There is a suspiciousiosking individual, holding lazge bag, leaning acainst ane vall. Be is armed with daadly stiletto. & quick thrust pinks your left
and blord starts to trickie braes lantern (prevising light) The thiet, tinding nothing ot value, lett disgusted. Feedback: You cont have the way.
lav: You carrying
Template probs: take CBT: 0,246, open OBJ: 0.211, pust OB3: 0.19, south: 0.091, west: 0.082, clase OB3: 0.039, 0.041, doans 0.326, around 083: 0.017,
0.012 object preast small: 0.097, window: 0.087, west: 0.078, dooz: 0.065, rug: 0.056, wooder: 0.045, lstter: 0.043, gothic: 0.038, strang: 0.934, larter: 0.032
on
Interactive cafecta: light, ramp, large, brass, silletto, crawlvay, dark, narrow, lanzern, bag, suspicieus, netal, all, wall, passagenay Action: south Score: 40
Obas Dese: Bast of Clasn You
the edge passaga coes norch, and the path you are on continues to the east. lavs You are carzying brass lantern (providing 1ight) Feedrock: The bracs lantorn i5 now on. East of Chaem You are cn the ezst edge of a chasn, the bocton of which cantes be seen. A nazzos passage goes norch, and the path you are on ccatinues the
the betten of which cannot
20

---

Published as a conference paper at ICLR 2020
Templaca probas put down 083: 0.132, take cBd: c,116, east: 0.1, open cad: 0.083, close OBJ: 0.39, ture on OBJ
0.07, throw 082 at OBJ: 0.061, wast: 0.059, push OBJ: p.057, dowe: 0.042
Object proas: vindew: 0,074, small: 0,057, lanter; 0.005, sword: 0,061, gochic: 0.029, worden: 0.032, lettor:
0.028 Interactive cofecta: light, brass, chasn, path, nazzow, lantern, passage, all Action: cast
0.03, bottle: 0,02, dooz: 0.025,
acore: 40
Obes
Duses Gallery This an ast lery. Rost of tie paintings have bown stolen by vandala ixceltional taste. The vandala laft through: either the north or wast exits. Forturacely, there is st1ll ona chance fer you to be a vandal, for the for Hall is a painting of unparalieied beauty. Inv: You are shanddanded, Peedback: Gallesy This
pain, have been stalen by vandals with excaptioral taste, The vandals left through either the north or veat exits. Partunately, thare La still ane chance for you to be vandal, for on the far wall is a painting of inporalicied beauty. CBT: 0.42, west: 0,377, Lake OBJ: 0,053, blew out OBJ: 0.039, eeat: 0.026, push OBJ: 0,02, down: 0,C15, around 0BJ; 0.01, north: 0.009, object probes windows 0.701, snall; 0.296
gallery. West
Template peobe:
0BJ: 0.005
Interact obfectt: art, painting, wall, all, paintings Action: opan sainti
Buore:
Obst Dese: Gallery This is an axt gallery. Rost of the paintings have been stolen by vandals with exceptional Laste. The vandala left througg wither the north west wxits. Fustimately Uiese is atlll one chanow you
is painting of unparalieled beauty. Inv: You aratyhanded. Feadback: You nust toll ne hox to that to painting.
vandal,
the far
Template prubes opea caI; 0.41, swat: 0.23, Lake O&J: 0.106, push 0B3; 0.033, bluw ou. CBJ: 0.032, dow: 0,023, turn on CBI: 0.015, east: 0,014, around caT: 0.01, south: 0.006
object probss windows 0.701,
0,238
Inve objectas art, painting, wall, all, paintings Actions take paiati
Seore: 40
Obss Dese: Gallery This an axt gailery. Kost, of the paintings have been stolen by vandals with exceptional taste. The vandala a lefe through wither the north or west exits. Tave You are carrylng A painting Peedbatkk Taken.
Terplate probss open caT: 0.492, push oRd: 0,296, take 0.036, weat: 0.033, ture or DBJ: 0,032, throw CBJ at ORd: 0,021, downs 0.015, clowe cBd: 0.013, put down oRd: 0,011, put OBd in OBT: 0.01
dbject probos windews 0.696, small: 0.302 Irteractivve ivv cofecte: art, painting, all, paintirgs
Actions open Boores
D.1 PENTARI TRANSCRIPT
ErelFred: exanine Cad 0.157, put 0BJ down 0.129, take CBJ 0.072, sit top of CBJ 0.067, liaten to 032 0.063 Obt:Fred: dirty 3.339, dacyer 0.280, sword 0.164, chest 0.092, all 0.071 Act: put dagger down, Rew 0, Score 9, Done salse, Value 0,339 awakened
beautiful day in Bostwin! zefreshed after restiul night's sicep. you've given your Company a veskend pass to get some R4R after your last geueling micsion. Masting precious little time your entire company took ott for the city proper to pack in as much driakine, carousing, brawling, veaching and other relaxing pursulta as siolds battle duty they're te precty much like evesy other bunch suldiers you' uu ever served with. à weekend off and what to do? Aftar the challenges of the past year when you were firs: premoted and assuned corman of Charlic Company you've nearly foreczton what free tine is and what you're supposed to do with it.
ray of sunlight poeking through window shade and rise
pessible 48
clite unit trained profeasionsis. ocr
Boward Sheenan, All Rights Reserved, Typa "Info" For Infernation About This Game or "Balp' for Instructions. Release 6 Sarial number 030205 / Inform ve.21 Library 6/10 Your quarters Bank cortainly docs have its privileges. Mhon you were your previous the lasury of private, suacious quartars fust outcids the bazracks thenseives. às furcher abase of your рoиeи your chest and bunk are not standard laace Htema from the Pentari millitasy supply cencer. You can see a poaccard here.
Pentari Incecactive Prequel To Firm. Light Copysight 1998
lowly Lisuterant you had to share cranged quezters with follow lieucemant casiqunent. commanding company your
affords
TmplFred: north 3.745, put CBJ down 0.127, take 032 0.126, sxamine cad 0.001, south 0.000
ObjaPredz postoard 0.817,
0.180, dagger 0.031, sall 0.001, Act: put. post down, Bow 0, Soore 0, Done False, Value 0.634
0,000
Obos The postaard is already here. TaglPewds nurth 3.785, put C&J down 0,108, Lake oad 0.105, exanine oad 0.002, south 0.C00 0.26', daeger 0.036, Act: north, Baw D, Soore o, Done False, Value 0.633
objirred: gosteard 0.620,
0.003, enerald 0.003
Ohes Charlis iiw Cumpany Congound Your cungany cungound condists of three large buindi border ing a large, open area where company fornationa precade the ordars of the day. To the weat the Company's 's aupply building while di reet ly abead you
your company's barracks which seen quieter than crypt. DIC to the east you hear Uhe distan: shouts ož anuthees Company dling Poor souls. FreiFreds north 3.915, put CBJ down 0,043, south: 0,0 take 083 0.010, exanina 0BJ 0.co5 PrjePredz sword 0.982, postoard 0.014, dagger 0.032, wll 0.001, enery 0.000 1.284
can
Aut: north, Rew 5,
Palas, Obas Charlis Company Bazracks the Arny of Bostwin ware to publish a nex edition of the soldier's hardbook, picture of this berracka should be included as the exapple other oonaniee should follow. The floors polishen to alaust blinding aline while each of the bunks lining both sides of this building perfect ly made. Every looker next each bank closed
looked, Nor a single acrap of poper, not one art icle of elothing, nay not eown dust pertiele looke out of place. The fowerfal snell of high guantities of pine cleanes nearly basne your mustrila. Damn, you zun good comapay. You can seu a Pentarian "raneporter here. Tegl enter 3.974, get in 0&J 0.022, south 0,304, put 0&J down 0.001, get up 0.co0 fojiit transport 0.389, aword 0.0:1, legenda 0.c01, poacoard 0.003, Lapesteie 0.000
Act entery Rew 3, Seore 0, Done Palse, Value 229
Obes The Charlie Company Barracks (in the Pentarian Transportwcl
TrglFredz city 0,976, get up 0.023, put 0Bd down. 0,0,3, take 082 0.030, eouth 0.000 Obj: Preds sword 0,996, postoard 0.002, cransport 3.001, blade 0.000, dirty 0.000
21

---

Published as a conference paper at ICLR 2020
: city, Bex 5, Boore 5, Done False, Value 5.665
Oboz
Your vision zwind as your sight of the barracks bluzs iato a nystad of chaotic colors. You feel a mement or stporianaation... of the Bank of Bostwin, where you're indoors! Mhat the hell is going on here...? Intrance Ball Your foststops esto loudly as you walk through this roon taking in the vest expanse or its cizo. The walis rise majestically upward and outwerd to fozm a huge roon which, cariocaly, is inpty. You cante help but wonder why such lazge room is arth
As you beçia to natoriclize yeur eyes grow vica as you realize you're ze not in tzont transporter usually takes you the city. You' not
outdoors,
needed tos the scrsight forvard task of acsepting quests and vicitors. Leada Le what must be a library shile Lo the nuzth you hardly help but noticw a hage, awni pircular narble acairoasa loading up to parta unknoan. 'Your score has 1est gona up by zive poinss.] Implered: cast 0,783, pet cBd doen 0.092, exemina OBJ 0.075, north 0.026, take CBJ 0.016 0.033
the
wido
fejdd aword 3.630, dauges 0.155, eneceld 0.097, all 0.051, Act: east, Bex 0, Boore 5, Done False, Valua 1.153
obs: Library This is a vast chancer Lined f1oor to celling with bookeases fili:d to capacity with bocks. You laven't seen auch collection of knowledge since your acadeny days. cursory glance at some beokabeivaa revaal several titles various topics of nacio, bistory, philoseshy, science and hundred other topaes. A fer large tables with thairs are in the center of the roon invit ing anyone to make use of these inpressive facilities. The kizard Horden is here stazing at you, the smalleat of amilea on his race. You can also see a copy of The Legends and Lere of Pentari here.
P cuvert 0,975,
old 0,008, put DRd down 0.007, push ORJ be 3Rd 0.005, Lake DBJ 0.002
j-ferred: leganda 0.700, sword C.176, floor 0.262, dargez 0.025, all 0.010 Act: covert, Row 0, Score 5, Dona Faiso, Value 2,394
Obos Spota in all the colosa of the rainbow danee in front of and instantly you as hard as any anemy ever has. Your hand shoots right for your aword in its seabbard on raflax and you grip the hil su hard your kaucklwa whilen. Coatle You are at the center of an int ion which runs east to weat as well as north to south. You can see The Dark EIC here. The Dark EIC walks around as if looking for sonething.
eyea as you feel yuurwelf di displaced... You
find yourself
the
Vanvevnav's
bear depreasion ion hit
Tegl south 9.936, west 0,017, put 0B2 down 0.C15, norch 0.010, wast 0.008 dbj::rod: avord 0.643, dacger 0.121, dirty 0.084, eneny 0,052, seal 0.043
Act south, Rew 5, Soore 5, Bone Palae, Value 5,257
thes Castle
slands jus inside the cealle's gale and would the place all visitora would flrst qo beforo being allovad entrance to the castle propar. quard atacion, long
This large open
abardoned, is here. A corridor screcches into the caxt le itself to the north, You can see a Nagical Smal here.
Tepl Freds hit
0,947, put ond down 0.029, north 0.014, exanine DBJ 0.005, south 0.005
Obj: seal 0,999, werd 0,000, sward 0.000, post 0.080, dagger 0.000
Ast: hit seal, Baw 10, Sooze 15, Done Palan, Value 10.024
Obes Your fist smashes the ward, shactering it into old of shimering dust. (Your acore hes just gone by ben points.]
ZeglEreds north 0.802, pat CBJ down 0.122, exanine caz 0.075, Lake caz 0.001, south 0.co0 Obj:Predz sword 3.850, postoard 0.06', blade 0,05), dirty 0,013, north 0.011 Act: north, Bew 5, Soore 15, hone False, Value 0.450
xolks around it looking something. Thel Preds north 3. 9,804, west 0,929, put and down 0,C24, souch 0,019, exant ne CRJ 0.017
Obss Castle You can see The Dark
here, The Dark
drjd-reed: sword 0.762, dagger 0,157, enerald 0.034, all 0.017, postcard 0,009 Act: north, Rew 3, Soore 15, Dore Faise, Value 1.362
obes Main Hall Tais once najestic room
visitors would cone relex and nees with the fornsl lord of the cest Ie in a soneuhat informal atnosghere. Several large confurtable couchiss are scallwowd about, dusty and altogether squalid. Many large tapestries still hang on the walls but are sorriblyy taded from wh huge Fireplace doninates the center of the roon
ege. large open erchways lead east and against the nozthern wall.
Trgl west 0,938, north 0.045, put 083 down 0,011, exanine 0B3 0.003, south 0.001 Pred: sword 335, tapestrne 0, D, eeat 0.652, all 0,033, wal" 0.024
Act: west, Rex 0, Score 15, Done Fales, Value 2.757
every type were stored here. Several tell racks probably speats while shorter ones nounted ed againe the wall stored variqua kinds swords, Other wall nounte, also engty, give you no ises what sort ot weapons may have been held by then. & large arcaway north is see jexel encrusted dagger here. ImpiFred: take cad 0,978, put QBJ cown 0.003, examine CaJ 0.007, cast 0.003, north 0.002 Fbo:Fred: all 0.867, dirty 0.034 cagger 0,031, other 0.024, emera.d 0.017
obss Amoryy Many death dealing weapone
partiallyy blokked by col lapsed stones and zubo You
Acts take all, Rew 5, Soore 20, hone talse, Value 5,752
obs: texel encrusted darger: Taken, (Your score has jest gone by tive poiats.]
Trgiered: east 0,926, exanine 0BJ 0.015, put down 0.029, north 0,007, take CBJ 0.003 Pred:red: sword 3.351, other 0.218, dagger 0-111, all 0.108, postcard 0.036
Act: cast, Rex 0, Score 20, Dane Falsc, Value 1.100
Dark Bit cown 0,034, Obs:Fred: dagger 0.212, all 0,141, wall 0.125, zword 0.112, tapestric 0.083
obst Main Hall You
Dark BiE xalks areand as looking for sometaing. 0.011, south 0,004
Tagi Fred: north 3,926, put
0.025, exanine
Act: north, Rev 3, Soore 20, Done Faise, Value 2,549
obs: Rain
at northern and of the main hall taking the panorana of what was once a majescic area whore perhaps treaties were signes, straregies lsid and plane roiled. you can see a Sarge Fireplace here.
by the Firoplace
TnelFred: enter 3.991, put CBJ down 0.003, examine caz 0.002, south 0.002, sit on cop of cBd 0.001 0,122, dagger 0.103, sword 0.062, ticor 0.020
objeFred: sireplacs 0.637,
Act: enter, Rew 5, Soore 20, Dore Faire, Value 5,182
Obs: The Main Ball, by the Pizeplace (in the Large Fireplacs) You can see a dirty scroll here.
IngiFred: take cad 0,914, get up 0.055, put 0BJ sown 0.019, exanine OBJ 0.007, city 0.c01 obfiFred: dixty 3.936, all 0.051, daeger 0.008, sword 0.col, emerald 0.001
Act: take all, Row 10, Score 20, Dene False, Value 10.
0bs: dicty seroll: Taken, (Your score has just gone up by ten points.l
IneiFred: get up 0.884, examine CBJ 0,04E, twoash 0,036, put 083 dowa 0,027, take OBJ 0.002
obiiFred: disty 3.535, sword 0.185, dagesr 0.19C, sll 0.085, emersid 0.004
Act: get up, Rox o, Score 20, Done Faise, Value 0.278
Obss Rain Hall, by the Firsplace You can see a larea Fireplace here.
ImplEred: south 3.938, cater 0.029, exanine DBJ 0.014, put 023 doon 0,014, take 0B3 0.002
abjiFreds disty 2.305, aword 0.230, all 0.182, dageer 0.154, rixepiace 039
Act: nouth, Rav 5, Soore 30, Dona Faiso, Value 1.176
0bs: Rain Hall
Implered: weat 0.859, notth 0.066, south 0.616, take 0B3 0.011, put QBJ down 009
obi:Fred: aword 0.435, dacter 0.253, 6irty 0.166, sil 0.103, 0.009 Act: west, Rex 0, Score J0, Dane Falsc, Value 2.303
22

---

Published as a conference paper at ICLR 2020
obas Azmory
InpiFred: north 3.994, put CBJ dawn 0.008, exanine caz 0.002, east 0.002, take CBJ 0.001 ObstFred: daggez 0,387, dizty 0.283, all 0.200, wall 0.050, award 0.043
Act: nosth, Ray 2, Score 30, Dona False, Value 5.338
Obs:
Treasusy in times past the vast sealth the bake would stored hare. Alss, with the castie taken and the pake gona co with hin went his riches. ha you remember your history lessons, you recall the cenealogy of the Dukes of Bostwin and theis love of cellecting treasures. You could only inagine what & rinsly crafted oaken chest is hare.
vast treasuras
conteined in
long
Teglfredz Iwoosh 0.972, south 0.010, north 0.026, put CBJ down 0.004, exanine 5B3 0,003
Ubi:Fred: daegoz 0.6,,, dizty 0.097, aword b.092, chest 0.080, all 0.043
Act: tuzosh, Row 10, Score (0, Danc Palse, Value 12.325
thas You leess et flames! The intensity of the firehall blasta the cheat open and niraculously, except foz slight discenfort of wormth, you are othervise unstfected. [Your score has jest gone up by zen points.) ZnglEred: Lake cad 0,822, put 0BJ 0.050, put oad in 0&J 0.022, close DBJ 0.019, south 0.015 0.035, amerald 0.012, snall 0.008, svard 0.007
of
fissbell naterialiaes
nusbece and flouda the unllow coon burst
obi:Fred: joval 0.904,
Act: Lake juwel, Rew 5, Score 45, bone Palse, ae 5.912
tbes You feul a suzge of power flow through you as you pick up the small ensrald.. Faken.. IFous score has just gona up by riva points.]
Teglfreds south D. 35754, close oad 0.085, put 0RJ in oad 0.056, Lake oad 0.027, pat 0&J down 0.024 objirrod: daggez 0.441, ensrald 0.228, aword 0.186, cleat 0.077, floor 0.025 Act: south, Ray D, Score 45, Dona Faiso, Value 1.267
abes Anmrry
082 down 0.c66, 0.043, south 0.015 Ob1:Fred: daegez 0.385, enarald 0.213, floor 0.135, aword 0.118, all 0.031
TueiFred:
0,755, exanine 05J 0.056,
Ast: put. dayger duwn, Rew 6, Suoce 45, Done Palse, Value 1.301 obes Dropped. The Dark EIf walks in.
Tagel Fredz enter 3.319,
0.294, exemine ond 0.272, put ond
0.019, west 0.013
Cbj.. venvwottes 0.409, dagyes 0.182, award 1,115, enerald 0.079, floue 0.061
Act: push vanvevnex to daeger, Raw 0, Scare 45, Dora Paise, Value 3.38E
thes I don't supgoss The Dark RIF would care for that. The Bark EIF senses your
of power and realizes you have the unerald lefs been looking fart bashee you the back of your head with the hill of his sword and steals the snall onerald fron you! Your vision swins and you can't guite your beazinga... You hear alst of shout ing nearby betueen Venveonee and lariliar sounding voice fall by an explosiio as the hair abands up on your bead! You doww your sword and allempt Lo
your bearings but you feel cust to behold two figures looked in sorceral combet! You can make out Morden murbl ing sonething under his beeath as be gestazea Lovards The Dask EIf and a belt of lightning hita Vanvemnse squacely in the cheat! Howliing in rage and pain, the Bark One returns the velley with fire strike which lights
displaced, Your vision struggles
clarity
you atterpt to look through the snoke
previously invigible defensive shield engagerent.... c. You look on bevilderment but know that you must sonething! Castle, Upper level This large roon hes caken on some new charasterics recently. ly... now the walls are covered wich blast narks of unknown origan fron flour to ceiling. Smoke welta throughout and it's about 10 degrwes warmer in here. box made of colid platinum sitting on the floor here. You can aieo see small eneraid, The Dark EIF and Horden here.
Murden! Essent ielly urburt, Norden laoke weskened fron this
Tage Preds hit 08J 0,966, undo 0BJ 0.010, push O&J to OBd 0.004, sest 0,004, exenine OBd 0,004
ObjeFred: wamvevmex 0,958, enerald 0.025, dagger 0.035, smail 9.003, eword 0,002
varveunew, Obess Wounded by the Risard Morden and dastsacted by the mortal conbat he angaged in, you are able to come up behind Vanvevnew and run hin through with your sword, loos at surprise crosses ais tace as be looks down and izes he is nortally wounded. Hee gives you blane look and atterpta to reach for your throst but coeen't quite nake it, crumpling to the eicor he breathes his last and dies. [Your score has just gone up by cen points.]
10, Score
se, Value 9.920
Tage Freds undo ond 0,205, examire ond 0.181, put Old on ond 0,176, put ORd down 0.166, posh ond to ONd 0.151 ObjeFred: dagger 0.239, tloor 0.181, emesald 0.165, box 0.158, all 0.101
undo degger, Rew o, Score 55, Done ralse, Valce -0.014 Ohess You can't see any auch thing-
TnglEred: undo oad 0,207, pat 0BJ en oad 0.174, examine 0B3 0.173, puch DBJ to CBJ 0,265, put bBd down 0.152 objePred: degger 0,224, floor 0.199, box 0.169, enerald 0.167, aword 0.120
Act: undo nox, Rew 0, Score 25, Done Faise, Valca -0.027
obs: You open the platinum box, revenling the Empirical Enerald.
Thgg Fred: put ORd on Gad 0,174, examine Ond 6616 pat ORJ down 0,151, push GBd to ond 0.147, pat OBd in DRJ 0.136
objeFreds dagger 0,240, tioor 9.196, emerald 0.166, 0.166, award 0.117
Act: pot box down, Rew 6, Score 55, Done Palse, Value -0.028
abst The platinum box is already here.
TreiFred: gut 0BJ down 0,171, exanine OBJ 0,161, pusa GBJ 082 0.150, GBJ 0BJ 0.142, take OBJ 0.119 orgePred: degger 0,299, floor 09199, box 0.166, enerald 0.165, aword 0,119
Act: clase dagost, Rew 0, Score 53, Dene Faise, Value -0.017
Obst You can't see any euch thing. TreiFred: examine ca2 0.159, put obg:Fred: dagger 0.240, tioor 0.178, emerald 0.163, box 0.157, all 0.129
down 0.158, pet 083 CBJ 0.149, push CBJ 0B3 0.147, take 0BJ 0.129
Act: take dagger, Rew o, Score 55, Done Faiso, Vaice -0.031 abst You can't see any such thing. TreiFred: pash
0B3 0.156, exanine OBJ 0.153, put 0BJ 0BJ 0.148, put 0B2 down 0.147, take 0BJ 0.134
Obg:Fred: degger 0,251, tloor 0.183, enerald 0.164, box 6.159, aword 0.126 Act: put dagger down, Rew o, Score 53, Deno Paise, Value -0.032
Obs: You can't see any such thing.
TreiFred: 0BJ
0.152, push cad to OBJ 0.150, put OBT dewn 0.148, exanina 0B3 0.148, put 0BJ in 0B3
0.139
Obs:Fred: dagger 0.236, eicor 0.18,, emeraid 0.166, box 0.159, sward 0.124
Act: put. bax on flcor, Bav o, Score 35, Dona False, Value -0.022
obs: The platinum box is aiready here.
TrgiFred: examine cad 0.161, pach oad OBJ 0.161, put OBJ down 9.160, take 0.136, put CBJ on 0BJ 0.131
ubf=Frod: daegez 0.233, floor 0.1, box 0.167, anerald 0.158, sward 0.125
Act: take dzggez, Row 0, Scors 25, Done Faise, Value -0.025 thing.
any meliFred: push cad to OBJ 0.170, exanine OBJ 0.166, put 023 down 0.159, put CBJ on 083 0.143, take OBJ 0.139 obt:Fred: dagger 0.235, #loor 0.181, box 0.165, emeroid 0.158, sll 0.127
tbss You c
Act: put flooz down, Bew 0, Score 55, Done False, Valca -0.034
to put the floor in? IngiFred: put 0BJ down 0.166, push OBJ to 082 0.164, examing 0B3 0.160, put CBJ on 0.138, take OBJ 0.130
obs: What do you
23

---

Published as a conference paper at ICLR 2020
obf:Fred: daggez 0.234, floor 0.186, bax 0.165, enerald 0.153, aword Act: put dagger in all, Rew 6, 5zoze 55, Done false, Value -0.021
Obst You can't see any cuch thing.
cad 0Bd 0.154, exanine 0Bd 0.153, ObjeFred: daggez 0.234, c10or 0.187, box 0.196, amerold 0.159,
TreiFred:
0.172,
down 0.149, 0B3 0.132
0.125
Act: put dagger dow,, Row 0, Score 55, Dono Falco, Volue -0.025
Obsz You can't see any such thing.
0.169, exanine abj-Fr degges 0.233, floos 0.184, bas 0.165, enecald 0.158, aword 0,127
TreiFred:
0.165, oad 0.160, CBI 0.152, OB3 0.134
Act: put sword en dacgez, Bow 0, Score 55, Dona False, Value -0.015
Obst You can't see any such thing.
0.188, wzamine Ob1:Fred: daggez 0.227, flooz 0.194, box 0.171, anorald 0.160, 0.123
TagiFredz
0.167, 0Bd 0.162, put down 0.157, 0B3 0.141
Act: put flooz en dapgez, BoN 0, scors 25, Done Tree, Value -0.025 Obez You can't L. see such thang.
24


Reviews:
[{"review_id": "rJxq_Sx0FB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a knowledge graph advantage actor critic (KG-A2C) model to allow an agent to do reinforcement learning in the interactive fiction game. Under the general framework of A2C, the core contribution of the paper is to apply a graph attention network on the knowledge graph to help learn better representation of the game state and reduce the action space. Experiments on Zork1 game environment are done to verify the effectiveness of the proposed method.\n\nOverall, this paper presents a novel contribution to reinforcement learning with augmented memory/world-state. However, I do have a few concerns regarding the baselines and other details. Given these clarifications and or comparisons in an author response, I would be willing to increase the score.\n\nPros:\n\n1, I like the idea of constructing the knowledge graph as the agent roll out. I think it is a better way to construct a structural representation of the world rather than assuming the agent gonna learn everything via single hidden state vector. It also permits more explainable policy in the future. Authors do make good progress along this line.\n\n2, The paper is well written and the design of the proposed new model seems technically reasonable.\n\nCons & Questions:\n\n1, The main concern I had is regarding to the baseline. I think it is more convincing to have a baseline which leverages the same entity extraction and template-action space. In particular, it should have the same model architecture except that it uses maybe a LSTM to decode the action rather than a GAT applied on knowledge graph. Note that the baseline I am referring to is different from the LSTM-A2C baseline reported in the paper as: (1) with entity extraction, although you may not get a graph mask, but you can still have a object-mask which also reduces the action space; (2) it is not clear to me that LSTM-A2C uses the same template-action mechanism as the KG-A2C, e.g., the valid action construction procedure described in section 4.1. Without such a baseline, it is hard to fully judge how helpful the knowledge graph is.\n\n2, How do you test the generalization of the proposed models? In particular, do you use different maps during training and testing? If the model is merely trained on one map as shown in figure 5, it may just memorizes it in the knowledge graph and overfit to this map. \n\n3, The details of the interaction fiction problem setup are sparse. It would be very helpful to explain what exactly the observations are in the example of Figure 2. For example, what are the game description, game feedback are in this case? \n\n4, In the caption of figure 1, “Solid lines represent gradient flow” is misleading. If I understood correctly, solid lines refer to the computation flow which has gradient back-propagated in the backward pass.\n\n5, Could you explain why KG-A2C converge slower than DRRN in figure 3?\n\n6, Do you think having a fully differentiable mechanism of building knowledge graph would help or not? Why?\n\n======================================================================================================\n\nMost of my concerns are addressed by the authors' response. I increased my score.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "r1ga-eIL5H", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper considers the problem of interactive fiction games in which an agent interacts with the world purely through natural language. The problem is challenging because one has to map natural language as observations to appropriate representations to partially infer the state space of the world, and actions are also defined in vast space of natural language sentences. One of the main contribution in the paper is to represent the partial observations of the world as a knowledge graph, and so as to efficiently infer appropriate actions.\n\nThe paper is very well written, especially the introduction section, demonstrating novelty in the context of fictional games literature, and showing good empirical results.\n\nHowever, I don't have any background in fictional games but dialog modeling. So it is hard for me to fairly assess how novel this work is. Ideas are simple and incremental, even if i rely upon literature overview provided by the authors in the related work section. Though, it can be effective.\n\nThe problem is challenging, and yet narrows down to fictional games. The proposed solution doesn't seem generic to be applied in other NLP or ML problems.\n\nAuthors argue that action space is super large even if generating sentences of length upto 5. Even though true, I think, this argument doesn't hold in the context of recent progress in NLP for problems like dialog modeling, where the action space of generating responses is even larger. I suggest the authors to relate their work to dialog models, as some of the ideas can be borrowed from there to simplify the solution for the considered problem setting.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BygA6oW15H", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper tackles the problem of developing agents to solve interactive fiction (IF) games. The authors propose an agent that builds a dynamic knowledge graph of each state from the textual observation provided by the games, while choosing actions from a template-based action space. While both these directions have been explored in prior work as pointed out, this paper combines them effectively to produce an agent that outperfoms existing methods on a benchmark of different IF games. \n\nPros:\n1. Writing is clear, method is easy to understand and design choices are clearly specified. \n2. Empirical results are good and presented on real IF games.\n\nCons:\n1. (minor) While the authors test their method on a suite of human-made IF games, it would also be great to have a study on synthetic cases like the Microsoft TextWorld environments, if only to see which aspects of the method are crucial to making the jump from synthetic to real IF games. \n\n\nOther comments:\n1. It looks like the knowledge graph is constructed for every state separately. The DRRN on the other hand incorporates more of the state history. From Figure 3(b), both DRRN and KG-A2C seem to do well - have you analyzed whether these methods are complementary? In general, it would be nice to have some more analysis on all the models across the different games rather than just Zork1.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper applies reinforcement learning to text adventure games by using knowledge graphs to constrain the action space. This is an exciting problem with relatively little work performed on it. Reviews agree that this is an interesting paper, well written, with good results. There are some concerns about novelty but general agreement that the paper should be accepted. I therefore recommend acceptance.
Decision: Accept (Poster)