Learning to Follow Instructions in Text-Based Games
Authors: Mathieu Tuli
Andrew C Li
Pashootan Vaezipoor
Toryn Q. Klassen
Scott Sanner
Sheila A. McIlraith
Venue: NeurIPS 2022 Accept (2022)
Keywords: reinforcement learning
linear temporal logic
textworld
text-based games
instruction following
natural language
instructions
GATA

Abstract: Text-based games present a unique class of sequential decision making problem in which agents interact with a partially observable, simulated environment via actions and observations conveyed through natural language. Such observations typically include instructions that, in a reinforcement learning (RL) setting, can directly or indirectly guide a player towards completing reward-worthy tasks. In this work, we study the ability of RL agents to follow such instructions. We conduct experiments that show that the performance of state-of-the-art text-based game agents is largely unaffected by the presence or absence of such instructions, and that these agents are typically unable to execute tasks to completion. To further study and address the task of instruction following, we equip RL agents with an internal structured representation of natural language instructions in the form of Linear Temporal Logic (LTL), a formal language that is increasingly used for temporally extended reward specification in RL. Our framework both supports and highlights the benefit of understanding the temporal semantics of instructions and in measuring progress towards achievement of such a temporally extended behaviour. Experiments with 500+ games in TextWorld demonstrate the superior performance of our approach.

Learning to Follow Instructions in Text-Based Games
Mathieu Tuli, Andrew C. Li, Pashootan Vaezipoor, Toryn Q. Klassen', Scott Sanner, Sheila A. Mcllraith'
University of Toronto, Toronto, Canada
Vector Institute for Artificial Intelligence, Toronto, Canada
Schwartz Reisman Institute for Technology and Society, Toronto, Canada {mathieutuli, andrewli, pashootan, toryn, sheila toronto. edu ssanner@mie utoronto... ca
Abstract
Text-based games present a unique class of sequential decision making problem in which agents interact with partially observable, simulated environment via actions and observations conveyed through natural language. Such observations typically include instructions that, in reinforcement learning (RL) setting, can directly indirectly guide a player towards completing reward-worthy tasks. In this work, we study the ability of RL agents to follow such instructions. We conduct experiments that show that the performance of state-of-the-art text-based game agents is largely unaffected by the presence or absence of such instructions, and that these agents are typically unable to execute tasks to completion. To further study and address equip RI. agents with internal structured
the task of instruction following, representation of natural language instructions in the form of Linear Temporal Logic (LTL), formal language that increasingly used for temporally extended reward specification in RL. Our framework both supports and highlights the benefit of understanding the temporal semantics of instructions and in measuring progress towards achievement of such a temporally extended behaviour. Experiments with 500+ games in TextWorld demonstrate the superior performance of our approach.
Introduction
Building AI agents that can understand natural language is an important and longstanding problem in AI. In recent years, instrumented text-based game (TBG) engines have served as compelling environments for studying a variety of tasks related to language understanding, affordance extraction. memory, and sequential decision making (e.g.. (e.g..(Côté al., 2018 Adhikari al. 2020 Liu al. 2022). They provide a simulated, partially observable environment where an agent can navigate and interact with environment objects, receiving observations and administering commands via natural language. TextWorld (Côté et al.) 2018)) is a TBG learning environment for training reinforcement learning (RL) agents. Successful play requires language understanding, effective navigation, memory, and an ability to follow instructions embedded within the text. Instructions may or may not be directly bound reward but can guide an RL agent towards completing tasks and collecting reward.
In this paper we study instruction following in text-based games and propose an approach that advances the previous state of the art. To this end, employ the state-of-the-art model-free TBG RI. agent called GATA (Graph Aided Transformer Agent) (Adhikari et al., (2020) that operates in the ToxlWorld environment. GATA has made significant advances in performance by augmenting TBG agents with long-term memory a critical component of effective game play. Despite GATA's improvement over previous baselines, our experiments (see Figure 1) show that GATA performance is largely unaffected by the presence or absence of instructions, leading us to conclude that GATA is not effectively following instructions. We also find that while GATA agents are able garner reward,
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Stripped Instructions
20 Games Level
100 Games
Level
20 Games Level
100 Games Level
14
18
1.6
hato
Text Nornalized Came Points
5500 2210
64
0.5
0.8
GATA-
GATAP
60
GATAP
GATAP
GATAP
GATAP
GATMP GATAP
0.0
Figure 1: Comparison of GATA performance when trained with instructions (GATA versus when instructions are stripped from environment observations (GATA --S). Agents were trained with 20 or 100 games, at increasing levels of task difficulty (level vs level 2). Note that normalized game point performance (solid blocks) and rate success (hashed blocks) are largely unchanged whether instructions are present or absent. Low success rate (i.e., task completion) rate is also seen in level 2.
they are not typically successful in completing tasks important vulnerability to the deployment of such techniques environments where partial completion of tasks can be unsafe.
To further study and address the task of instruction following, we equip GATA with an internal structured representation natural language instructions specified in Linear Temporal Logic (LTL)
(Pnueli 1977). formal language that is increasingly used for temporally extended goal and preference
specification in symbolic planning, and for reward specification, and other purposes
RL. (e.g..
Bacchus & Kabanza
2000 Baier & Mcllraith) 2006 [Baier et al., 2009) Patrizi et al. 2011 Camacho
McIIraith] 2019 Littman et 2017] Toro learte al. 201 Camacho et al.. 2019 Leon et
2020: Kuo et al. 2020; Vaezipoor et al. 2021). LTT. also provides mechanism to monitor progress
towards completion of instructions. Our framework both supports and highlights the benefit of
understanding the temporal semantics of instructions and in measuring progress towards achievement of temporally extended behaviour. We perform experiments that illustrate the superior performance of our TBG agent and its ability to follow instructions. Contributions of this work include:
Experiments that expose the lack of instruction following and low task completion rate in state-of-the f--the-art art TBG agent.
An approach to the study and deployment of instruction following TBG environments via exploitation of formal language: LTL. LTL provides well-defined semantics and supports a measure of progress towards satisfaction of instructions.
An augmentation to an existing state-of-the-arr architecture for TBGs to equip a TBG agent with esstuutioonfollowinn capabilities.
Comprehensive experiments and insights that study our and others' approaches to instruction following, and that highlight the superior performance of our proposed approach.
2
Background
In this section we introduce TexlWorld, the TBG engine that we use, together with the Cooking domain that we employ in our experiments. We also overview Linear Temporal Logic, which (as described in section we use in our approach as an internal representation for instructions.
2.1 Text-Based Games: TextWorld
Text-based games are partially observable multi-turn games where the environment and the player's action choices are represented textually. In this work, we use TextWorld (Côté et al. 2018) as our text- based game engine. A text- text-based game can be viewed as (discrete time) partially observable Markov decision process (POMDP) (5, T, A, O, Ω2 R, (Côté et al. 2018) where S is the environment's state space,
is the conditional transition probability between states 81+1 and given action O is the set of (partial) observations that the agent receives, 22 18,, ax-1) is the set of conditional observation probabilities, R S x A -> R is the reward function, and y E [0, 1] is the discount factor. An agent's goal is to learn some
action space, T [8,, at) where 81+1, 8t S and a,

---

optimal policy T- (a)o) (or a policy that conditions on historical observations or on some internal memory) that maximizes the expected discounted return. In this work, we focus on the choice-based variant of games, similar to previous works (Adhikari et al., 2020 Narasimhan et al. 2015). The
action space A is list of possible commands and at each time-step the game, the agent must select action ar E C2 from the current subset of permissible actions C₁ C A.
2.1.1
Environment Setting
We focus on the TextWorld Cooking domain, popularized by Adhikari et al. (2020) and Microsoft's s First World Problems: A Language and Reinforcement Learning Challenge (FTWP) (Trischler
et al. 2019). The game tasks agents with gathering and preparing various cooking ingredients described by an in-game recipe that to be found. Game points (rewards) are earned for each of (1) collecting required ingredient, (2) performing preparatory step (some cutting or cooking action) on an ingredient as required by the recipe, (3) preparing the meal once all of the ingredients have been prepared, and (4) eating the meal. The game's 's partial observations can contain instructions that guide the agent towards completion of tasks, but not all instructions correspond directly to rewards. The game first instructs the agent to examine cookbook, which elicits a recipe to be followed. The act examining the cookbook returns no reward, but following its recipe will return reward. See Appendix C for more details. Success is determined by whether the recipe is fully completed and eaten. Preparing ingredients can also involve collecting certain tools (e.g., a knife). The game may also involve navigation the agent may need to navigate to the kitchen or to find certain ingredients.
2.2 Linear Temporal Logic (LTL)
Linear Temporal Logic (LTL) (Pnueli 1977) is formal language propositional logical lan- guage with temporal modalities that can be used describe properties of trajectories. We will use LTL to specify instructions. LTI. formulas are constructed from propositional variables (e.g.. player-has-carronn connectives from propositional logic (e.g.. m, A), and two temporal operators: O (NEXT) and U (UNTIL). Formally, we define the syntax of L.TI. per Baier & Katoen (2008) as
6 == p x A 1 O U
where E P for some finite set of propositional symbols P. Satisfaction of LTL formula is determined by a sequence of truth assignments a = (ao, 01, 02.02....) for P. where p E o, iff proposition p P holds at time step i. Formally, satisfies yp at time i 0, denoted as (a, | g. under the following conditions:
(a, 2) = p iff E og where PEP P
() } (p iff (a, i) F y and (a, i) } &b
(, 1) p iff (a, 1) F
(,,)) F p U iff there exists j such that S j and
i0, = Opp iff (a,i+1)
(a,5) - vh, and (a, k)) F for all & E (i.5)
sequence a is then said to satisfy yp iff (a, F
Any LTL formula can be defined in terms of E P, (negation), A (and), O (NEXT), and U (UNTIL). From these operators, we can also define the Boolean operators (or) and - (implication), and the temporal operators D (ALWAYS) and & (EVENTUALLY), where (o, 0) } Dip if şo always holds in o, and (or, 0} = Oup if yo holds some point in a.
2.2.1 LTL Progression
LTL formulas also be progressed along sequence of truth assignments (Bacchus & Kabanza 2000 Toro Icarte et al.] 20188). In other words, as an agent acts in the environment, resulting truth assignments can be used to update the formula to reflect what has been satisfied. The updated formula would now reflect the parts of the original formula that are remaining to be satisfied or whether the formula has been violated/satisfied. The progression operator prog(a;, sp) is defined as follows.
Definition 2.1. For LTL formula y, truth assignment a, over P. and p E P, prog(oi, (p) is defined as
prog(a,, p) true
if E o.
prog(o,, (91 A (2)) prog(o (2) A prog( (22)
false prog(a;, mp) prog(a,, ()) prog(o,, NEXT up) = &
otherwise
prog(ø,, i UNTIL
prog(o (2) V (prog(a), vI) A A (NTIL)p))
3

---

In the context of TextWorid, the progression operator can be applied at every step in the episode to update the LTL instruction fed to the agent. To do so, it's necessary to have a labelling function that can indicate when propositions are true as the agent acts during an episode (e.g.. to detect that player-has wwa--hassaarraraa carrot is true when the player has the carrot). We discuss how this labelling occurs in section 4, and give an example of how progression works in Appendix D.
3 Following Instructions with GATA
In order to evaluate the effectiveness of state-of-the-ar text-based game agents at following instruc- tions, conducted experiments on the Cooking domain using the state-of-the-an model-free RL agent for TextWorld, GATA (Adhikari et al. 2020). GATA uses a transformer variant of the popular LSTM-DQN (Narasimhan et al. (215)) combined with dynamic belief graph that is updated during game-play. The aim is to use this belief graph as long-term memory to improve action selection by modelling the underlying game dynamics (Adhikari al.) 2020 Formally, given the POMDP. GATA attempts to learn some optimal policy x (alo,g) where is the belief graph.
While GATA's belief graph can capture goal relations (e.g. apple mepple-needs-uut cut), it turns out that agents trained to condition on observations and the GATA belief graph alone largely ignore in in-game instructions. We tested GATA agent on levels 1 and in the Cooking domain, after training on either the 20-game or 100-gat 100-game training set, and found that in none of those settings was the cookbook examined more than 15% of the time (3/20 testing games). In short, the GATA agent usually doesn't 't observe what the recipe is for the current game, meaning it has no way of knowing what the actual goal of the game is (except eventually from the rewards it gets and when the episode ends).
We further investigate how GATA agents fail to follow instructions by training these agents using modified game observations that have their instructions stripped (specifically, instructions directing the agent to examine the cookbook, the recipe text within the cookbook, and instructions to grab a knife if attempting cut an ingredient without first holding the knife were removed from observations). This has two effects: (1) the agent no longer receives text-based instructions about what the goal is or what it should do; and (2) GATA's belief state will no longer capture goal relations like 'needs'. The results of this experiment are in Figure I and demonstrate how GATA's performance remains largely unchanged. This suggests that GATA is (here at least) (a) not exploiting text- eext-based instructions that would lead it to success and (b) even not exploiting the goal goal-related relations in its own belief state.
The results in Figure 1 also show drop in GATA's performance when moving from level 1 to level 2 in the Cooking domain, where the games' complexity is increased by just one added ingredient preparation step in the recipes (see Table T for more details on the levels). GATA has difficulty in fully completing tasks on level games, where its success rate is roughly half that of its achieved normalized game points (only the latter metric was used by Adhikari et al. (2020))
Given these insights, we wish to further study and address instruction following TBGs. In the next section, we propose using LTL and demonstrate how existing work can be easily augmented.
4 An Approach to Following Instructions
We now investigate a mechanism for both studying and advancing the ability of an RL agent to follow
instructions. We do by translating instructions to an internal structured representation of language in the form of LTL, a formal language that is increasingly being used for reward specification in RL
agents ( (Vaezipoor et al. 2021 Leon et al. 2020 Kuo et al. 2020 Camacho et al. 2019 Toro Icarte jet al. 20T8b). We describe how to augment the GATA architecture with these LTL instructions and
how to monitor progress towards their completion.
4.1 Generating and Representing LTL Instructions for ToxlW
We use three types of LTL instructions for the Cooking domain. The first instruction identifies the need to examine the cookbook, and is the formula NEXT wewtbbok-is-examiinn simply states that the agent should examine the cookbook (i.e. -ookbook-is-exxmined = true) in the next step of the game. The second instruction is the actual recipe that gets elicited from the cookbook. We format this instruction to be order-invariant and incomplete. Order-invariance allows the agent to complete the instructions in any order, but is still constrained by any ordering
This instruction

---

that the TextWorld engine may enforce. "Incomplete" simply refers to the fact that not every single action required to complete the recipe is encoded (e.g.. grabbing knife before slicing a carrot, or opening the fridge). The agent must still learn to do these things to accomplish its tasks, but is not directly instructed to do so. Assuming the recipe requires that predicates P1.P2.... ........ Pn be true, the cookbook instructions are modelled as (EVENTUALLY p1 ) (EVENTUALLY) P2) A... (EVENTUALLY p.))....EVENNTULLL Pr))
For example, in the Cooking Domain, this instruction might be the conjunction
(EVENTUALLY apple-in-piae inlayyerr) A (EVENTUALLY meal-in inlpeer
(EVENTUALLY
The third and final type of instruction identifies the need to navigate to the kitchen. This instruction defined EVENTUALLY player-at- aihhetn This instruction will come prior the first two described above, but is only used in games with navigation (see Table
We build simple LTL translator that generates these instructions from the textual observations, similar to the goal generator used in [ii et al. (2022). TextWorld's observations are easily parsed to extract the goal information already contained within them, which we then formalize and keep track of using LTI. We provide examples of these observations and more details in Appendix E. Note that these observations are only used to generate the instruction itself, and subsequently LTL progression is used, with the GATA belief state as our labelling function, to monitor completion of instruction steps and update instructions that remain be addressed.
One possible criticism with such an LTL translator is its reliance on domain knowledge. While not the main focus of this paper, complementary research problem that has begun to be explored is to automatically translate natural language instructions to LTL (c.g.. Dzifcak et al.) 2209:] Finucane et al. 2010; Wang et al.) (2020). Traditionally, such approaches have required large corpora of training data or hard- rules, and were restricted to a specific domain. However, pretrained large language models such GPT-3 introduce the potential for general natural-language -o-ggugge-----LL..... translation
Finucanee
scheme with minimal domain-specific adaptation (Hahn et al. 2022 Huang et al. 2022 Brohan et al., 2022). We explore this prospect by applying GPT-3 to Worl (susecction 5.5
Finally, we note that in this work, GATA provides the domain-dependent vocabulary for describing while our LTL augmentation provides the domain- independent temporal modalities (i.e., NEXT, EVENTUALLY, etc.) and the logical connectives for composing those properties of state into the instructions use. In this way, technique is very generalizable, limited only by the recognizable properties of state (which in our case are provided by GATA) and instructions that can be extracted in game-play.
properties of state (e.g. carrot-is-chopper
4.2 LTL Augmented Rewards and Episode Termination
We can also define an augmented reward function Run (s, a, a. :p), where yp is an LTL formula, that rewards the agent for completing instructions. Given a labelling function L S x A + 2 that assigns truth values the propositions in P.
if prog(I(ss.)) a), sp) true
Rsn R(s, a) +
-1 if prog(L(s, a), (p) false
0 otherwise
In other words, bonus reward is given for every LTL instruction the agent satisfies and penalty is given if the agent fails to complete an instruction. We perform ablative study on the effect of this reward in Appendix H.3.2. We henceforth refer to this modified reward function as the LTL reward. The maximum bonus reward an agent receives is either 2 there no navigation task. or 3.
Further, because we wish to satisfy instructions, we can also use the instructions to modify episode termination. That is, if our LTL instruction is violated, have arrived in terminal state, even World has not indicated so. We perform an ablative study on the effect of this L.''--baed termination in Appendix H.3.2.
4.3 LTL-GATA Model Architecture
We build a similar model to GATA's original architecture, augmented to include the LTL encoding of instructions and their progression according to observed system state. We dub this model LTL-GATA, which we describe detail below. FFiuure depicts an episode step interaction of LTL-GATA with World and Figure 3 depicts the model itself. Additional details can be found in Appendix F.
5

---

you the "ooking modern app nich 3rd ed. ) and start reading: redie 4 gather all following ingredients and follow the directions prepare this meal ingredients: red potato: cirection: chop the red posoto, prepore meal"
examine cookbook
env
C
TextWorld
C.
(and Jeventually red, potatoiinnn daylee (and, (evenitually red, posato,, is scchopp)d) (and, (eventuatly meat prepared) leventallly meal cnnsumellf!
LTL-GATA
9w
Graph Updater
9:
Policy Model
Updater
Figure 2: An example of single step in episode of TextWorld. The game environment returns an observation and action candidate set Ct in response to action dt-1- In turn, the agent's graph updater (GATA) updates its belief graph It in response to both or and Dt-1. Next, Ot and or update the LTL instructions. fr is generated from o, after the cookbook is examined and thereafter Gr-1 is progressed yPr at each time step. The policy network selects action a from C₁ conditioned on oz. SPr, and Ir and the cycle repeats.
Graph Updater: We use the original GATA-GTP model (Adhikari et al. 2020).
which generates a
discrete belief graph list of triplets of the form (object, relationship. object). It is composed of two sub-components: (a) the belief state updater, which generates 9r from observation and the graph gr-1: and (b) the graph encoder, which encodes the current graph into vector as GE(g.) = gi E E for some latent dimension D. The graph encoder is a relational graph convolutional network (R- GCN) (Schlichtkrull et al., 2018) using basis regularization (Schlichtkrull et al., 2018) and highway connections (Srivastava et al. (2115). We refer the readers to Adhikar et al. (2020) for more details.
LTL Updater:
The LTL updater generates and progresses LTL instructions. LTL instructions defining the need to arrive at the kitchen and examine the cookbook are generated from the initial observation On- The subsequent instruction defining the recipe is generated from game observation Ot, as described in [subsection 4.1] when the action examine cookbook is executed at time t. For the truth assignments (i.e. the labelling function L). leverage GATA's highly accurate belief state from the graph updater. We use the Spot engine (Duret-Lutz et al.) 2016) to perform the progression.
Text Encoders: For encoding the action choices Ce, observations O,, as well as encoding the LTL instructions yt,, we use a simplified version the Transformer architecture presented by] Vaswani et al. (2017). This is the same architecture used by Adhikari et al. (2020). For LTL instructions, encode them directly as a string. For example, the LTL formula y = (EVENTUALLY p1) A (EVENTUALLY p2).
where P1 gepper-in-playe and p2 pepper- iinee-issuuttt has the string representation
str(yp) "eventually player_ has pepper and eventually pepper is cut
We format each predicate single token, and we show in Appendix H.3.1 that method is robust to predicate format. For some input string v E R of length f, the text encoder outputs a single vector TE(u) = v' RD' of dimension D, which is the same latent dimension the graph encoder.
Action Selector: The action selector is a 2-layer multi-layer perceptron (MLP). The encoded state vectors TE(o:) of E RR, TE({::) if E RDD and GE( = s E R.P are concatenated to form agent's final state representation 2, [ol; s/: s/] E RRDD contrast to Adhikari et al. (2020), we concatenate features rather than use the bi-directional iitention-bassed aggregator. This simplified the model's 's complexity and worked just as well experimentally. This vector is then repeated n. times and concatenated with the encoded actino choices C% E x where ne is the number of action choices. This input matrix is fed to the MLP which returns the vector of Q-values for each action qe E "o
Training. Formally, for belief state g and LTL instruction s, LTL-GATA aims to learn an optimal policy x (alo, g, sp). To learn this optimal policy, we implement Double DQN (DDQN) (Van Hasselt et al. 2016) with reward function and termination criteria discussed in subsection 4.2) We use prioritized experience replay buffer (Schaul al., 2016). Refer to Appendix G.2 for further details.
5 Experiments
Our experimental assessment was designed both to understand how well GATA was exploiting obser- vational instructions, discussed in secction 3] and to assess the instruction-followin performance

---

9'
Table 1: Cooking Levels
9
Graph Encoder
0
Text Encoder
Action Selector
Level
Recipe
Size
Rooms
Max
Score
Nood
[Grab.
Cut.
Cook]
Text Encoder
Text Encoder
Policy Model
0
1
3
(V.
Figure LTL-GATA's policy model. The model chooses
1
1
1
4
2
5
action E C2 conditioned on the state [olsspii g/. The action selector chooses de based on the predicted Q-values.
3
3
(xXxx)
of our proposed approach relative to this state of the art (not only in terms of game points but also successful completion). We additionally strove to assess features of our approach (such as monitoring instruction progress) that contributed to its performance, well as general challenges to text-based game playing that limited its performance (such as navigation)
5.1 Experimental Setup
Games. To have as fair comparison with Adhikari et al. (0200) possible, reused the sets of games they had generated. For the training games, they had created two sets: one set that contains 20 unique games per level and another that contains 100 unique games per level. Both the validation and testing sets have 20 unique games each per level. The levels we chose to use in our assessment are shown in Table Note that
omit Levels and 5. Level is augmentation to Level 3 that adds more ingredients; both GATA and LTL-GATA at this level suffer from the navigation issues we discuss later with respect to Level 3. As we wanted to focus on instruction following and not navigation, we omitted this level and chose to use Level 0 instead. Level 5 is simply a random combination of all levels,
assessment
is omitted for similar reasons.
(1) a batch size of 200 instead of 64 when training the 100 game set, (2) for level 3, Boltzmann action selection, and (3) we use Adam Kingma & Ba (015) with a learning rate 0.0003
We replicate all but three hyper- parameters from Adhikari et al. (2020): :
instead of RAdam AAdammLuse al. (2020) with learning rate of 0.001. These changes boosted performance for all models. See Appendix H.T T for more details.
Baselines. We compare against (1) TDQN (Adhikari et al. 2020), the transformer variant of the LSTM-DQN (Narasimhan et al. 2015) model, (2) GATA and (3) GATAP. GATA GATA's best performing model (GATA-COC) that uses continue graph-updater pre-trained using contrastive observation classification. GATAP D is a similarly performant model (GATA-GTP) that uses a discrete graph-updater pre- trained with ground-truth graphs from the FTWP dataset. Finally, note that found few issues with GATA's original code and have since fixed them (see Appendix H.4.1). 1). For comparison, we include the original paper GATA models, labelled as GATA, and GATAP.
Measuring performance. We measure performance using two metrics: normalized accumulated game points and game success rate. We report averaged results over 3 seeds for each experiment. Previous works only compared using the normalized accumulated game points; however, this may sometimes be misleading an agent could get 3/4 0.75 points on games but never actually succeed on any. In contrast, measuring the success rate alongside the normalized game points allows for a more complete analysis of the agent's 's ability to play and complete these games.
5.2 LTL-GATA Compared to Baselines
Consistently high performance with 20 training games. We see from Figure 4 that LTL-GATA exhibits consistently high performance across levels as compared to the baselines when trained on the 20 games set. In particular, LTL-GATA maintains its performance on level 2, where the game's slight increase in complexity causes large performance drop-offs in other methods. Our agent can easily complete the added task and maintain similar performance to the previous level 1.
I Our code for the experiments can found at htt.e//ttttt/tttt.tooo/oaaaaaaaa Atw.a//oiiiiiiiiiiiaaaaaaAAAAAA released under the open-source MIT License.
7

---

20 Games
Level
Level
Level
Level 3
1.6
Normalized Game Points
0.8
3.-
10OM
GAta
Garay
Gater
G47e"
OR-0452
cate
dara
(T--AAA
GATAG
CATaS
GaTe"
in on-
roon
cares
CATs
Gaze
GATA
Ut
Model
100 Games
Level
Level
Level
Level 3
Porta Success
(00)
Came Points
Nettoomalies
0022000 Patt
10U8
GATAS
GATAR
GATA
GATAP
D7-04x
1008
DATA
Sang
GATE
GRIR"
TOON
GATAS
aang
Bape
G4Ts>
Gata
moW
CATAS
care
date
GaMe
GaTA
Model
Figure 4: Testing scores across various levels and on both the 20 (top) and 100 (bottom) game training sets. We select the top-performing models (per seed) on the validation set during training and apply those models on the test set and report the average scores.
Examining the Gookbook
LTL Progression
Gamess Level1
100 Games Lovel 1
Games Lovid
150 cames Lovel
100 Games Level
Thes Noraaooo
Come Pertte
04
0p/64 140.044
05481
14Q-093
Model
Opean
optn
1644116
Model
(a) Forcing GATA to Examine the Cookbook
(b) Progression Ablation
Figure 5: (a) A comparison of GATAD performance when given the Option to examine the cookbook vs. when it is Required to examine the cookbook. (b) A comparison of LTL-GATA with (Prog) and without (NoProg) using LTL. progression.
Large performance gains with 100 training games. We see from Figure 4hthat LTL.-GATA gains considerable performance when trained on 100 games. With the added games, agent is exposed to more predicates and can now generalize better to the testing set. Future work may look at how to achieve this kind of generalization without having to expose our agent to more predicates.
Success rate and normalized game points. Looking the performance of GATA on level it becomes apparent why measuring the success is important. Although it achieves almost 0.4 normalized points, the actual success rate near for original GATA models, and 60% of the normalized points for the fixed models average across both training sets. In contrast, LTL-GATA exhibits high normalized points and success rate, where the average success rate across both training sets is 82% of the normalized points.
Competitive performance on level 3. Level 3 introduces the added challenge of navigation. LTL- GATA outperforms GATA in this level as well, but not to the degree of previous levels. Inspecting testing trajectories, it becomes evident that both LTL-GATA and GATA methods struggle with navigation in this level, and have difficulties even navigating to the kitchen in the first place. Exploring test time find items and rooms in an unknown environment is major challenge built into text- based games. Hypothetically. LTL could contribute to addressing this challenge. LTL could be used to dictate strategy and/or to simply track such exploration (e.g.. for remembering which rooms have been previously visited). LTL. might also be used to encode learned navigation instructions (e.g. "find the blue door, go through it, then turn right"). We do not pursue this vector of research here, but an interesting direction for future work.
8

---

5.3 Does LTL Progression Matter?
We show in Figure 5b that the use of progression is critical performance, where LTL-GATA without progression incurs large performance drop-off, dropping below the performance of the baselines as well. Without progression, the LTL instruction will not reflect the changes incurred by the agent's actions. This appears to confuse the agent considerably, demonstrated by its performance drop-off.
5.4 Forcing GATA to Examine the Cookbook
Because LTL-GATA is always tasked with examining the cookbook, we question whether a similar tasking for GATA improves performance. We experiment with GATA by forcing the agent to examine the cookbook on the first step of the episode. Forcing GATA examine the cookbook will elicit goal relations like (apple, needs, cut) in the belief state. We show however in in/Figure Sa that GATA does not improve when being given the cookbook. This shows that GATA cannot make use of the information elicited from the cookbook, continuing to ignore important instructions. Even with the presence of goal relations in its belief state, GATA fails properly attend to this information. This highlights the benefits a formalized representation of instructions used by LTL-GATA.
5.5 On Automatic Translation: Natural Language Instructions to LTL
While LTL-GATA relies on handcrafted LTL translator to provide initial instructions from text observations, we investigate the potential of automating this step using pretrained large language models. This is not central focus of the paper. Rather, we include this exploration as a proof of concept that the use of L.TL not barrier to broad deployment of the work presented here. To this end, we evaluate whether GPT-3 (Brown et al.) 2020) can few-shot learn to translate TextWorld observations to LTL, given only six examples and without additional training.
We experiment with two models of GPT-3 from OpenAI: Ada (the fastest model) and Da Vinci (the most powerful model). We perform few-shot translation by constructing prompts that contain six example translations, followed by the natural language observation to translate (the test case). The examples remain fixed for all test cases, and follow the form "NL: <natural language observation>. LTL: <ltl-formulas>". Our test case follows the form "NL: <natural language observation.. ITL.:", where the model must complete the prompt, thereby performing translation. We consider response that exactly matches the ground-truth L.TL. formula as absolutely correct, a response that is otherwise correct except for parentheses and spaces as almost correct, and all other responses incorrect. Further details and examples can be found in Appendix H.6.
Out of 234 test cases, Da Vinci translated 93.2% absolutely correctly and another 5.6% almost cor- rectly, with only 1.3% of examples incorrect. Da Vinci displayed an impressive ability to generalize to unseen adjectives (e.g. is-grilled) nouns (e.g. carrot), and compositions of formula. Unfortu- nately. the weaker model, Ada, translated all 100% of examples incorrectly. We found that Ada com- monly hallucinated new nonsensical words and predicates such as ingredient sssssss___________s__sss_______s falt_is_diced or banana_ aactll_ppo___eee chhp_chopppp_p inliiin_all in_yeee leading to erroneous translations.
6 Related Work
Text-based games. In this work we equip a text-based deep RL agent with formalized LTL in- structions, building on previous works that employed belief graphs for solving text-based games. Adhikari et al. (2020) focused on supervised (i.e. translation) and self-supervised learned mecha- nisms to construct such belief graphs, whereas Ammanabrolu & Hausknecht (2020); Yin & May (2019b): Ammanabrolu & RiedI (2019) employed rule-based methods. At larger scope, there a host of other works on playing text-based games using deep reinforcement learning (Hausknecht al. 2020 Zahavy et al. 2018 Jain al. 2020 Yin & May] 2019a). (uuan et al. (2018) used
count- based memory to shape the reward to improve in exploration and generalization in a simple domain. Narasimhan et al. (2015) and He et al. (2066) proposed variations of an -STM-based model, which the TDQN model, used in this work, is built from. In just published work, [iu et al. (2022) took a model-based approach, focusing on object-oriented dynamics. However, these works do not address the role and representation of instructions that defines our work. Kimura et al. (2021) does employ neuro-symin symrolic RL method using Logical Neural Networks. However, does not focus on instructions, operates over all logical facts of the environment, and is applied to simpler domain.

---

Instruction following and Linear Temporal Logic. Vaezipoor et al. (2021) trained an RL agent to follow various LTL instructions in both discrete and continuous action-space visual environments. They used R-GCNs to learn representations of the L.T. instructions and also employed L.T. pro- gression. Their model showed good generalization performance on similar and much larger unseen instructions than those observed during training. However, in contrast to the work presented here, they relied on ground-truth labelling functions and operated in fully observable settings, while we use GATA's learned belief graphs, in a partially observable setting, to evaluate the truth or falsity of propositions and to progress formulae. We further distinguish ourselves from this work by opting for training the LTL semantics end-to-end using transformer rather than R- R-GCN. Works using LTL for reward specification (Leon et al. 2020 Kuo et al., 2020 Camacho et al. 2019 Toro Icarte et al., 2018 8b Littman et al. 2017) advice (Toro Icarte al. (018a) in RL agents exist, however they do not focus on text-based environments nor partially observable ones.
7 Conclusion
We studied the ability of RL agents to follow instructions in text-based games using TextWorld. We conducted experiments to show how current state-of- ofrthe-art model-free agents largely fail to exploit instructions and do not typically complete prescribed tasks. We then showed how LTL can be used to construct internal structured representations for state augmentation that result in large performance improvements and more reliable instruction following and task completion. Experiments showed that monitoring instruction progress was critical to these gains. Our method inherits limitations in dealing with navigation and unseen games from prior work, but these concerns are somewhat orthogonal to our focus on instruction following.
Furthermore, we consider the broader impact of this work by relating the critical need for good instruction following in safety-oriented domains such as autonomous transport or health care. We would like to suggest that works towards building better language agents should also emphasize the importance of completing instructions. To illustrate, for an agent to help a person half-way across street, or to start but not finish medical operation, may be worse than for to do nothing at all. To that end, we have proposed using (game) success rate as a metric for future work, and demonstrated how LTL-GATA very successful in the games it plays, relative to the state-of-the-art Overall, we intend this paper to highlight the importance of studying instruction following in environments like TextWorld that act as a proxies to the general class of problems dealing with language understanding and human-machine interaction.
Finally, in follow-on work we would like explore more complex text-based games such as the Jericho environment (Hausknecht et al. 2020). These games involve a number of distinct challenges, including exploration, navigation, puzzle solving, language understanding. and instruction following, In this vein, we'd d like to see whether LTL can be exploited to capture (learned) domain-specific strategic advice, or memory to tackle both navigation and exploration challenges. We'd like to further explore seamless ways to exploit the merits of natural language together with the benefits afforded by the compositional syntax and semantics of formal languages such as LTL. To this end, further advancing our explorations translating natural language to LTL is of interest and import, for this and a diversity of other applications in and outside RI..
Acknowledgements
We thank the NeurlPS reviewers for their constructive feedback, and also the reviewers from the Wordplay: When Language Meets Games workshop at NAACL 2022, where a preliminary version of this paper appeared (Tuli et al. 2022). We gratefully acknowledge funding from the Natural Sciences and Engineering Research Council of Canada (NSERC), the Canada CIFAR AI Chairs Program, and Microsoft Research. Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring the Vector Institute for Artificial Intelligence (www. iectorinstitute ai si/partnnrs)) Finally, we thank the Schwartz Reisman Institute for Technology and Society for providing a rich multi-disciplinary research environment.
10

---

References
Adhikari, A., Yuan, X., Côté, M.-A., Zelinka, M., Rondeau, M.-A., Laroche, R., Poupart, P., Tang, J., Trischler, A., and Hamilton, W. Learning dynamic belief graphs to generalize on text-based games. Advances in Neural Information Processing Systems (NeurIPS2020), 33:3045-3057, 2020.
Ammanabrolu, P. and Hausknecht, M. J. Graph constrained reinforcement learning for natural language action spaces. In 8th International Conference on Learning Representations, JCLR 2020. Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. |tLptttss://oppneeee het./forun21dBLxxxx0:::: Bix6v0EtsH
Ammanabrolu, P. and Riedl, M. O. Playing text-adventure games with graph-based deep rein- forcement learning. In Burstein, J., Doran, C., and Solorio, T. ), Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
MAACL-LLT 2019, Minneapolis. MN. USA. June 2-7, 2019, Volume I (Long and Short Papers). pp. 3557-3565. Association for Computational Linguistics, 2019. doi: 10.1 (8653////119-///// 1358.. URL htttps://dde org/ 101.....1/1111111111100
Bacchus, F. and Kabanza, F. Using temporal logics to express search control knowledge for planning. Artificial Intelligence,
2000.
Baier, C. and Katoen, J. Principles of Model Checking. MIT Press, 2008
Baier, J. and Mcllraith, Planning with temporally extended goals using heuristic search. In Pro- ceedings of the 16th International Conference on Automated Planning and Scheduling (ICAPS06), pp. 342-345, June 2006.
Baier, J. A., Bacchus, F., and Mcllraith, S. A. A heuristic search approach to planning with temporally extended preferences. Artificial Intelligence, 173(5-6):593 618, 2009.
Brohan, A., Chebotar, Y., Finn, C., Hausman, K., Herzog, A., Ho, D., Ibarz, J., Irpan, A., Jang, E., Julian, R., al. Do can, as I say: Grounding language in robotic affordances. In 6th Annual Conference on Robol Learning, 2022.
Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., Agarwal, S., Herbert- Voss, A., Krueger, G., Henighan, T., Child, R., Ramesh, A., Ziegler, D., Wu, J., Winter, C., Hesse, C., Chen, M., Sigler, E., Litwin, M., Gray, S., Chess, B., Clark, J., Berner, C., MeCandlish, S., Radford, A., Sutskever, I., and Amodei, D. Language models are few-shot learners. Advances in Neural Information Processing Sys- tems, 33: 33:1877-1901,, 2020. URL, (457000666
Paper. . pdf
Camacho, A. and Mcllraith, S. A. Strong fully observable non-deterministic planning with LTL and LTT.ff goals. In Proceedings of the Twenty-Fighth International Joint Conference on Artificial Intelligence (IJCAI), pp. 5523-5531, 2019.
Camacho, A., Toro Icarte, R., Klassen, T. Q., Valenzano, R. A., and Mcllraith, S. LTL and beyond: Formal languages for reward function specification in reinforcement learning. In Proceedings of the 28th International Joint Conference on Artificial Intelligence (IJCAI), pp. 6065-6073, 2019.
Côté, M., Kádár, Á., Yuan, X., Kybartas, B., Barnes, T., Fine, E., Moore, J., Hausknecht, M. J., Asri, L. E., Adada, M., Tay, W., and Trischler, A. Textworld: A learning enviromment for text- based games. In Cazenave, T., Saffidine, A., and Sturtevant, N. R. (eds.), Computer Games 7th Workshop, CGW 2018, 8, Held Conjunction with the 27th International Conference on Artificial Intelligence, IJCAI 2018, Stockholm, Sweden, July 13, 2018, Revised Selected Pupers, volume 1017 of Communications Computer and Information Science, pp. 41-75. Springer, 2018. doi: 1111077978-3--00-24333
URL htttps://doll orgooo////// 10. 10.00.007/7788 9978 000-00330--------- -
Duret-Lutz, A., Lewkowicz, A., Fauchille, A., Michaud, T., Renault, E., and Xu, L. Spot 2.0 2.0-A Framework for LTL and w-AAutomaaa Manipulation. In Proceedings of the 14th International Symposium on Automated Technology for Verification and Analysis (ATVA), pp. 122-129. Springer, 2016.
11

---

Dzifcak, J., Scheutz, M., Baral, C., and Schermerhorn. P. What to do and how to do it: Translating natural language directives into temporal and dynamic logic representation for goal management and action execution. In Proceedings of the 2009 IEEE International Conference on Robotics and Automation (ICRA), pp. 4163-4168. IEEE, 2009.
Finucane, C., Jing, G., and Kress-Gazit, H. LTLMoP: Experimenting with language, temporal logic and robot control. In 2010 IEEE/RSJ International Conference on Intelligent Robots and Systems, pp. 1998-1993. IEEE, 2010.
Hahn, C., Schmitt, F., Tillman, J., Metzger, N., Siber, J., and Finkbeiner, B. Formal specifications from natural language. arXiv preprint arXiv:2206.0199) 2022.
Hausknecht, M., Ammanabrolu, P., Côté, M.-A., and Yuan, X. Interactive fiction games: A colossal
adventure. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 7903-7910, 2020.
He, J., Chen, J., He, X., Gao, J., Li, L., Deng, L., and Ostendorf, M. Deep reinforcement learning with natural language action space. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germuny, Volume Long Pupers. The Association for Computer Linguistics, 2016. doi: 10. (0.18653/v1p116-1155. URL https: org/10 10. 10w118666vv/// 18653/11/111111155
Huang, W., Abbeel, P., Pathak, D., and Mordatch, I. Language models as zero-shot planners: Extracting actionable knowledge for embodied agents. arXiv preprint 2iii:2201.002001 2022.
Jain, V., Fedus, W., Larochelle, H., Precup. D., and Bellemare, M. G. Algorithmic improvements for deep reinforcement learning applied to interactive fiction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp. 4338-4336, 2020.
Kimura, D., Ono, M., Chaudhury, S., Kohita, R., Wachi, A., Agravante, J., Tatsubori, M., Munawar, A., and Gray, A. Neuro-symbolic reinforcement learning with first-order logic. In Moens. M., Huang, X., Specia, L., and Yih, S. W. (eds.). Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pp. 3505-3511. Association for Computational Linguistics, 2021. doi: 10. 36633/1120221.emmlppp m88. 283. URL httpp:::/oo..org//...ooo.. 100110.11.111151//0//00
emm amnnp-main... www..ain....... 283
Kingma, D. P. and J. J. Adam: A method for stochastic optimization. Bengio, Y. and LeCun, Y. (eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA. USA, May 7-9, 2015, Conference Track Proceedings, 2015. httt:tttt//aiiii cetrxivv.orr/aae orgg/bs////11
6980
Kuo, Y., Katz, B., and Barbu, A. Encoding formulas as deep networks: Reinforcement learning for zero-shot execution of LTL. formulas. IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2020, Las Vegas, NV, USA, October 24, 2020 January 24, 2021, pp. IEFE, 2020. doi: 10. S00,IROOSSTT44..2009441111 URL /ttps:///////
1109/ 2020 9934125
Leon, B. G., Shanahan, M., and Belardinelli, F. Systematic Generalisation through Task Temporal Logic and Deep Reinforcement Learning. arXiv preprint arXiv:2006.0m 087677 2020.
Littman, M. L., Topeu, U., Fu, Jr., C. L. I., Wen, M., and MacGlashan, Environmenn:indeeeedeen task specifications via GLTL. CoRR, abs/1704.04341, 2017. URL http://arxxv.org/ /bs// 1704. 04341
Liu, G. Adhikari, A., Farahmand, A.m.. and Poupart, P. Learning object-oriented dynamics for planning from text. In ICLR 2022, April 2022.
URL https://wwis attps://www.mccrooott. www.omootooo.c.oom.mm. www/en-usmeeeeneeeeeeeeeeee paaneelllannnnggn from--ttet.rrttmtttttee
Liu, L., Jiang, H., He, P., Chen, W., Liu, X., Gao, J., and Han, J. On the variance of the adaptive learning rate and beyond. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.ne net, 2020.
12

---

Narasimhan, K., Kulkarni, T. D., and Barzilay, R. Language understanding for text-based games using deep reinforcement learning. In Märquez. L., Callison-Burchh C., Su. J., Pighin, D., and Marton, Y. (eds.), Proceedings of the 2015 Conference Empirical Methods in Natural
Language Processing, EMNLP 2015, Lishon, Portugal, September 17-21, 2015, pp. 1-11. The Association for Computational Linguistics, 2015. doi: 12116653/1//15510000 URL https org/o0..///. 111111101111//1111-111 /
//doi.org/1..1110
Patrizi, F., Lipovetzky, N., Giacomo, G. D., and Geffner, H. Computing infinite plans for LTL goals using a classical planner. In Proceedings of the 22nd International Joint Conference on Artificial
Intelligence (IJCAI), pp. IJCA111-334, URL httph:
ICCALAAAL, 2011. doi: 00.5591/978-1-5773.. 86-81 -
Pnueli, A. The temporal logic of programs. In Proceedings of the 18th IEEE Symposium on Foundations of Computer Science (FOCS), pp. 46-57. IEEE, 1977.
Schaul, T., Quan, J., Antonoglou, I., and Silver, D. Prioritized experience replay. In Bengio, Y. and
LeCun, Y. (eds.), ), 4th International Conference on Learning Representations, ICLR 2016, San Juan. Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016. URL http://atxii org/
abs/1511.0533
Schlichtkrull, M., Kipf, T. N., Bloem, P., Berg, R. V. d., Titov, I,, and Welling, M. Modeling relational data with graph convolutional networks. European Semantic Web Conference, pp. 593 607. Springer, 2018.
Srivastava, R. K., Greff, K., and Schmidhuber, Highway networks. arXiv:1505.00387 2015.
arXiv preprint
Toro Iearte, R., Klassen, T. Q., Valenzano, R., and Mcllraith, S. A. Advice-based exploration in model-based reinforcement learning. In Proceedings of the 31st Canadian Conference on Artificial Intelligence (CCAT). pp. 72-83. Springer, 2018a.
Toro Icarte, R., Klassen, T. Q., Valenzano, R., and Mcllraith, S. A. Teaching multiple tasks to an RL agent using LTL. In Proceedings of the 17th International Conference on Autonomous Agents and MultiAgent Systems, pp. 452-461, 2018b.
Trischler, A., Côté, M.-.,, and Lima, P. First TextWorld Problems, the competition: Us- ing text-based games to advance capabilities of Al agents. Microsoft Research Blog. URL
wtwpee/wwwwoot-oooooeeeeeeeoeeee 2019.
Tuli, M., Li, A., Vaezipoor, P., Klassen, T. Q., Sanner, S., and Mellraith, S. A. Instruction following in text-based games. In Wordplay: When Language Meels Games Workshop @ NAACL 2022, 2022.
Vaezipoor, P., Li, A. C., Toro Icarte, R. A., and Mcilraith, S. A. LTL2action: Generalizing LTL instructions for multi-task RL. In International Conference on Machine Learning, pp. 10497- 10508. PMLR, 2021.
Van Hasselt, H., Guez, A., and Silver, D. Deep reinforcement learning with double q- q-learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 30, 2016.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, E., and Polosukhin, I. Attention is all you need. In Advances in Neural Information Processing Systems, volume 30, 2017.
Wang. C., Ross, C., Kuo, Y., Katz, B., and Barbu, A. Learning natural- to LTL executable semantic parser for grounded robotics. Kober, J., Ramos, F,, and Tomlin, C. (eds.), 4th Conference on Robot Learning, CoRL. 2020, 16- 18 November 2020, Viriual Event / Cambridge, MA, USA, volume 155 of Proceedings of Machine Learning Research, pp. 1706-1718, PMLR, html
2020. URL http:tttpss:ccceeeeeeee
mlr wateass/v155/waaagg
Yin, X. and May, J. Comprehensible context-driven text game playing. In 2019 IEEE Conference on Games (CoG), pp. 1-8. IEEE, 2019a.
13

---

Yin, X. and May, J. Learn how to cook new recipe in new house: Using map familiarization, curriculum learning, and bandit feedback to learn families of text-based adventure games, arXiv preprint M0Kiv:/908.04777 2019b.
Yuan, X., Côté, M., Sordoni, A., Laroche, R., des Combes, R. T., Hausknecht, M. J., and Trischler, A. Counting to explore and generalize text- text-based games. CoRR, abs/1806. 11525, 2018. URL http://arxiv..oo/ aat///bbba//........
1806.886610 11525
Zahavy, T., Haroush, M., Merlis, N., Mankowitz, D. J., and Mannor, S. Learn what not to learn: Action elimination with deep reinforcement learning. In Bengio, S., Wallach, H. M., Larochelle, H., Grauman, K., Cesa-Bianchi, N., and Garnett, R. (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, prggeedinocee neurips. cttpcccppppa htce/pappr/2211200 http//aahhh/aaaaeeee http://///0/0///0000002222220060 attp:///acacttteeoaataaataaaaaaa
December 3-8, 2018, Montréal, Canada, pp. 3577, 2018. https: /
Abstract. html
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Limitations are described in both section Ilnndiootion
(c) Did you discuss any potential negative societal impacts of your work? [Yes] A broader impact is considered in section 7, and further discussion can be found in Appendix
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material URL)? [Yes] All code can be found at htttt:::// //github com
(b) Did you specify all the training details (c.g.. data splits, hyperparameters, how they were chosen)? [Yes] Hyper-parametert are discussed briefly in /section 5 and in full detail in Appendices G and H.
(c) Did you report error bars (e.g., with respect to the random seed after running ex- periments multiple times)? [Yes] Yes, error bars can be found in every results figure.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] This information can be found in Appendix H.2.
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] The footnotes in ssccionn 5) include links the original assets, and in-text citations give credit throughout the paper.
(b) Did you mention the license of the assets? links to the original assets.
[Yes] The footnotes in section 5 include
(c) Did you include any new assets either in the supplemental material or as URL? [Ys]] The only new asset in this work is our code, which we provide here: hitps //github.
LATAAAAAAAAA
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]
(c) Did you include the estimated hourly wage paid participants and the total amount spent on participant compensation? [N/A]
15


Reviews:
[{"review_id": "gy-_9cWdn5X", "reviewer": "Reviewer_erG7", "summary": "This paper proposes a new agent for Text Based Environment (TBG). The new agent is based on Linear Temporal Logic (LTL) and address the shortcomings of the existing SOTA agent GATA (Graph Aided Transformer Agent). Authors show that GATA agent is agnostic to instructions and are typically not successful in completing tasks in TBG. Authors incorporate formal representation of natural language instructions (via LTL) and this provides the agent with a measure of progress in task completion and helps the agent to complete tasks. ", "questions": "None", "limitations": "The main limitation of the proposed model is that it is not generalizable to many (and possibly unknown) domains. See also limitation above. ", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. Authors experimentally show that the SOTA agent GATA largely ignores initial instructions while playing the game. Further, authors show that the agent is ignorant of the final goal of the game (in this the agent does not know what it is suppose to cook as the final goal), the agent is solely driven by rewards alone. This is an interesting empirical insight by the paper. \n\n2. Authors develop and add a new component to the existing GATA architecture to overcome its limitation. The new component is a LTL Updater that generates LTL instructions from natural language observations and it also progresses the instructions.  \n\n\n\nWeaknesses:\n\n1. The use of formal structured language improves the performance but it is not generalisable to other domains. Authors in this case develop the specifications for cooking domain but if the agent were to be deployed for another domain new set of specifications would be required. LTL limits the possibility of deploying an agent to a real world setting where agent would have to deal with different tasks from varied domains. In such cases it is not possible to come up with specification for each possible domain as the number of such possibilities in principle is infinite. \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "gtBzqpqg7L9", "reviewer": "Reviewer_QS9K", "summary": "In this work, the authors first identity some important issues in recent text-based game agents that they tend to ignore instructions, these instructions can be dominant in solving text-based games. Then, the authors propose to equip agents with an internal structured representation of instructions, in the form of LTL, a formal language.\n\nDesigned for the cooking games in TextWorld, the authors use three types of instructions:\n1. the agent needs to examine the cookbook to know the goal of the current game (which changes in every game);\n2. the information written in the cookbook;\n3. go to the kitchen if the agent is spawned elsewhere. \n\nThe instructions can be used in multiple ways:\n1. (main use) as an additional input being fed into the model, it can guide the agent to act in a favorable manner.\n2. (additional use) to help shaping rewards, but as the authors discuss in Appendix H.4.2, this is not very helpful.\n\nThe proposed agent contains a minimal modification from prior work (the LPL updater), the authors show that equipped with the LPL module, their agent clearly outperforms baselines (GATA). \n\n\n", "questions": "1. Please see weakness 1.\n2. Related to Q1, it would be nice if the authors could discuss some potentials using large pre-trained language models (e.g., GPT-3) for instruction generation, then using similar large LMs (e.g., Codex) to translate natural language instructions into formal language (e.g., LTL).\n3. Regarding the first implementation issue discussed by the authors [HERE](https://github.com/xingdi-eric-yuan/GATA-public/blob/c1afc3c9ab38256f839b3e0ddf8243796df5bd77/dqn_memory_priortized_replay_buffer.py#L120-L123) and authors' code (experience_replay.py, L122-L129). I agree that length should be `n`, but the reward line should have `self._buffer[idx + j + i + 1]` instead of `self._buffer[idx + j + i]`? This is because that at a game step $t$, what's being pushed into the replay buffer is the reward $r_{t-1}$ resulted by the action at previous game step. Correct me if I'm wrong. \n\n", "limitations": "The authors discussed broader impact including potential societal concerns in the appendix. I agree and I do not see major issue regarding this. ", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- Strengths\n1. The authors identify issues in existing (top performing) RL agents, and aim to mitigate these issues. The motivation is clear.\n2. The authors show results that equipped with the proposed LPL module, GATA can obtain a performance gain on all the settings, this is especially obvious in level 2 games, where the tasks are the most difficult (ignoring navigation). The contribution is solid. \n2. The proposed system contains only minimal modification from prior work's model architecture, it is easy to compare the proposed method against baselines.\n3. The presentation if clear, the idea is easy to follow.\n4. Using success rate + game score as metric makes a lot of sense. \n5. The authors point out some implementation issues of prior works and provide bug fixes. \n\n- Weaknesses\n1. The authors fails to sufficiently discuss the weaknesses of LPL-aided agents. For instance, it seems that for every domain (e.g., cooking, crafting, navigation), the proposed system requires to some level human annotated instructions. To what extend do the proposed instructions scale or transfer? Given a new domain (either a simulation environment or some real world tasks), what are ways to design good instructions? I am not against leveraging humans as domain experts, but these discussion can help the readers to better understand the bigger picture.\n2. I suggest the authors to include numbers in the appendix, so future works can compare with their numbers. \n\n- Typo\n1. L192, Table 3 --> Figure 3\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "E4TkIqK4S4L", "reviewer": "Reviewer_HDhj", "summary": "This paper shows that the performance of the GATA agent is not affected by stripping the text instructions from the environment observations. This shows that the agent doesn’t consider the task instruction when making decisions. This paper proposes a method to improve task completion by exploiting progress monitoring in LTL. The resulting agent shows improvements in the Cooking domain at level 1 and 2. ", "questions": "- What’s the diversity of the converted LTL formulas? From the paper and the supplemental materials, the instructions are mainly concatenated with Eventually and Next. If the instructions are in simple forms, we may only need to co-train an event detector to match the events mentioned in the instructions without LTL formulas. Ablation of the LTL formula but keeping the event detector can show how much the text-based games require LTL understanding.\n- The experiments show most improvements in level 1 and 2, but not 3 because navigation is still an issue. Is it possible that the progress of navigation can be encoded as the LTL formulas so the same method can help navigation as well?", "limitations": "While a simple LTL translator works for the Cooking domain, for more complex text-based instructions, the efficacy of the proposed method is limited by the quality of the semantic parser that parses natural language sentences into LTL formulas. It would be useful if the authors can discuss the impact of parser quality.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n- This paper shows an analysis that reveals the weakness of the previously proposed model.\n- The experiments at different levels and ablations show how incorporating LTL in text-based games can help task execution.\n\nWeakness\n- Novelty: Exploiting LTL progress monitoring to improve RL agents’ performance has been demonstrated in non-text-based games [Vaezipoor 2021]. It is good that the same method is applied to the text-based games but this shows limited novelty of the proposed method. It will be helpful if the authors can discuss how LTL can affect the training or design of the agents in text-based games compared to non-text-based games.\n- Generalization: GATA was evaluated in 500+ games in TextWorld. However, the proposed method is only evaluated in the Cooking domain. It is unclear how the model can be extended to other domains. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "DZNRCtcA595", "reviewer": "Reviewer_CeYJ", "summary": "The paper proposes to incorporate Linear Temporal Logic (LTL) into a TextWorld game agent (GATA) to (1) provide augmented rewards based on subgoal completion, (2) encode text instruction. Authors find that GATA cannot properly attend to task instructions, and the performance does not drop even if instruction information is not given. Instead, LTL-GATA improves test results in Level 1-2 of TextWorld, but still limited in Level 3 by navigation.", "questions": "- In GATA, is a text instruction of the whole task always fed to the model (as part of text observation)? It seems so in Figure 2, just want to confirm. \n  - If so, is there any hypothesis why? \n  - Similarly, why GATA fails to attend to cookbook information?\n  - Is cookbook information same as text instruction of the whole task?\n- Line 37: Why \"partial completion of tasks can be unsafe\"?\n- Between line 179-180: what is $\\Phi$ in $R_{\\Phi}$? \n", "limitations": "The limitation about navigation is briefly discussed, but I think more limitations should be discussed (see weaknesses).", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n- The analysis of LTL failing to attend to instructions points to important problems in TextWorld agent design. \n- The introduction of LTL is novel in text games.\n\n**Weaknesses**:\n- The title is a bit misleading. Text-based games can mean synthetic games like TextWorld or harder human-designed games like Zork I, but apparently the method cannot apply to the latter. Maybe better to explicitly say \"TextWorld\" instead of \"Text-based Games\".\n\n- The formula of LTL seems very simple in TextWorld, and I'm not sure if the approach can be useful beyond simple, synthetic domains with clean logic. Experiments on more complex or fuzzy (e.g. more natural language) domains will make the work much stronger.\n\n- I feel the comparison of LTL-GATA and GATA is a bit unfair, as the former is given with more processed information (e.g. turning NL instructions into LTL synthetic language, and subtask completion maintenance) which assumes ground truth access to the \"logic\" of the instruction. It'd be better if the model can learn to turn raw game observations into the logic itself, instead of relying on gold annotations.\n\n- The introduction of LTL and LTL progression is formal but not very intuitive. Maybe combine some examples from Section 4.1 is better.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept