ALFWorld: Aligning Text and Embodied Environments for Interactive Learning
Authors: Mohit Shridhar
Xingdi Yuan
Marc-Alexandre Cote
Yonatan Bisk
Adam Trischler
Matthew Hausknecht
Venue: ICLR 2021 Poster (2021)
Keywords: Textworld
Text-based Games
Embodied Agents
Language Grounding
Generalization
Imitation Learning
ALFRED

Abstract: Given a simple request like Put a washed apple in the kitchen fridge, humans can reason in purely abstract terms by imagining action sequences and scoring their likelihood of success, prototypicality, and efficiency, all without moving a muscle. Once we see the kitchen in question, we can update our abstract plans to fit the scene. Embodied agents require the same abilities, but existing work does not yet provide the infrastructure necessary for both reasoning abstractly and executing concretely. We address this limitation by introducing ALFWorld, a simulator that enables agents to learn abstract, text-based policies in TextWorld (Côté et al., 2018) and then execute goals from the ALFRED benchmark (Shridhar et al., 2020) in a rich visual environment. ALFWorld enables the creation of a new BUTLER agent whose abstract knowledge, learned in TextWorld, corresponds directly to concrete, visually grounded actions. In turn, as we demonstrate empirically, this fosters better agent generalization than training only in the visually grounded environment. BUTLER’s simple, modular design factors the problem to allow researchers to focus on models for improving every piece of the pipeline (language understanding, planning, navigation, and visual scene understanding).

Published as a conference paper at ICLR 2021
ALFWORLD: ALIGNING TEXT AND EMBODIED ENVIRONMENTS FOR INTERACTIVE LEARNING
Mohit Shridhar'
Xingdi Yuan" Mare-Alexandre Côté 9
Yonatan Bisk
Adam Trischler
Matthew Hausknecht
'University of Washington
Microsoft Research, Montréal
CCamegie Mellon University
Microsoft Research
ALFWorld iitoorld.gitub..2. io
ABSTRACT
Given a simple request like Put a washed apple in the kitchen fridge, humans can reason in purely abstract terms by imagining action sequences and scoring their likelihood of success, prototypicality. and efficiency, all without moving a muscle. Once we see the kitchen in question, scene. Embodied agents require the same abilities, but existing work does not yet provide the infrastructure necessary for both reasoning abstractly and executing concretely. We address this limitation by introducing ALFWorld, simulator that enables agents learn abstract, text-based policies in Tex(World (Côté et al. 2018) and then execute goals from the ALFRED benchmark (Shridhar et al. (020) in a rich visual environment. ALFWorld enables the creation of a new BUTLER agent whose abstract knowledge, learned in TextWorld, corresponds directly concrete, visually grounded actions. In turn, as we demonstrate empirically, this fosters better agent generalization than training only in the visually grounded environment. BUTLER's 's simple, modular design factors the problem to allow researchers to focus models for improving every piece of the pipeline (language understanding, planning, navigation, and visual scene understanding).
update our abstract plans to fit the
INTRODUCTION
TextWorld
69
Consider helping friend prepare dinner in an unfamiliar house: when your friend asks you to clean and slice an apple for an appetizer, how would you approach the task? Intuitively, one could reason abstractly: (1) find an apple (2) wash the apple in the sink (3) put the clean apple on the cutting board (4) find knife (5) use the knife to slice the apple (6) put the slices in a bowl. Even in unfamiliar setting, abstract reasoning can help accomplish the goal by leveraging semantic priors. Priors like locations of objects - apples are commonly found in the kitchen along with implements for cleaning and slicing, object affordances a sink is useful for washing an apple unlike a refrigerator, pre-conditions - better to wash an apple before slicing it, rather than the con- verse. We hypothesize that, learning to solve tasks using abstract language, unconstrained by the particulars of the physical world, en- ables agents to complete embodied tasks in novel environments by leveraging the kinds of semantic priors that are exposed by abstrac- tion and interaction.
RoteteRight
LookDows
NavoAhead
You are is the miette of the zoon, Looking acound your you microwoee and osbines.
tor disisprable.
Pickap
goto the cakixet
You aczive at the cabinet.
the cabinet is closed.
opez
the cabinet is eapty.
goto
Rovualead Lock-oun
You urrive at tau stove. Bour the atove, yus pen, a pot, and winsbottie.
pas fou take the pas from the scove.
goto disingtable
Put
You aczive at the diningtable,
zou pet the pan on the dinsnptabie.
Figure 1: ALFWorld: Interactive aligned text and embodied worlds. An example with high-level text actions (left) and low-level physical actions (right).

---

Published as a conference paper at ICLR 2021
To test this hypothesis, we have created the novel ALFWorld framework, the first interactive, parallel environment that aligns text descriptions and commands with physically embodied robotic simulation. We build ALFWorld by extending two prior works: TextWorld (Côté al. 2018) engine for interactive text-based games, and ALFRED (Shridhar et al. 2020) a large scale dataset for vision- language instruction following in embodied environments. World provides two views of the same underlying world and two modes by which to interact with it: TextWorld an abstract, text-based environment, generates textual observations of the world and responds to high-level text actions; ALFRED, the embodied simulator, renders the world in high-dimensional images and responds to Unlike prior work instruction following (MacMahon et al. 2006 Anderson et al., 2018a), which typically uses a static corpus of cross- modal expert demonstrations, we argue that aligned parallel environments like ALFWorld offer distinet advantage: they allow agents to explore, interact, and learn in the abstract environment of language before encountering the complexities of the embodied environment.
low-level physical actions from robot (Figure
While fields such as robotic control use simulators like MuloCo (Toorrvv et al. 2012) to provide infinite data through interaction, there has been no analogous mechanism short of hiring a human around the clock - for providing linguistic feedback and annotations to an embodied agent. TextWorld addresses this discrepancy by providing programmatic and aligned linguistic signals during agent exploration. This facilitates the first work, to our knowledge, in which an embodied agent learns the meaning of complex multi-step policies, expressed in language, directly through interaction.
Empowered by the ALFWorld framework, we introduce BUTLER (Building Understanding in Textworld via Language for Embodied Reasoning), an agent that first learns to perform abstract tasks in TexrWorld using Imitation Learning (II.) and then transfers the learned policies to embodied tasks in ALFRED. When operating in the embodied world, BUTLER leverages the abstract understanding gained from WorlWorld to generate text-based actions; these serve as high-level subgoals that facilitate physical action generation by a low-level controller. Broadly, we find that BUTLER is capable of generalizing in a zero-shot manner from TextWorld to unseen embodied tasks and settings. Our results show that training first in the abstract text- based environment is not only 7x faster, but also yields better performance than training from scratch in the embodied world. These results lend credibility to the hypothesis that solving abstract language-bass tasks can help build priors that enable agents to generalize unfamiliar embodied environments.
Our contributions are as follows:
2
The first parallel interactive text-based and embodied environment. BUTLER architecture: An agent that learns high-level policies in language that transfer to low-level embodied executions, and whose modular components can be independently upgraded.
ALFWorld environment:
3
4
Generalization: We demonstrate empirically that BUTLER, trained in the abstract text domain, generalizes hetter to unseen embodied settings than agents trained from corpora of demonstrations or from scratch in the embodied world.
2 ALIGNING ALFRED AND TEXTWORLD
The ALFRED dataset (Shridhar et al.) 2020), set in the THOR simulator (Kolve et al. 2017). is a benchmark for learning to com-
Task type
train
Fusseen
Pick & Place
790
35
24
Examine Light
308
plete embodied household tasks using natural language instruc-
Clean & Place
650
27
31
tions and egocentric visual observations. As shown in Figure (right), ALFRED tasks pose challenging interaction and naviga- tion problems to agent in high-fidelity simulated environment. Tasks are annotated with a goal description that describes the objective (e.g., "put a pan the dining table").. We consider both template-basee and human-annow on goal specification can be found in Appendix H Agents observe the world through high-dimensional pixel images and interact using low-level action primitives: MOVEAHEAD, ROTR LOOOOWD PICKUP, PUT, OPEN, CLOSE, and TOGGLEON/OFF.
Heat & Place
459
16
23
Cool & Place
533
25
21
Pick Two & Place
813
24
17
3,553
140
Table 1: Six ALFRED task types with heldout seen and unseen evaluation sets.
goals; further details
Note: Throughout this work, for clarity of exposition, use ALFRED to refer both tasks and the grounded simulation environment, but rendering and physics are provided by THOR (Kolve al. 2017).
2

---

Published as a conference paper at ICLR 2021
The ALFRED dataset also includes crowdsourced language instructions like "turn around and walk over to the microwave" that explain how to complete a goal in a step-by-ster manner. We depart from the ALFRED challenge by omitting these step-by-step instructions and focusing the more diffcult problem of using only on goal descriptions specifying what needs to be achieved.
Our aligned ALFWorld framework adopts six ALFRED task-types (Table of various difficulty levels Tasks involve first finding a particular object, which often requires the agent to open and search receptacles like drawers or cabinets. Subsequently, all tasks other than Pick & Place require some interaction with the object such as heating (place object microwave and start it) or cleaning (wash object in sink). To complete the task, the object must be placed in the designated location.
Within each task category there significant variation: the embodied environment includes 120 rooms (30 kitchens, 30 bedrooms, 30 bathrooms, 30 living rooms), each dynamically populated with a set of portable objects (e.g.. apple, mug), and static receptacles (e.g., microwave, fridge). For each task type construct larger train set. well as seen and unseen validation evaluation sets: (1): seen consists of known task instances (task-type, object, receptacle, room] rooms seen during training, but with different instantiations of ob object locations, quantities, and visual appearances (e.g. two blue pencils on a shelf instead of three red pencils in a drawer seen in training). (2): unseen consists of new task instances with possibly known object-receptar always in unseen rooms with different receptacles and scene layouts than in training tasks.
pairs, but
The seen is designed measure in-distribution generalization, whereas the unseen set measures out-of-
cutt-of-distributee di-ddriituutin generalization. The scenes in ALFRED are visually diverse, so even the same task instance can lead to very distinct tasks, c.g., involving differently colored apples, shaped statues, or textured cabinets. For this reason, purely vision-based agents such as the unimodal baselines in Section 5.2 offte struggle to generalize to unseen environments and objects.
The World framework (Côté et al., 2018) procedurally generates text-based environments for training and evaluating language-basse based agents. In order to extend TextWorld to create text-based analogs of each ALFRED scene, we adopt common latent structure representing the state of the simulated world. ALFWorld uses PDDL- - Planning Domain Definition Language (McDermott et al. 1998) to describe each scene from ALFRED and to construct an equivalent text game using the TextWorld engine. The dynamics each game are defined by the PDDL domain (see Appendix [ additional details). Textual observations shown in Figure I are generated with templates sampled from a context-sensitive grammar designed for the ALFRED environments. For interaction, TextWorld environments use the following high-level actions:
goto (recep)
take (obj) from (recep) close (recep)
put (obj) in/on (recep)
open (recep) clean (obj) with (recep)
toggle (obj) (recep) cool (obj) with ( (recep)
heat [obj) with (recep)
where (obj) and (recep) correspond to objects and receptacles. Note that heat, cool, clean, and goto are high-level actions that correspond to several low-level embodied actions.
ALFWorld. in summary, is an cross-modal framework featuring a diversity of embodied tasks with analogous text-based counterparts. Since both components are fully interactive, agents may be trained in either the language embodied world and evaluated on heldout test tasks in either modality. We believe the equivalence between objects and interactions across modalities make FLFForrld an ideal framework for studying language grounding and cross modal learning.
3 INTRODUCING BUTLER: AN EMBODIED MULTI-TASK AGENT
We investigate learning in the abstract language modality before generalizing to the embodied setting. The BUTLER agent uses three components to span the language and embodied modalities: BUTLER:::RAIN the abstract text agent, ButleR:: the language state estimator, and BUTLER :ODY the low-level controller. An overview of BUTLER is shown in Figure 2 and each component described below.
"oo start with, we focus subset of the ALERED dataset for training and evaluation that excludes tasks involving slicing objects using portable container (e.g., bowls).
3

---

Published as a conference paper at ICLR 2021
put two plates the coffeetable
table cotfentable
State Frenntt MA
dieing plate
Text Agent (Rear)
(-0
stebar 1. and statte
Engåne
t=T
Pre-training
Controller
take plate from dintng table
Figure 2: BUTLER Agent consists of three modular components. 1) BUTLER:BRAIT text agent pre-trained with the TextWorld engine (indicated by the dashed yellow box) which simulates an abstract textual equivalent of the embodied world. When subsequently applied to embodied tasks, generates high-level actions that guide the controller. 2) BUTLER:
state estimator that translates, at each time step, the visual frame from the embodied world into textual observation or using a pre-trained Mask R-CNN detector. The generated observation or, the initial observation Og, and the task goal g are used by the text agent the to predict the next high-level action de- 3) BUTLER::BODY: a controller that translates the high-level text action ag into a sequence of one or more low-level embodied actions.
3.1 BUTLER::BRIN (TEXT AGENT) Op. ot: 9 at
BUTLER::BRAIN a novel text-based game
g
h
agent that generates high-level text actions in a token-by-token fashion akin to Natural Lan-
Obsarvation Queue
Encodar
Asgrogator
guage Generation (NLG) approaches for dia-
0p
logue (Sharma et al. (017)) and summarization GGhhraan al.J 2018). An overview of the agent's architecture is shown in Figure 3 At game step 1, the encoder takes the initial text ob- servation og, current observation or, and the goal description g as input and generates context- aware representation of the current observable game state. The observation ou explicitly lists all the navigable receptacles in the scene, and goal g is sampled from a set of language templates (see Appendix H). Since the games partially observable, the agent only has access to the observation describing the effects of its previous action and its present location. Therefore, we incorporate two memory mechanisms imbue the agent with history: (1) recurrent aggregator, adapted from Yuan et al. 2018), combines the encoded state with recurrent state h from the previous game step; (2) an observation queue feeds in the k most recent, unique textual observations. The decoder generates an action sentence a, token-by-token to interact with the game. The encoder and decoder are based on Transformer Seq2Seq model with pointer softmax mechanism (Gulcehre al., 2016). We leverage pre-trained BERT embeddings (Sanh et al.) (201)), and tie output embeddings with input embeddings (Press and Wolf 2016). The agent trained in an imitation learning setting with DAgger (Ross et al., 2011) using expert demonstrations. See Appendix A for complete details.
ht
Figure 3: BUTLER::BRIIN: The text agent takes the initial/current observations oglo, and goal to generate textual action a, token-by-token.
When solving a task, an agent might get stuck certain states due to various failures (e.g., action grammatically incorrect, wrong object name). The observation for a failed action does not contain any useful feedback, so fully deterministic actor tends to repeatedly produce the same incorrect action. To address this problem, during evaluation in both TextWorld and ALFRED, BUTLER:: BRAIN uses Beam Search (Reddy et al. 1977) to generate alternative action sentences in the event of a failed action. But otherwise greedily picks sequence of best words for efficiency. Note that Beam Search is not used to optimize over embodied interactions like prior work (Wang et al. 2019). but rather to simply improve the generated action sentence during failures.
3.2 BUTLER::VISON (STATE ESTIMATOR) 1 - oz
At test time, agents in the embodied world must operate purely from visual input. To this end, BUTLER::VISIN''S language state estimator functions as a captioning module that translates visual pre-trained Mask R-CNN detec-
observations v, into textual descriptions o, Specifically,
4

---

Published as a conference paper at ICLR 2021
tor (He et al. 2017) to identify objects in the visual frame. The detector is trained separately in supervised setting with random frames from ALFRED training scenes (see Appendix D). For each frame Ur, the detector generates N detections {(c) mm),,)).. ) (c2, m2)........ (CN, my )} where cn is the predicted object class, and m, is pixel-wise object mask. These detections are formatted into a sentence using template e-g., On table 1, you see mug 1, tomato 1, and tomato 2. To handle multiple instances of objects, each object is associated with a class On and number ID e.g., t tomato 1. Commands goto, open, and examine generate a list of detections, whereas all other commands generate affirmative responses if the action succeeds e.g., ar put mug on desk 1++1= : You put mug on desk 2, otherwise produce Nothing happens to indicate failures or state-change. See Appendix for full list of templates. While this work presents preliminary results with template-based descriptions, future work could generate more de- scriptive observations using pre-trained image-captioning models (Johnson et al. (0T6)), video-action captioning frameworks (Sun et al. 2019), or scene-graph parsers (Tang et al. 2020).
3.3 BUTLER::BODY (CONTROLLER) U/2 a: + {ā), .2,)...,á.))
The controller translates high-level action a, into a sequence of L low-level physical actions {ā,, (ā1.ā2..... aa......d. } that are executable in the embodied environment. The controller handles two types of commands: manipulation and navigation. For manipulation actions, we use the ALFRED API to interact with the simulator by providing an API action and a pixel-wise mask based on Mask R-CNN detections m, that was produced during state-estimation. For navigation commands, each episode is initialized with a pre-built grid-map of the scene, where each receptacle instance is associated with a receptacle class and an interaction viewpoint (x, y, 0, Φ) with x and representing the 2D position, 0 and é representing the agent's yaw rotation and camera tilt. The goto command invokes an A planner to find the shortest path between two viewpoints. The planner outputs a sequence of L displacements in terms of motion primitives: MOVEAHEAD, ROTATERIGHT, ROTATELEFT, LOokUP, and LooKDOwN, which are executed in an open-loop fashion via the ALFRED API. We note that a given pre-built grid-map of receptacle locations is strong prior assumption, but future work could incorporate existing models from the vision-language navigation literature (Anderson et al. 201 Sa Wang et al. 2019) for map-free navigation.
4 EXPERIMENTS
We design experiments to answer the following questions: (1) How important is an inteructive language environment versus static corpus? (2) Do policies learnt in TextWorld transfer to embodied environments? (3) Can policies generalize to human-annotated goals? (4) Does pre- pre-training in an abstract textual environment enable better generalization in the embodied world?
4.1 IMPORTANCE OF INTERACTIVE LANGUAGE
The first question addresses our core hypothesis that training agents in interactive TextWorld environ- ments leads to better generalization than training agents with static linguistic corpus. To test this hypothesis, we use DAgger (Ross et al. 2011) to train the BUTLER::: compare it against Seq2Seq, an identical agent trained with Behavior Cloning from an equivalently- sized corpus of expert demonstrations. The demonstrations come from the same expert policies and we control the number of episodes to ensure a fair comparison. Table 2 agents trained in TextWorld and subsequently evaluated in embodied environments in zero-shot manner. The agents are trained independently on individual tasks and also jointly on all six task types. For each task category, (from 8 random seeds); this is done separately for each split: seen and unseen. These best-performing agents then evaluated the heldout seen and unseen embodied ALFRED tasks. For embodied evaluations. we also report goal-condition success rates, metric proposed in ALFRED (Shridhar et al. 2020 to measure partial goal completion
agent in TextWorld and
presents results for
select the agent with best evaluation performance in TextWorld
FFor instance, the task "put a hot potato on the countertop" is composed of three goal-condee coal-conditions: (1) heating some object, (2) putting potato the countertop, (3) heating potato and putting it on the countertop. If the agent manages put any potato on the countertop, then 1 1/3 0.33 goal-conditions satisfied, and so on.
5

---

Published as a conference paper at ICLR 2021
TextWorkd
Seg2Sey
BUTLER
BUTLER-ORACE
Human Gouls
task-type
secn
unsoon
soen
unseen
5e00
unseen
socn
unseen
seen
unseen
Pick & Place
69
50
28 (28)
(17)
30 (30)
24424)
53 (53)
(31)
20 200)
(1
Examinc in Light
69
39
5 (13)
0 (6)
10 (26)
(15)
22 (41)
127
2 (9)
0 (8)
Clean & Place
(41)
(31)
239)39)
(36)
(81))
(29)
Heat & Place
888
83
(29)
(23)
17 (38)
16.(39)
60 (66)
672)
8 8
5(30)
& Place
(
(34)
5(21)
(33)
(49)
(44)
(266)
(34)
Pick Two & Place
54
65
(23)
0 (26)
15 (33)
8 (30)
32 (42)
29 (44)
6(1))
0 (6)
All Tasks
40
35
(15)
5 (14)
19 (31)
(0,)
37 (46)
(67)
(
(12))
Table 2: Zero-shot Domain Transfer. Left: Success percentages of the best BUTLER:BRAII agents evaluated purely in TextWorld. Mid-Left: Success percentages after zero-shot transfer embodied environments. Mid-Right: Success percentages of BUTLER with an oracle state-estimator and controller, an upper-bound. Right: Success percentages of BUTLER with numan-annotated goal descriptions, an additional source of generalization difficulty. All successes averaged across three evaluation runs. Goal-condition success rates (Shridhar et al.) 2020) are given in parentheses. The Seq2Seq baseline is trained in TextWorld from pre- recorded expert demonstrations using standard supervised learning. BUTLER is our main model using the Mask R-CNN detector and A navigator. BUTLER-ORACLA uses oracle state-estimator with ground-truth object detections and oracle controller that directly teleports between locations.
Comparing BUTLER to Seq2Seq, we see improved performance on all types of seen tasks and five of the seven types of unseen tasks, supporting the hypothesis that interactive TextWorld training is a key component in generalizing to unseen embodied tasks. Interactive language not only allows agents explore and build understanding of successful action patterns, but also recover from mistakes. Through trial-and-error the BUTLER agent learns task-guided heuristics, e.g., searching all the drawers in kitchen to look for a knife. As Table shows, these heuristics are subsequently more capable of generalizing to the embodied world. More details on World training and generalization performance can be found in Section 5.1
4.2 TRANSFERRING TO EMBODIED TASKS
Since TextWorld is abstraction of the embodied world, transferring between modalities involves overcoming domain gaps that are present in the real world but not in TextWorld. For example, the physical size of objects and receptacles must be respected while TextWorld will allow certain objects to be placed inside any receptacle, in the embodied world it might be impossible to put larger object into a small receptacle (e.g. large pot into microwave).
Subsequently, TextWWrld-trained agent's ability to solve embodied tasks is hindered by these domain gaps. So study the transferability of the text agent in isolation, introduce BUTLER-ORACLE in T an oracle variant of BUTLER which uses perfect statc-estimation, object-detection, and navigation. Despite these advantages, we nevertheless observe a notable drop in performance from TextWorld to BUTLER-ORACIE. This performance gap results from the domain gaps described above as well as misdetections from Mask R-CNN and navigation failures caused by collisions. Future work might address this issue by reducing the domain gap between the two environments, or performing additional fine-tuning the embodied setting.
The supplementary video contains qualitative examples of the BUTLER agent solving tasks in unseen environments. It showcases 3 successes and I failure of TextWorld-only agent trained on All Tasks. In "put a watch in the safe". the agent has never seen the Waatch''saff combination as a goal.
4.3 GENERALIZING TO HUMAN- JUMAN-ANNOTATED ANNOTATED GOALS
BUTLER is trained with templated language, but in realistic scenarios, goals are often posed with open-ended natural language. In Table we present Human Goals results of BUTLER evaluated on human-annotate ALFRED goals, which contain 66 unseen verbs (e.g.. "wash', 'grab', 'chill') and 189 unseen nouns (e.g., 'rag'. 'lotion', 'disc'; see Appendix H for full list). Surprisingly, we find non-trivial goal-compietion rate indicating that certain categories of task, such as pick and place, are quite generalizable to human language. While these preliminary results with natural language are encouraging, we expect future work could augment the templated language with synthetictto-real transfer methods (Marzoev et al. 2020) for better generalization.

---

Published as a conference paper at ICLR 2021
4.4 To PRETRAIN OR NOT TO PRETRAIN IN TEXTWORLD?
Given the domain gap between TextWorld and the embodied world, Why not elimi- nate this gap by training from scrutch in the embodied world? To answer this question, investigate three training strategies: (i) EMBODIED-ONLY. pure embodied training, (ii) TW-ONLY: pure TextWorld training fol- lowed by zero-shot embodied transfer and (iii) HYBRID training that switches between the two environments with 75% probability for TextWorld and 25% for embodied world. Table [able presents success rates for these agents trained and evaluated on All Tasks. All evaluations were conducted with an oracle state-estimator and controller. For fair comparison, each agent is trained for 50K episodes and the training speed is recorded for each strategy. We report peak performance for each split.
Training Strategy
Iminn (suce %6)
(suce %)
(suce %))
Irain speed
(eps/s)
EMRODIED ONLY
0.9
TW-ONLY
23.1
27.1
34.3
6.1
HYBBRID
11.9
2144
23.1
0.7
Table 3: Training Strategy Success. Trained on All Tasks for 50K episodes and evaluated in embodied scenes using an oracle
and controller.
Results indicate that TW-ONLY generalizes better to unseen environments while EMBODIED-ONLL quickly overfits to seen environments (even with a perfect object detector and teleport navigator). We hypothesize that the abstract TextWorld environment allows the agent to focus on quickly learning tasks without having to deal Ieccution-failuree and expert-failures caused by physical constraints inherent to embodied environments. TextWorld training is also 7x faster since it does not require running rendering or physics engine like in the embodied setting, See Section for more quantitative evaluations on the benefits of training in TextWorld.
5 ABLATIONS
We conduct ablation studies further investigate: (1) The generalization performance of BUT- LER::BRAN within TextWorld environments, (2) The ability of unimodal agents to learn directly through visual observations or action history, (3) The importance of various hyper-parameters and modeling choices for the performance of BUTLER::BRAIH
5.1 GENERALIZATION WITHIN TEXTWORLD
We train and evaluate BUTLER::BRIN in abstract TextWorld environments spanning the six tasks in Table as well as All Tasks. Similar to the zero-shot results presented in Section the All Tasks setting shows the extent to which single policy can learn and generalize the large set of 3,553 different tasks, but here without having to deal with failures from embodied execution.
We first experimented with training BUTLER::BRR BRAIN through reinforcement learning (RL) where the agent is rewarded after completing goal. Due to the infesibility of using candidate commands or command templates discussed in Section the RL agent had generate actions token-by-toker Since the probability of randomly stumbling upon a grammatically correct and contextually valid action is very low (7.02e-44 for sequence length 10), the RL agent struggled to make any meaningful progress towards the tasks.
After concluding that current reinforcement learning approaches were not successful on our set of training tasks, we turned to DAgger (Ross et al.) (011) assisted by rule-based expert (detailed in Appendix BUTLER:BRA. is trained for 100K episodes using data collected by interacting with the set of training games.
Results in Table show (i) Training success rate varies from 16-60% depending on the category of tasks, illustrating the challenge of solving hundreds to thousands of training tasks within each category. (ii) Transferring from training to heldout test games typically reduces performance, with the unseen rooms leading to the largest performance drops. Notable exceptions include heat and cool tasks where unseen performance exceeds training performance. (iii) Beam search is a key contributor to test performance; its ablation causes a performance drop of 21% on the seen split of All Tasks. (iv) (Seq2Seq) model
Further ablating the DAgger strategy and directly training a Sequence-to-Sequene
'For a fair comparison, all agents in Table 3 use batch-size of 10. THOR instances use 100MB Siite of GPU memory for rendering, whereas TexlWorld instances are CPU-only and thus much casier scale up.
7

---

Published as a conference paper at ICLR 2021
Pick & Place
Examine in
Clean & Place
Heat &
Cool &
Pick Two & Place
All Tasks
Light
Place
Place
BUTLER
34
61
46
59
39
22
37
44
39
60
81
74
46
60
100
27
29
24
40
37
BUTER
43
26
60
46
38
12
Seu2Sey
31
26
8
44
31
11
34
30
42
36
50
30
27
32
33
17
8
9
10
Table 4: Generalization within WextWorld environments: We independently train BUT- LER::BRAN on each type of TextWorld task and evaluate on heldout scenes of the same type. Respectively, tn/sn/un indicate success rate train/seen/unseen tasks. All and scores are computed using the random seeds (from 8 in total) producing the best final training score on each task type. BUTLER trained with DAgger and performs beam search during evaluation. Without beam search, BUTLER, decodes actions greedily and gets stuck repeating failed actions. Further removing DAgger and training the model in a Seq2Seq fashion leads worse generalization. Note that tn scores for BUTLER are lower than sn and un as they were computed without beam search.
with pre-recorded expert demonstrations causes a bigger performance drop of 30% on seen split of All Tasks. These results suggest that online interaction with the environment, as facilitated by DAgger learning and beam search, is essential for recovering from mistakes and sub-optimal behavior.
5.2 UNIMODAL BASELINES
Table 5 presents results for unimodal baseline comparisons to BUTLER. For all basclines, the action space and con- troller are fixed, but the state space is substituted with differ- ent modalities. To study the agents' capability of learning single policy that generalizes across various tasks, we train and evaluate on All Tasks. In VISION (RESNET18) the textual observation from the state-estimator is replaced with ResNet- 18 fc7 features (He et al.) (016) from the visual frame. Similarly. VISION (MCNN- pre-trained Mask R-CNN from the state-estimator to extract FPN layer features for the whole image. ACTION-ONLY acts without any visual or textual feedback. We report peak performance for each split.
Agent
(suce %)
seen
(suce %)
unseen
BUTLER
18.8
10.1
VISION (RESNETI8)
10.0
VISION (MCNN-FPN)
11.4
4.5
ACTION-ONLY
Table 5: Unimodal Baselines. Trained on All Tasks with 50K episodes and evaluated in the embodied environment.
uses the
The visual models tend to overfit to seen environments and generalize poorly to unfamiliar envi- ronments. Operating in text-space allows better transfer of policies without needing to learn state representations that are robust to visually diverse environments. The zero-performing ACTION-ONLY baseline indicates that memorizing action sequences is an infeasible strategy for agents.
5.3 MODEL ABLATIONS
#Observation
Init Obs
Recurrency
*Train Data
Figure 4 /illusttrtes more factors that affect the performance of : The three rows of plots show training curves, evaluation curves in seen and unseen set- tings, respectively. All experiments were trained and evaluated on All Tasks with 8
75
50
seen
80
random seeds.
In the first column, we show the effect of using different observation queue lengths k as described in Section [.I] in which size refers to not providing any obser- vation information to the agent. In the second column, we examine the effect of explicitly keeping the initial observation Og. which lists all the receptacles in the scene. Keeping the initial observation og accurately for unseen tasks, but may be unnecessary in seen environments. The third column sug- gests that the recurrent component in our aggregator is helpful in making history-based decisions
fabe
true fake
2-2
unseen
Irue
Figure 4: Model ablations on All Tasks. x-axis: 0 to 50k episodes; y-axis: normalized success from 0 to 75%.
facilitates the decoder to generate receptacle words more
8

---

Published as a conference paper at ICLR 2021
particularly in environments where keeping track of object locations is useful. Finally, in the fourth column, we see that using more training games can lead to better generalizability in both seen and unseen settings. Fewer training games achieve high training scores by quickly overfitting, which lead to zero evaluation scores.
6 RELATED WORK
The longstanding goal of grounding language learning in embodied settings (Bisk et al. (022)) has lead substantial work on interactive environments. ALFWorld extends that work with fully-interactin aligned environments that parallel textual interactions with photo-realistic renderings and physical interactions.
Interactive Text-Only Environments: We build on the work of text-based environments like Werl (Côté et al.) 2018) and Jericho (Hausknecht et al., 2020). While these environment allow for textual interactions, they are not grounded in visual or physical modalities.
Vision and language: While substantial work exists on vision-language representation learning e.g., MAttNet Yuu et al. 2018b)) CMN (Hu et al., (017)) VQA (Antol et al., 2015), CLEVR (Johnson
ct al. 2017). ViLBERT (Lu et al. 2019), they lack embodied or sequential decision making.
Embodied Language Learning: To address language learning in embodied domains, number of interactive environments have been proposed: Baby Chevalier-Boisvert et al. (2019), Room2Room (Anderson et al., 2018b), ALFRED (Shridhar et al. 2020), InteractiveQA (Gordon et al. 2018).
EmbodiedQA (Das et al. 2018). and NetHack (Küttler et al. 2020).. These environments use language to communicate instructions, goals, queries to the agent, but not fully-interacteo textual modality.
Language for State and Action Representation: Others have used language for more than just goal-specification. [Schwartz et al. (2019) use language as an intermediate state to learn policies in VizDoom. Similarly, Narasimhan al. (2018) and [Zhong et al. (2020) use language as an intermediate representation to transfer policies across different environments. Hu et al. (2019) use natural language instructor command low-level executor, and Jiang al. (2019) use language as an abstraction for hierarchical RL. However these works do not feature an interactive text environment for pre-training the agent an abstract textual space. Zhu et al. (2017) use high-level commands similar to ALFWWrlld to solve tasks in THOR with II and RL-finetuning methods, but the policy only generalizes small set of tasks due the vision-based state representation. Using symbolic representations for state and action is also an inherent characteristic of works in task-and-motion- planning Kaelbling and Lozano-Péres 2011 Konidaris et al. 2018) and symbolic planning (Asai and Fukunaga 2017).
World Models:
The concept of using TextWorld as a "game engine" to represent the world is broadly related to inverse graphics (Kulkarni et al. 2015) and inverse dynamics (Wu al. (017) where abstract visual or physical models are used for reasoning and future predictions. Similarly, some results in cognitive science suggest that humans language as cheaper alternative to sensorimotor
simulation (Banks et al. 2020; Dove 2014).
7 CONCLUSION
We introduced ALFWorld, the first interactive text environment with aligned embodied worlds. ALFWorld allows agents to explore, interact, and learn abstract polices in a textual environment. Pre-training our novel BUTLER agent TextWorld, we show zero-shot generalization embodied tasks in the ALFRED dataset. The results indicate that reasoning in textual space allows for better generalization to unseen tasks and also faster training, compared other modalities like vision.
BUTLER is designed with modular components which can be upgraded in future work. Examples include the template-based state-estimator and the A navigator which could be replaced with learned modules, enabling end-to-end training of the full pipeline. Another avenue of future work is to learn "textual dynamics models" through environment interactions, akin to vision- baseed world models (Ha and Schmidhuber 2018). Such models would facilitate construction of text-engines for new domains, without requiring access to symbolic state descriptions like PDDL. Overall, we are excited by the challenges posed by aligned text and embodied environments for better cross-modal learning.

---

Published as a conference paper at ICLR 2021
REFERENCES
Adhikari, A., Yuan, X., Côté. M.-A., Zelinka, M., Rondeau, M.-A., Laroche, R., Poupart, P., Tang, J., Trischler, A., and Hamilton, W. L. (2020). Learning dynamic belief graphs generalize on text-based games. In Neural Information Processing Systems (NeurIPS).
Ammanabrolu, P. and Hausknecht, M. (2020). Graph constrained reinforcement learning for natural language action spaces. In International Conference on Learning Representations.
Anderson, P.. Wu, Q.. Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid. I., Gould, S., and van den Hengel, A. (2018a). Vision-and aasoonand-langeee language navigation: Interpreting saually-grounder navigation instructions real environments. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Anderson, P., Wu, Q.. Teney, D., Bruce, J., Johnson, M., Sünderhauf, N., Reid. I., Gould, S., and van den Hengel, A. (2018b). Vision-and-Languaal Language Navigation: Interpreting visually-grounded navigation instructions real environments. Proceedings of the IEFE Conference on Computer Vision and Puttern Recognition (CVPR).
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L., and Parikh, D. (2015). VQA: Visual Question Answering. In International Conference on Computer Vision (ICCV).
Asai, M. and Fukunaga, A. (2017). Classical planning in deep latent space: Bridging the subsymbolic- symbolic boundary. arXiv preprint arXiv: arXiv:1705.0015::
Ba, L. J., Kiros, R., and Hinton,
(2016). Layer normalization. CoRR, abs/ 1077.66776666
Banks, B., Wingfield, C., and Connell, I.. (2020). Linguistic distributional knowledge and sensorimo- tor grounding both contribute to semantic category production.
Bisk, Y., Holtzman, A., Thomason, J., Andreas, J., Bengio, Y., Chai, J., Lapata, M., Lazaridou,
A., May, J., Nisnevich, A., Pinto, N., and Turian, J. (2020). Experience Grounds Language. In Empirical Methods in Natural Language Processing.
Chevalier-Boisvert, M., Bahdanau, D., Lahlou, S., Willems, L., Saharia, C., Nguyen, T. H., and Bengio, Y. (2019). BabyAI: First steps towards grounded language learning with a human in the loop. In International Conference on Learning Representations.
Cho, K., van Merriënboer, B., Gulcehre, C., Bahdanau, D., Bougares, F,, Schwenk, H., and Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).
Côté, M.-A., Kádár, A., Yuan, X., Kybartas, B., Barnes, T., Fine, E., Moore, J., Tao, R. Y., Hausknecht, M., Asri, L. E., Adada, M., Tay. W., and Trischler, A. (2018). Textworld: A learning environment for text-based games. CoRR, 1tt/1880.115333
Das, A., Datta, S., Gkioxari, G., Lee, S., Parikh, D., and Batra, D. (2018). Embodied Question Answering. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR).
Dove, G. (2014). Thinking in words: language as an embodied medium of thought. Topics in cognitive science, 6(3):371-389.
Gehrmann, S., Deng, Y., and Rush. A. (2018). Bottom-up abstractive summarization. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing.
Gordon, D., Kembhavi, A., Rastegari, M., Redmon, J., Fox, D., and Farhadi, A. (2018). Iqa: Visual question answering interactive environments. In Computer Vision and Pattern Recognition (CVPR), 2018 IEEE Conference on.
Gulcehre, C., Ahn, S., Nallapati, R., Zhou, B., and Bengio. Y. (2016). Pointing the unknown words. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers).
10

---

Published as a conference paper at ICLR 2021
Ha, D. and Schmidhuber, J. (2018). Recurrent world models facilitate policy evolution. In Advances in Neural Information Processing Systems 31.
Hausknecht, M. and Stone. P. (2015). Deep recurrent q-learning for partially observable mdps. arXiv preprint ar Xiv: Tt0.::500000577
Hausknecht, M. J., Ammanabrolu, P., Côté, M.-A., and Yuan, X. (2020). Interactive fiction games: A colossal adventure. In AAAI.
He, K., Gkioxari, G., Dollár, P., and Girshick, R. (2017). Mask r-cnn. In Proceedings of the IEEE international conference on computer vision.
He, K., Zhang, X., Ren, S., and Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition.
Helmert, M. (2006). The Fast Downward planning system. Journal of Artificial Intelligence Research.
Hu, H., Yarats, D., Gong. Q. Tian, Y., and Lewis, M. (2019). Hierarchical decision making by generating and following natural language instructions. In Advances in Neural Information Processing Systems.
Hu, R., Rohrbach, M., Andreas, J., Darrell, T., and Saenko, K. (2017). Modeling relationships in referential expressions with compositional modular networks. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition.
Jiang, Y., Gu, S. S., Murphy, K. P., and Finn, C. (2019). Language abstraction for hierarchical deep reinforcement learning. In Advances in Neural Information Processing Systems.
Johnson, J., Hariharan, B., van der Maaten, L., Fei-Fei, L., Zitnick, C. L., and Girshick. R. (2017).
Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR.
Johnson, J., Karpathy, A., and Fei-Fei, L. (2016). Densecap: Fully convolutional localization networks for dense captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition.
Kaelbling, L. P. and Lozano-Péree T. (2011). Hierarchical task and motion planning in the now. In 2011 IEEE International Conference on Robotics and Automation, pages 1470-1477. IEEE
Kingma, D. P. and Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint
Kolve, E., Mottaghi, R., Han, W., VanderBilt, E., Weihs, L.,, Herrasti, A., Gordon, D., Zhu, Y., Gupta, A. and Farhadi, A. (2017), Ai2-thor: An interactive 3d environment for visual ai. arXiv preprint arXiv: 1.X2:1172..55744,
Konidaris, G., Kaelbling, P.,, and Lozano-Perez, (2018). From skills symbols: Learning sym- bolic representations for abstract high-level planning. Journal of Artificial Intelligence Research, 61:215-289.
Kulkarni, T. D., Whitney, W. F., Kohli, P., and Tenenbaum, J. (2015). Deep convolutional inverse graphics network. In Advances in neural information processing systems.
Küttler, H., Nardelli, N., Miller, A. H., Raileanu, R., Selvatici, M., Grefenstette, E., and Rocktäschel, T. (2020). The nethack learning environment.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., and Zitnick, C. (2014). Microsoft coco: Common objects in context. In European conference computer vision.
Lu, J., Batra, D., Parikh, D., and Lee, S. (2019). Vilbert: Pretraining agsk-agnostic visiolinguistic representations for vision-and iision-and-langune languuge tasks. In Advances in Neural Information Processing Systems.
MacMahon, M., Stankiewicz, B., and Kuipers, B. (2006). Walk the talk: Connecting language, knowledge, and action in route instructions. Proceedings of the 21st National Conference on Artificial Intelligence (AAAI-2006).
11

---

Published as a conference paper at ICLR 2021
Marzoev, A., Madden, S., Kaashoek, M. F., Cafarella, M., and Andreas, J. (2020). Unnatural language processing: Bridging the gap between synthetic and natural language data. arXiv preprint 134455
McDermott, D., Ghallab, M., Howe, A., Knoblock, C., Ram, A., Veloso, M., Weld, D., and Wilkins, D. (1998). Pddl-the planning domain definition language.
Narasimhan, K., Barzilay, R., and Jaakkola, T. (2018), Grounding language for transfer in deep reinforcement learning. JAIR, 63(1):849 -74.
Press, O. and Wolf, I.. (2016). Using the output embedding to improve language models. arXiv preprint arXiv:1608.05859.
Reddy, D. R. et al. (1977). Speech understanding systems: A summary of results of the five-year research effort. Department of Computer Science. Camegie-Mell University, Pitsburgh, PA, 17.
Ross, S., Gordon, G., and Bagnell, D. (2011). A reduction of imitation learning and structured prediction no-regret online learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics.
Sanh, V., Debut, L., Chaumond, J., and Wolf, T. (2019). Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint wxXiv:1900.01000
Schwartz, E., Tennenholtz, G., Tessler, C., and Mannor, S. (2019). Language is power: Representing states using natural language in reinforcement learning.
Sharma, S., Asri, L. E., Schulz, H., and Zumer, J. (2017). Relevance of unsupervised metries in task- oriented dialogue for evaluating natural language generation. arXiv preprint /76600977099999
Shridhar, M., Thomason, J., Gordon, D., Bisk, Y., Han, W., Mottaghi, R., Zettlemoyer, L., and Fox, D. (2020). Alfred: A benchmark for interpreting grounded instructions for everyday tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10740-10749.
Sun, C., Myers, A., Vondrick, C., Murphy, K., and Schmid, C. (2019). Videobert: A joint model for video and language representation learning. In Proceedings of the IEEE International Conference on Computer Vision.
Tang, K., NIu, Y., Huang, J., Shi, J., and Zhang, H. (2020). Unbiased scene graph generation from biased training. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition.
Todorov, E., Erez, T., and Tassa, Y. (2012). Mujoco: A physics engine for model-based control. In 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems.
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L.,, Gomez, N., Kaiser, L. u., and Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems 30.
Wang, X., Huang, Q., Celikyilmaz, A., Gao, J., Shen, D., Wang, Y.-F., Wang, W. Y., and Zhang, L. (2019). Reinforced cross-modal matching and self-supervised imitation learning for vision- language navigation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Wu, J., Lu, E., Kohli, P., Freeman, B., and Tenenbaum, (2017). Learning see physics via visual de-animation. In Advances in Neural Information Processing Systems.
Yu, A. W., Dohan, D., Le, Q., Luong, T., Zhao, R., and Chen, K. (2018a). Fast and accurate reading comprehension by combining self-attention and convolution. In International Conference on Learning Representations.
Yu, L., Lin, Z., Shen. X., Yang. J., Lu, X., Bansal, M., and Berg. L. (2018b). Mattnet: Modular attention network for referring expression comprehension. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition.
12

---

Published as a conference paper at ICLR 2021
Yuan, X., Côté, M.-A., Sordoni, A., Laroche, R., Combes, R. T. d., Hausknecht, M., and Trischler, A. (2018). Counting to explore and generalize in text-based games. arXiv preprint arXiv: 11525.
Zhong, V., Rocktäschel, T., and Grefenstette, E. (2020). RTFM: Generalising to novel environment dynamics via reading. In ICLR.
Zhu, Y., Gordon, D., Kolve, E., Fox, D., Fei-Fei, L.., Gupta, A., Mottaghi, R,, and Farhadi, A. (2017).
Visual semantic planning using deep successor representations. In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017.
13

---

Published as a conference paper at ICLR 2021
A DETAILS OF BUTLER:BRAIN
In this section, we use or to denote text observation at game step i, to denote the goal description provided by game.
We use L to refer to a linear transformation and L' means it is followed by a non-linear activation function f. Brackets [ denote vector concatenation, denotes dement-wise multiplication.
A.1 OBSERVATION QUEUE
we utilize observation queue cache the text observations that have been seen recently. Since the initial observation On describes the high level layout of room, including receptacles present in the current game, we it visible to BUTLER::BRAIN at all game steps, regardless of the length of the observation queue. Specifically, the observation queue has extra space storing On- any game step, we first concatenate all cached observations the queue, then prepend the oo to form the input to the encoder. We find this helpful because it facilitates the pointer softmax mechanism in the decoder (described below) by guiding it to point to receptacle words the observation. An ablation study this is provided in Section
As mentioned in Section 3.1
A.2 ENCODER
We transformer- encoder, which consists of embedding layer and transformer block (Vaswani et al. (017)). Specifically, embeddings are initialized by pre-trained 768-dimensional BERT embeddings (Sanh al. 2019). The embeddings fixed during training all settings.
The transformer block consists of a stack of 5 convolutional layers, a self-attention layer, and a 2-layer MLP with ReLU non-linear activation function in between. In the block, each convolutional layer has 64 filters, each kernel's 's size is 5. In the self-attention layer, we use a block hidden size H of 64, as well single head attention mechanism. Layernorm (Ba et al., 2016) is applied after each component inside the block. Following standard transformer training, we add positional encodings into each block's input.
At every game step f, we use the same encoder to process text observation and goal description The resulting representations are ho, IR T xff and hg E R " where La, is the number of tokens in or, T, denotes the number of tokens in g. = 64 is the hidden size.
A.3 AGGREGATOR
We adopt the context-query attention mechanism from the question answering literature Yu et al. 2018a) aggregate the two representations ho, and hg-
Specifically, a tri-linear similarity function is used to compute the similarity between each token in he, with each token in hg. The similarity between i-th token in ho and j-th token in hy is thus computed by (omitting game step t for simplicity):
Sim W(h,, Nag ho: hgy )
(1)
where W is a trainable parameter in the tri-linear function. By applying the above computation for each h, and h, pair, get similarity matrix S IR L.
By computing the softmax of the similarity matrix along both dimensions (number of tokens in goal description T,y and number of tokens in observation I.,). get S, and Sg, respectively. The two representations are then aggregated by:
hog = [ho: Pi O Pih, o @Q]
P 5,
(2)
Q S,S ho:
where hog E TR LoXH is the aggregated observation representation.
14

---

Published as a conference paper at ICLR 2021
Next, a linear transformation projects the aggregated representations space with size H = 64:
hog L tanh 1 (hog).
To incorporate history, we use a recurrent neural network. Specifically, we use a GRU (Cho et al. 2014):
hRNN Mean (hogn
(4)
h4 GRU(hRN, ht-1),
in which, the mean pooling is performed along the dimension of number of tokens, i.e., IRNN R he-1 is the output of the GRU cell at game step 1.
A.4 DECODER
Our decoder consists of an embedding layer, transformer block and a pointer softmax mechanism (Gulcehre et al. 2016). We first obtain the source representation by concatenating hog and h4-
resulting Agee E R 1
Similar to the encoder, the embedding layer frozen after initializing it with pre-trained BERT embeddings. The transformer block consists of two attention layers and a 3-layer MLP with ReLU non-linear activation functions inbetween. The first attention layer computes the self attention the input embeddings helr contextual encoding for the target tokens. The second attention layer then computes the attention O'e R L between the source representation hare and the i-th token in hugl The i-th target token is consequently represented by the weighted sum of h with the weights Ore This generates source information-awan target representation figg R L where Lugi denotes the number of tokens in the target sequence. Next, big is fed into the 3-layer MLP with ReLU activation functions inbetween, resulting hugs E L The block hidden size of this transformer is H = 64.
Taking hug input, a linear layer with tanh activation projects the target representation into the same space as the embeddings (with dimensionality of 768), then the pre-trained embedding matrix E generates output logits (Press and Wolf 2016), where the output size is same the vocabulary size. The resulting logits are then normalized by softmax to generate probability distribution over all tokens in vocabulary:
Pa(v') = Solumax (Tige())e)))
(5)
in which, (y() ) is the generation (abstractive) probability distribution.
We employ the pointer softmax (Gulcehre et al. 2016 mechanism to switch between generating token y' (from vocabulary) and pointing (to token in the source text). Specifically, the pointer softmax module computes scalar switch s' each generation time-step and uses it interpolate the abstractive distribution (yyy) over the vocabulary (Equation 5) and the extractive distribution (yyy') Osee over the source text tokens:
p(v') s' p.()'') (1- -') pp)('))
(6)
where 8 conditioned on both the attention -winthoe source representation Σ, 3. hd and the decoder state i
[.
sigmoid
(tanh T
122 a 1(we) + La ()))))).
In which, L, E TRElhx HxI
Lg E R 2HxH and La HxHl are linear layers, 64.
B TRAINING AND IMPLEMENTATION DETAILS
In this section, we provide hyperparameters and other implementation details.
For all experiments, we use Adam (Kiggma and Ba 2014) as the optimizer. The learning rate is set to 0.001 with clip gradient norm of 5.
15

---

Published as a conference paper at ICLR 2021
During training with DAgger, use batch size of 10 to collect transitions (tuples of (on, Or: g, âz)) at each game step i, where â, is the ground-truth action provided by the rule-based expert (see Sec- tion E) We gather sequence of transitions from each game episode, and push each sequence into replay buffer, which has capacity of 500K episodes. We set the max number of steps per episode to be 50. If the agent uses up this budget, the game episode is forced to terminate. We linearly anneal the fraction of the expert's ' assistance from 100% to 1% across a window of 50K episodes.
The agent is updated after every 5 steps of data collection. We sample a batch of 64 data points from the replay buffer. In the setting with the recurrent aggregator, every sampled data point is a sequence of 4 consecutive transitions. Following the training strategy used in the recurrent DQN literature (Hausknecht and Stone 2015; Yuam et al.) 2018). we use the first 2 transitions to estimate the recurrent states, and the last transitions for updating the model parameters.
BUTLER::BAII learns to generate actions token-by-token where we set the max token length to be 20. The decoder stops generation either when it generates a special end-of-sentence token [EOS], or hits the token length limit.
When using the beam search heuristic to recover from failed actions (see F we use a beam width of 10, and take the top-5 ranked outputs as candidates. We iterate through the candidates in the rank order until one of them succeeds. This heuristic is not always guaranteed to suc- ceed, however, we find it helpful in most cases. Note that we do not employ beam search when we evaluate during the training process for effi- ciency, e.g., in the seen and unseen curves shown in Figure 4 We take the best performing check- points and then apply this heuristic during eval- uation and report the resulting scores in tables (e.g., Table 2
from
countertop
sospboiile
take
microware
close
microware
cablieet
drawer
Pobakkity
Figure 5: Beam search for recovery actions.
By default unless mentioned otherwise (ablations), we use all available training games in each of the task types. We use an observation queue length of 5 and use a recurrent aggregator. The model is trained with DAgger, and during evaluation, we apply the beam search heuristic to produce the reported scores. All experiment settings in TextWorld are run with 8 random seeds. All text agents trained for 50,000 episodes.
C TEXTWORLD ENGINE
Internally, the TextWorld Engine is divided into two main components: a planner and text generator.
Planner TextWorld Engine uses Fast Downward (Helment, 2006), a domain-indeependen classical planning system to maintain and update the current state of the game. A state is represented by set of predicates which define the relations between the entities (objects, player, room, etc.) present in the game. A state be modified by applying production rules corresponding to the actions listed in Table All variables, predicates, and rules are defined using the PDDL language.
For instance, here is simple state representing player standing next microwave which is closed and contains a mug:
at miipoaaee,icrrooaeee
microwave) @ in(mug, miggg,icrowaaa @ ccoseedmiccoowwv/
openable(m..croxvve)
where the symbol @ is the linear logic multiplicative conjunction operator. Given that state, valid action could be open mi crowave, which would essentially transform the state by replacing cl closed(m mi crowave) with
Text generator The other component of the TextWorld Engine, the text generator, uses context- sensitive grammar designed for the ALFRED environments. The grammar consists text templates similar to those listed in T When needed, the engine will sample template given some context,
16

---

Published as a conference paper at ICLR 2021
Le., the current state and the last action. Then, the template gets realized using the predicates found in the current state.
D MASK R-CNN DETECTOR
We use Mask R R-CNN detector (He et al. 2017) pre-trained on MSCOCO (Lin et al. 2014) and fine-tune it with additional labels from ALFRED training scenes. To generate additional labels, we replay the expert demonstrations from ALFRED and record ground-truth image and instance segmentation pairs from the simulator (THOR) after completing each high-level action e.g-, goto, pickup etc. We generate a dataset of 50K images, and fine-tune the detector for 4 epochs with a batch size of 8 and learning rate of 5e-4. The detector recognizes 73 object classes where each class could vary up to 1-10 instances. Since demonstrations in the kitchen are often longer as they involve complex sequences like heating, cleaning etc., the labels are slightly skewed towards kitchen objects. To counter this, we balance the number of images sampled from each room (kitchen, bedroom, livingroom, bathroom) so the distribution of object categories uniform across the dataset.
E RULE-BASED EXPERT
To train text agents an imitation learning (II.) setting, we use rule-based expert for supervision. given task is decomposed into sequence of subgoals (e.g., for heat & place: find the object, pick the object, find the microwave, heat the object with the microwave, find the receptacle, place the object in the receptacle), and a closed-loop controller tries to sequentially execute these goals. We note that while designing sule-based experts for ALFWorld is relatively straightforward, experts operating directly in embodied settings like the PDDL planner used in ALFRED prone to failures due to physical infeasibilities and non-deterministic behavior in physics-based environments.
F BENEFITS OF TRAINING IN TEXTWORLD OVER EMBODIED WORLD
Pre-training in TextWorld offers several benefits over directly training in embodied environments. Figure 6 presents the performance of an expert (that agents are trained imitate) across various environments. The abstract textual space leads to higher goal success rates resulting from successful navigation and manipulation subroutines. World agents also do not suffer from object mis- detections and slow execution speed.
Excert Rollouts 1000 episodes
Walk ock Time 1000 episades
Test/Madd
Entedted (Omako
Hyard (Oreca)
Hyand (AUTLER)
10
144
Gral
Madgatien 260
Man oorboaazinn
Total Ma lcock Time bect
Figure 6: Domain Analysis: The performance of an expert across various environments.
17

---

Published as a conference paper at ICLR 2021
G OBSERVATION TEMPLATES
The following templates are used by the state-estimator to generate textual observations or- The object IDs {obj id} correspond Mask R-CNN objects detection ground-truth instance IDs. The receptacle IDs (recep id] are based on the receptacles listed in the initial observation Og- Failed actions and actions without any state-changes result in Nothing happens.
Actions
Templates
(a) You arrive at (loc id). On the (recep id), you see a (objl id),
and a (objN id)
goto
(b) You arrive at (loc id}. The (recep id) is closed.
(c) You arrive
(loc id). The (recep id) is open. and a (objN id).
On it, you see a (objl id),
take
You pi ck up the {obj id) from the (recep id).
put
You put the (obj id} on the (recep id).
(a) You open the (recep id). it, you see a (obj1 id), and a (objN id) (b) You open the (recep id). The {recep id) is empty.
open
close
You close the (recep id)
toggle
You turn the {obj id) on.
heat
You heat the {obj id) with the (recep id)
cool
You cool the {obj id) with the (recep id).
clean
You clean the (obj id) with the {recep id).
(a) You are carrying: {obj id)
inventory
(b) You are not carrying anything.
(a) On the (recep id), you see (objl id),
and (objN id)
examine
(b) This is hot/cold/clean (obj).
Table 6: High-level text actions supported in ALFWorld along with their observation templates.
18

---

Published as a conference paper at ICLR 2021
H GOAL DESCRIPTIONS
H.1 TEMPLATED GOALS
The goal instructions for training games are generated with following templates. Here obj, recep, lamp refer object, receptacle, and lamp classes, respectively, that pertain particular task. For each task, the two corresponding templates are sampled with equal probability.
task-type
Templates
(a) put {obj} in (recep).
Pick & Place
(b) put some (obj} on (zecep).
(a) look at [obj) under the (lamp).
Examine in Light
(b) examine the [obj) with the (1amp).
(a) put clean [obj) in (recep).
Clean & Place
(b) clean some (obj) and put it in (recep).
(a) put a hot (obj} in (recep).
Heat Place
(b) heat some [obj} and put it in (zecep).
(a) put cool (obj) in (recep).
Cool & Place
(b) cool some [obj} and put it in (recep).
(a) put two [obj] in (recep).
Pick Two & Place
(b) find two (obj) and put them (recep).
Table Task-types and the corresponding goal description templates.
H.2 HUMAN ANNOTATED GOALS
The human goal descriptions used during evaluation contain 66 unseen verbs and 189 unseen nouns with respect to the templated goal instructions used during training.
acquire, arrange, can, carry, chill, choose, cleaning, clear, cook, cooked, cooled, dispose, done, drop, end, fill, filled, frying, garbage, gather, go, grab, handled, heated, heating, hold, holding, inspect, knock, left, lit, lock, microwave, microwaved, move, moving, pick, picking, place, placed, placing, putting, read, relocate, remove, retrieve, return, rinse, serve, set, soak, stand, standing, store, take, taken, throw, transfer, turn, turning, use, using, walk, warm, wash, washed.
Unseen Verbs:
alarm, area, back, baisin, bar, bars, base, basin, bathroom, beat, bed, bedroom, bedside, bench, bin, books, bottle, bottles, bottom, box, boxes, bureau, burner, butter, can, canteen, card, cardboard, cards, cars, eds, cell, chair, cheair, chest, chill, cistern. cleaning, clock, clocks, coffee, container, containers, control, controllers, controls, cooker, corner, couch, count, counter, cover, cream, credit, cupboard, dining, disc, discs, dishwasher, disks, dispenser, door, drawers, dresser, edge, end, floor, food, foot, freezer, game, garbage, gas, glass, glasses, gold, grey, hand, head, holder, ice, inside, island, item, items, jars, keys, kitchen, knifes, knives, laddle, lamp, lap, left, lid, light, loaf, location, lotion, machine, magazine, maker, math, metal, microwaves, move, nail, newsletters, newspapers, night, nightstand, object, ottoman, oven, pans, paper, papers, pepper, phone, piece, pieces, pillows, place, polish, pot, pullout, pump, rack, rag, recycling, refrigerator, remote, remotes, right, rinse, roll, rolls, room, safe, salt, scoop, seat, sets, shaker, shakers, shelves, side, sink, sinks, skillet, soap, soaps, sofa, space, spatulas, sponge, spoon, spot, spout, spray, stand, stool, stove, supplies, table, tale, tank, television, textbooks, time, tissue, tissues, toaster, top, towel, trash, tray, tv, vanity, vases, vault, vegetable, wall, wash, washcloth, watches, water, window, wine.
Unseen Nouns:
I ACTION CANDIDATES VS ACTION GENERATION
BUTLER::BRAN generates actions in a token-br token-by-token by-token fashion. Prior text-based agents typically list of candidate commands from the game engine (Adhikari et al.) 2020) populate list of command templates (Ammanabrolu and Hausknecht (220)). We initially trained our agents with candidate commands from the WoxtWorld Engine, but they quickly ovefit without learning affordances.
19

---

Published as a conference paper at ICLR 2021
commonsense, pre-conditions, and had zero performance embodied transfer. In the embodied setting, without access to a TextWorld Engine, it is difficult to generate candidate actions unless a set of heuristics is handcrafted with strong priors and commonsense knowledge. We also experimented with populating a list command templates, but found this to be infeasible as some scenarios involved 1000s of populated actions per game step.
ALFRED TASK DESCRIPTIONS
The following descriptions describe the processes involved in each of six task-types:
Pick & Place (e.g., "put plate on the coffee table") the agent must find object of the desired type, pick it up, find the correct location to place it, and put it down there.
Examine in Light (e.g., "examine a book under the lamp") the agent must find an object of the desired type, locate and
light source with the desired object in-hand.
Clean & Place (e.g.. "clean the knife and put in the drawer") the agent must find an object the desired type, pick it up, go to a sink or a basin, wash the object by turning on the faucet, then find the correct location place it, and put it down there.
Heat & Place (e.g., "heat a mug and put on the coffee table") the agent must find an object of the desired type, pick up, go to a microwave, heat the object turning on the microwave, then find the correct location to place it, and put down there.
Cool & Place (e.g., "put cool bottle on the countertop"). the agent must find an object of the desired type, pick up. go fridge. put the object inside the fridge and cool then find the correct location to place it, and put it down there.
Pick Two & Place (e.g., "put two pencils in the drawer")) the agent must find an object of the desired type, pick it up. find the correct location to place it, put it down there, then look for another object of the desired type, pick it up, return to previous location, and put it down there with the other object.
20

---

Published as a conference paper at ICLR 2021
K ALFWORLD TEXT GAME EXAMPLES
Three examples of ALFWorld text game walkthroughs:
You are in the middle of zoon. Locking quickly around you, see towelholder bandtoweibolder 2, draver tolletpaperhange
toilet 1, bathtubbarin
6,
countertop sinkbasin handtoweiholder 1, a towelbolder 2, sinkbasin 2, and gazbagecan 1.
5,
eloth bathtubbosin.
go to drawer
You arrive lcc 9, The draver is closed.
drawer
open
ogen. you
close drawer
You close the dreNer 5.
countertop
arrive
countertop 1, you cloth soapbottle 1,
and a candle 3.
take cloth 1 from countertop countertop
cloth
go sinkbasin
You arrive at lcc 20. On the ainkbasin 1, you see nothing.
clean cloth with sinkbasin
the cloth using siakbasin
go bathtubbasin
You arrive at Icc 14. On the bothtubbasin 1, you see soapbar 1.
put cloth in/os bathtubbasin!
You are in the middle of zoon. Looking quickly around ornchair
drawer 15, dravez 19, sidetable
sidetable draver 4, a draver 20, 1, drawez 13, drawve 21, a aidetable 4, a sidetable 2, a draver 5, drawez 11, a draver 1, garbagacan 1, dreaser cabinet 1, drawer drower 18, sidetable 1, draver 14, Brawer 6,
drawer
Your task ia to: find two renotacontrol and put. then in arnchair.
sidetable
arrive 34. aidetable 2, you remotacontrol
take remotecontrol 1 from sidetable
pick up the renotacontrol from the sidetable
arnchaiz you see nothing. put renotacostrol 1 in/on arnchaie 1
You put. the renotacontrol 1 in/on the arnchair 1. arrive 18.
1, you see creditoard 1, television 1, laptop 1, book 1, a statue 2, and a cellphone 2.
to sidetable 5
widetable 5, you You arrive at loc 2. On the soča 1, you see nevegager 1, pillow and remotecontrol. 2,
go
take remotecontrol 2 from sofa
up renotecoetrol
go to arnchair
You arriva at: loe 1. On the arnchair 1, you see renotacontro) 1. put romotacostrol 2 in/oa arnchair 1
the middle bed 1, draver 1, shelf 5, shelf 2, a sidetable 2, a shelf 3, drewer 3, a shelf 1, sidetable 1, desk 1, and garbagezan 1.
tooking quickly you, you
desklang.
go to deak 1
arrive loo on desk
bowl alarnelock pencil 2, penail 3, keychain
1, take alarmelock from desk
crediteard
You pick up the alarnclook 2 fron the desk 1. sidetable
you desklanp
use desklamp
You von!
21


Reviews:
[{"review_id": "yGRFjffJL3M", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper presents a new interactive environment which is both text-based and contains visual simulation which are aligned. The authors also propose a first agent architecture which uses the visual observations as well as the text-based (named BUTLER). The authors tested the generalization capabilities of the proposed BUTLER architecture compared to a seq2seq transformer model.\n\nStrong points:\n- novel environment for text-based and aligned visual content (could potentially lead to follow up research) - a significant contribution to the community.\n- have demonstrated that visual representation helps to generalize in these kind of environments (text + visual)\n- the paper is nicely written and easy to follow\n- the figures plots and tables are clear and help to understand the research\n\nWeak points:\n- the complex system: e.g.,  a pre-trained M-RCNN, a pre-trained text-agent, and training the text-agent using imitation learning (DAgger) biases the experimentation (makes the results less convincing). Maybe an intermediate naive baseline should have been considered.", "extra_scores": {}, "extra_text": {}}, {"review_id": "m1BjkYpKect", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work suggests an approach to improve the embodied agents interacting in an environment solving language-specified tasks. The approach introduces a parallel text-based (minus any physical interaction or visual input) environment which can be used to pretrain the agent by learning language-only policies. This text-based environment is an extension of the TextWorld framework.\n\nStrengths:\n\n* The paper is clearly written.\n* The premise of the paper that such a parallel environment allows agents to explore/learn in an abstract language-only environment while also allowing transfer of the learned knowledge to physical embodied environments is well motivated.\n\nWeaknesses:\n\n* The main premise is that such a text-based environment can be used to learn abstract (high-level) policies that can then be transferred to an embodied agent to solve language-specified tasks in a physically simulated (with visual input) environment. However, the main experiment to prove this claim in Table 4 falls short of proving this. The results in that table show agents learned with an oracle state estimator which means there is no visual input processing during this mode. It can also be noted that the Controller is also a heuristic module with no learning. Which means under this setting, learning/evaluating in embodied environment is pure text-driven which is same as TextWorld, hence it is not surprising that TW-Only performs better.\n* The performance of BUTLER-ORACLE in Table 3 is similar to the performance of BUTLER on TextWorld (on All Tasks) which further proves that using oracle state estimator is essentially reducing the embodied environment tasks to TextWorld task. \n* Furthermore, given that this is the main premise of the paper, Table 4 needs to be complete with all the tasks.\n* It is unclear why the components in BUTLER Agent interact using discretized constructs (of natural language) instead of continuous vector spaces. \n* There seem to be several hacks adopted to obtain good results (such as using beam search during evaluation, using rule-based expert for supervision) which make this approach highly specific to the given dataset/environment limiting its general applicability.\n\nOverall: The strong motivation of this work is not supported by empirical results. The results presented in the paper use a number of hacks, are incomplete and lack insights.\n\nUpdate after author response:\n\nThe author response is much appreciated. However, my two main concerns remain unaddressed. The authors may add these additional experiments/results to Table 3 and 4 in further revisions for a stronger submission.\n* Table 3 is the main result of the paper which claims policies learned in TextWorld (TW) environment can be transferred over to ALFRED (ALF) environment under zero-shot setting. This result alone has several weaknesses -- (1) Evaluation done on non-human goals in ALF seem to use same template as that used in TW, so it is not surprising that agents will have non-zero success rates on similar language specifications. (2) When evaluation is done on human goals (which seems to be the real test), the agent's performance is very low. Also, there are no baselines (e.g., random) provided to compare those scores against. (3) Why are the experiments only conducted in zero-shot setting? This actually brings me to the second weakness.\n\n* Since the transfer learning is happening from a pure-text TW environment to a physically simulated (with visual input) ALF environment, it is more interesting/relevant to see how the language module pre-trained on text-only TW adapts to multimodal setup in ALF. This adaptation will require further training/fine-tuning on ALF so that visual/control modules can adapt to this pre-trained language module. This experiment was attempted in Table 4, however, as pointed in my initial review, this falls short of proving any claims made in the paper because the agents in Table 4 are learned with an oracle state estimator which means there is no visual input processing during this mode. It can also be noted that the Controller is also a heuristic module with no learning. Which means the setting used in Table 4 reduces learning/evaluating in embodied ALF environment to a pure text-driven environment.", "extra_scores": {}, "extra_text": {}}, {"review_id": "Ws-cdBeCf0b", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\n\nThis paper describes an approach for training an agent which completes goals specified via a language instruction in the ALFRED environment. The paper proposes a model and training scheme. \n\nThe model decomposes the problem into three steps: (1) a perceptual module which takes as input an environment observation and generates a textual description of it including the objects and their spatial relations, (2) a goal-planning module which takes as input the high-level goal (which may require completing multiple subgoals), the textual description generated from module 1 and generates a textual description of a subgoal, such as an action the agent should take with arguments, and (3) a controller module which takes as input the state of the environment and the subgoal description generated from module 2, and generates a sequence of actions which execute this subgoal. The entire model is ran each time the agent completes a subgoal, using the current environment observation as input to module 1.\n\nThe proposed training scheme focuses on pre-training the second module. This module is pre-trained using a proposed environment, ALFWorld, which is modeled after text adventure games like TextWorld (Côté et al. 2018), and is intended to abstract away the perception and control problems from the reasoning module. Training environments in ALFRED are converted into ALFWorld environments, and for each training goal, an oracle is constructed (which is a function that maps from any ALFWorld environment state to an optimal subsequent action that leads to the goal). This facilitates pre-training the second module using DAgger through interactions with ALFWorld.\n\n----------------------------------\n\nReasons for score:\n\nI vote for accepting the paper. The idea of explicitly training the model to reason about subgoals is intriguing. However, there are several assumptions and limitations of the proposed approach.\n\n----------------------------------\n\nStrengths:\n\n- A model which decomposes high-level goals into low-level action sequences is very valuable and interesting.\n- Comparison of single-goal vs. multiple-goal models.\n- Good set of ablations, experiments, and comparisons, although I am not sure why Section 4.3 only looked at a single task.\n\n----------------------------------\n\nWeaknesses:\n\n- Because ALFWorld is an abstraction of the problem, it must make assumptions about perception and/or physical control. The paper describes several simplifications it makes: e.g., as far as I can tell, it disregards physical attributes of objects, and only retains the existence of particular object types when BUTLER::Vision processes an observation. Also, ALFWorld does not implement physical/semantic constraints (e.g., what can fit in a microwave). The setup of BUTLER means that losing perceived attributes in the output of the first module may result in the reasoning module being unable to reason about these attributes, and it is up to the designers of ALFWorld to decide which attributes are important to retain.\n- As above, ALFWorld is manually designed and requires making decisions about what aspects of the problem to model and which aspects should be abstracted away. This also includes requiring design of an oracle function, or alternatively an effective reward function (i.e., a denser reward function than binary task completion), which was not explored.\n\n----------------------------------\n\nQuestions for the authors:\n\n- Can you elaborate more on what it means for a sequence to be prototypical (in the abstract)?\n- What is the difference between the high-level goals and low-level step-by-step language in ALFRED? Is this similar to the goal \"put a pan on the dining table\" vs. \"go to the cabinet\", \"open the cabinet\"?\n- What exactly are the generalization dimensions for unseen rooms? Novel combinations of object instances that existed during training, novel instances (e.g., color) of object types that existed during training, or completely novel object types? Or something else?\n- In the last paragraph of section 3.1, does this refer to playing a game at inference time using ALFRED, or during training with ALFWorld?\n- Beam search is only done on the subgoal descriptions output by BUTLER::Brain, not the action actions taken in the ALFRED environment by the controller?\n- If the Human Goals setting is using human-written (natural language) goal specifications, then what are the goals in the other evaluation settings? Auto-generated?\n\n----------------------------------\n\nAdditional feedback:\n\n- The color coding that begins on the third page is confusing, and the colors are very hard to tell apart (at least for me). \n- In the related work on embodied language learning, the last sentence is a bit confusing as ALFRED is a fully interactive modality, right? If not, what is considered fully-interactive?\n\n----------------------------------\n\nRating after discussion: lowering to a 6, as I share concerns with R1 about experiments and generalizability of the proposed approach.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: 
The paper proposes ALFWorld, which combined TextWorld and ALFRED to create aligned scenarios (one that is text-only, and the other in an embodied visual simulator) so that high-level policies in language can be learned in a simpler world, and then transferred to the embodied one (using the proposed BUTLER architecture).  The proposed BUTLER model consists of three components: 1) a perceptual module (converts environment observation to specification of objects and relations using text), 2) goal-planning module for generating textual specification of subgoals (from observed environment state) and 3) controller module which takes outputs from 1) and 2) and generates a sequence of actions.  Experiments show that using the textual specification, it is possible to models pretrained in the text world can generalize better to embodied settings.

Review Summary: The submission received slightly divergent reviews with R2, R3 recommending acceptance (score 7) and R1 recommending reject (score 4).  All reviewers recognized the novelty of the work, and the potential for follow-up work based on the submission.  After considering the author response and discussion between reviewers, both R2 and R3 agreed that there are indeed flaws with the work as pointed out by R1 (R3 lowered their rating to 6).  Despite the concerns, both R2,R3 remained on the positive side.

Pros: 
- The work and proposed framework can stimulate further research on transferring policies from simple text environments to more realistic visual environment. (R2)
- The decomposition of high-level goals into low-level actions sequences is a good direction for future research (R3)
- Good set of experiments and comparisons (R3)
- the paper is clearly written and easy to understand (R2)

Cons:
- The main claim of the work (high-level policies learned in a text-based environment can be transferred to a physically simulated environment) is not properly substantiated by the experimental results. (R1)
- The proposed method is a complex system and simpler baselines should be considered (R2)
- Some assumptions are made in ALFWorld need to be hand designed and may miss important aspects of perception (R3) 
- More experiments and ablations are needed to properly evaluate the framework

Despite the issues pointed out by R1, the AC believe that the work can inspire future work in this area, and thus recommend acceptance.  The paper is also well-written and easy to understand. 
Decision: Accept (Poster)