ALFWorld: Aligning Text and Embodied Environments for Interactive Learning

1 INTRODUCTION TextWorld Embodied Welcome ! You are in the middle of the room . Looking around you , you see a diningtable , a stove , a microwave , and a cabinet . Your task is to : Put a pan on the diningtable . > goto the cabinet You arrive at the cabinet . The cabinet is closed . > open the cabinet The cabinet is empty . > goto the stove You arrive at the stove . Near the stove , you see a pan , a pot , a bread loaf , a lettuce , and a winebottle . > take the pan from the stove You take the pan from the stove . > goto the diningtable You arrive at the diningtable . > put the pan on the diningtable You put the pan on the diningtable . Figure 1 : ALFWorld : Interactive aligned text and embodied worlds . An example with high-level text actions ( left ) and low-level physical actions ( right ) . Consider helping a friend prepare dinner in an unfamiliar house : when your friend asks you to clean and slice an apple for an appetizer , how would you approach the task ? Intuitively , one could reason abstractly : ( 1 ) find an apple ( 2 ) wash the apple in the sink ( 3 ) put the clean apple on the cutting board ( 4 ) find a knife ( 5 ) use the knife to slice the apple ( 6 ) put the slices in a bowl . Even in an unfamiliar setting , abstract reasoning can help accomplish the goal by leveraging semantic priors . Priors like locations of objects – apples are commonly found in the kitchen along with implements for cleaning and slicing , object affordances – a sink is useful for washing an apple unlike a refrigerator , pre-conditions – better to wash an apple before slicing it , rather than the converse . We hypothesize that , learning to solve tasks using abstract language , unconstrained by the particulars of the physical world , enables agents to complete embodied tasks in novel environments by leveraging the kinds of semantic priors that are exposed by abstraction and interaction . To test this hypothesis , we have created the novel ALFWorld framework , the first interactive , parallel environment that aligns text descriptions and commands with physically embodied robotic simulation . We build ALFWorld by extending two prior works : TextWorld ( Côté et al. , 2018 ) - an engine for interactive text-based games , and ALFRED ( Shridhar et al. , 2020 ) - a large scale dataset for visionlanguage instruction following in embodied environments . ALFWorld provides two views of the same underlying world and two modes by which to interact with it : TextWorld , an abstract , text-based environment , generates textual observations of the world and responds to high-level text actions ; ALFRED , the embodied simulator , renders the world in high-dimensional images and responds to low-level physical actions as from a robot ( Figure 1 ) .1 Unlike prior work on instruction following ( MacMahon et al. , 2006 ; Anderson et al. , 2018a ) , which typically uses a static corpus of cross-modal expert demonstrations , we argue that aligned parallel environments like ALFWorld offer a distinct advantage : they allow agents to explore , interact , and learn in the abstract environment of language before encountering the complexities of the embodied environment . While fields such as robotic control use simulators like MuJoCo ( Todorov et al. , 2012 ) to provide infinite data through interaction , there has been no analogous mechanism – short of hiring a human around the clock – for providing linguistic feedback and annotations to an embodied agent . TextWorld addresses this discrepancy by providing programmatic and aligned linguistic signals during agent exploration . This facilitates the first work , to our knowledge , in which an embodied agent learns the meaning of complex multi-step policies , expressed in language , directly through interaction . Empowered by the ALFWorld framework , we introduce BUTLER ( Building Understanding in Textworld via Language for Embodied Reasoning ) , an agent that first learns to perform abstract tasks in TextWorld using Imitation Learning ( IL ) and then transfers the learned policies to embodied tasks in ALFRED . When operating in the embodied world , BUTLER leverages the abstract understanding gained from TextWorld to generate text-based actions ; these serve as high-level subgoals that facilitate physical action generation by a low-level controller . Broadly , we find that BUTLER is capable of generalizing in a zero-shot manner from TextWorld to unseen embodied tasks and settings . Our results show that training first in the abstract text-based environment is not only 7× faster , but also yields better performance than training from scratch in the embodied world . These results lend credibility to the hypothesis that solving abstract language-based tasks can help build priors that enable agents to generalize to unfamiliar embodied environments . Our contributions are as follows : § 2 ALFWorld environment : The first parallel interactive text-based and embodied environment . § 3 BUTLER architecture : An agent that learns high-level policies in language that transfer to low-level embodied executions , and whose modular components can be independently upgraded . § 4 Generalization : We demonstrate empirically that BUTLER , trained in the abstract text domain , generalizes better to unseen embodied settings than agents trained from corpora of demonstrations or from scratch in the embodied world . 2 ALIGNING ALFRED AND TEXTWORLD . The ALFRED dataset ( Shridhar et al. , 2020 ) , set in the THOR simulator ( Kolve et al. , 2017 ) , is a benchmark for learning to complete embodied household tasks using natural language instructions and egocentric visual observations . As shown in Figure 1 ( right ) , ALFRED tasks pose challenging interaction and navigation problems to an agent in a high-fidelity simulated environment . Tasks are annotated with a goal description that describes the objective ( e.g. , “ put a pan on the dining table ” ) . We consider both template-based and human-annotated goals ; further details on goal specification can be found in Appendix H. Agents observe the world through high-dimensional pixel images and interact using low-level action primitives : MOVEAHEAD , ROTATELEFT/RIGHT , LOOKUP/DOWN , PICKUP , PUT , OPEN , CLOSE , and TOGGLEON/OFF . 1Note : Throughout this work , for clarity of exposition , we use ALFRED to refer to both tasks and the grounded simulation environment , but rendering and physics are provided by THOR ( Kolve et al. , 2017 ) . The ALFRED dataset also includes crowdsourced language instructions like “ turn around and walk over to the microwave ” that explain how to complete a goal in a step-by-step manner . We depart from the ALFRED challenge by omitting these step-by-step instructions and focusing on the more diffcult problem of using only on goal descriptions specifying what needs to be achieved . Our aligned ALFWorld framework adopts six ALFRED task-types ( Table 1 ) of various difficulty levels.2 Tasks involve first finding a particular object , which often requires the agent to open and search receptacles like drawers or cabinets . Subsequently , all tasks other than Pick & Place require some interaction with the object such as heating ( place object in microwave and start it ) or cleaning ( wash object in a sink ) . To complete the task , the object must be placed in the designated location . Within each task category there is significant variation : the embodied environment includes 120 rooms ( 30 kitchens , 30 bedrooms , 30 bathrooms , 30 living rooms ) , each dynamically populated with a set of portable objects ( e.g. , apple , mug ) , and static receptacles ( e.g. , microwave , fridge ) . For each task type we construct a larger train set , as well as seen and unseen validation evaluation sets : ( 1 ) : seen consists of known task instances { task-type , object , receptacle , room } in rooms seen during training , but with different instantiations of object locations , quantities , and visual appearances ( e.g . two blue pencils on a shelf instead of three red pencils in a drawer seen in training ) . ( 2 ) : unseen consists of new task instances with possibly known object-receptacle pairs , but always in unseen rooms with different receptacles and scene layouts than in training tasks . The seen set is designed to measure in-distribution generalization , whereas the unseen set measures out-of-distribution generalization . The scenes in ALFRED are visually diverse , so even the same task instance can lead to very distinct tasks , e.g. , involving differently colored apples , shaped statues , or textured cabinets . For this reason , purely vision-based agents such as the unimodal baselines in Section 5.2 often struggle to generalize to unseen environments and objects . The TextWorld framework ( Côté et al. , 2018 ) procedurally generates text-based environments for training and evaluating language-based agents . In order to extend TextWorld to create text-based analogs of each ALFRED scene , we adopt a common latent structure representing the state of the simulated world . ALFWorld uses PDDL - Planning Domain Definition Language ( McDermott et al. , 1998 ) to describe each scene from ALFRED and to construct an equivalent text game using the TextWorld engine . The dynamics of each game are defined by the PDDL domain ( see Appendix C for additional details ) . Textual observations shown in Figure 1 are generated with templates sampled from a context-sensitive grammar designed for the ALFRED environments . For interaction , TextWorld environments use the following high-level actions : goto { recep } take { obj } from { recep } put { obj } in/on { recep } open { recep } close { recep } toggle { obj } { recep } clean { obj } with { recep } heat { obj } with { recep } cool { obj } with { recep } where { obj } and { recep } correspond to objects and receptacles . Note that heat , cool , clean , and goto are high-level actions that correspond to several low-level embodied actions . ALFWorld , in summary , is an cross-modal framework featuring a diversity of embodied tasks with analogous text-based counterparts . Since both components are fully interactive , agents may be trained in either the language or embodied world and evaluated on heldout test tasks in either modality . We believe the equivalence between objects and interactions across modalities make ALFWorld an ideal framework for studying language grounding and cross-modal learning . 3 INTRODUCING BUTLER : AN EMBODIED MULTI-TASK AGENT . We investigate learning in the abstract language modality before generalizing to the embodied setting . The BUTLER agent uses three components to span the language and embodied modalities : BUTLER : :BRAIN – the abstract text agent , BUTLER : :VISION – the language state estimator , and BUTLER : :BODY – the low-level controller . An overview of BUTLER is shown in Figure 2 and each component is described below . 2To start with , we focus on a subset of the ALFRED dataset for training and evaluation that excludes tasks involving slicing objects or using portable container ( e.g. , bowls ) . 3.1 BUTLER : :BRAIN ( TEXT AGENT ) ∶ o0 , ot , g → at BUTLER : :BRAIN is a novel text-based game agent that generates high-level text actions in a token-by-token fashion akin to Natural Language Generation ( NLG ) approaches for dialogue ( Sharma et al. , 2017 ) and summarization ( Gehrmann et al. , 2018 ) . An overview of the agent ’ s architecture is shown in Figure 3 . At game step t , the encoder takes the initial text observation o0 , current observation ot , and the goal description g as input and generates a context- aware representation of the current observable game state . The observation o0 explicitly lists all the navigable receptacles in the scene , and goal g is sampled from a set of language templates ( see Appendix H ) . Since the games are partially observable , the agent only has access to the observation describing the effects of its previous action and its present location . Therefore , we incorporate two memory mechanisms to imbue the agent with history : ( 1 ) a recurrent aggregator , adapted from Yuan et al . ( 2018 ) , combines the encoded state with recurrent state ht−1 from the previous game step ; ( 2 ) an observation queue feeds in the k most recent , unique textual observations . The decoder generates an action sentence at token-by-token to interact with the game . The encoder and decoder are based on a Transformer Seq2Seq model with pointer softmax mechanism ( Gulcehre et al. , 2016 ) . We leverage pre-trained BERT embeddings ( Sanh et al. , 2019 ) , and tie output embeddings with input embeddings ( Press and Wolf , 2016 ) . The agent is trained in an imitation learning setting with DAgger ( Ross et al. , 2011 ) using expert demonstrations . See Appendix A for complete details . When solving a task , an agent might get stuck at certain states due to various failures ( e.g. , action is grammatically incorrect , wrong object name ) . The observation for a failed action does not contain any useful feedback , so a fully deterministic actor tends to repeatedly produce the same incorrect action . To address this problem , during evaluation in both TextWorld and ALFRED , BUTLER : :BRAIN uses Beam Search ( Reddy et al. , 1977 ) to generate alternative action sentences in the event of a failed action . But otherwise greedily picks a sequence of best words for efficiency . Note that Beam Search is not used to optimize over embodied interactions like prior work ( Wang et al. , 2019 ) . but rather to simply improve the generated action sentence during failures . 3.2 BUTLER : :VISION ( STATE ESTIMATOR ) ∶ vt → ot At test time , agents in the embodied world must operate purely from visual input . To this end , BUTLER : :VISION ’ s language state estimator functions as a captioning module that translates visual observations vt into textual descriptions ot . Specifically , we use a pre-trained Mask R-CNN detec- tor ( He et al. , 2017 ) to identify objects in the visual frame . The detector is trained separately in a supervised setting with random frames from ALFRED training scenes ( see Appendix D ) . For each frame vt , the detector generates N detections { ( c1 , m1 ) , ( c2 , m2 ) , . . . , ( cN , mN ) } , where cn is the predicted object class , and mn is a pixel-wise object mask . These detections are formatted into a sentence using a template e.g. , On table 1 , you see a mug 1 , a tomato 1 , and a tomato 2 . To handle multiple instances of objects , each object is associated with a class cn and a number ID e.g. , tomato 1 . Commands goto , open , and examine generate a list of detections , whereas all other commands generate affirmative responses if the action succeeds e.g. , at : put mug 1 on desk 2→ ot+1 : You put mug 1 on desk 2 , otherwise produce Nothing happens to indicate failures or no state-change . See Appendix G for a full list of templates . While this work presents preliminary results with template-based descriptions , future work could generate more descriptive observations using pre-trained image-captioning models ( Johnson et al. , 2016 ) , video-action captioning frameworks ( Sun et al. , 2019 ) , or scene-graph parsers ( Tang et al. , 2020 ) . 3.3 BUTLER : :BODY ( CONTROLLER ) ∶ vt , at → { â1 , â2 , . . . , âL } The controller translates a high-level text action at into a sequence of L low-level physical actions { â1 , â2 , . . . , âL } that are executable in the embodied environment . The controller handles two types of commands : manipulation and navigation . For manipulation actions , we use the ALFRED API to interact with the simulator by providing an API action and a pixel-wise mask based on Mask R-CNN detections mn that was produced during state-estimation . For navigation commands , each episode is initialized with a pre-built grid-map of the scene , where each receptacle instance is associated with a receptacle class and an interaction viewpoint ( x , y , θ , φ ) with x and y representing the 2D position , θ and φ representing the agent ’ s yaw rotation and camera tilt . The goto command invokes an A * planner to find the shortest path between two viewpoints . The planner outputs a sequence of L displacements in terms of motion primitives : MOVEAHEAD , ROTATERIGHT , ROTATELEFT , LOOKUP , and LOOKDOWN , which are executed in an open-loop fashion via the ALFRED API . We note that a given pre-built grid-map of receptacle locations is a strong prior assumption , but future work could incorporate existing models from the vision-language navigation literature ( Anderson et al. , 2018a ; Wang et al. , 2019 ) for map-free navigation .

Summary: The paper presents a new interactive environment which is both text-based and contains visual simulation which are aligned. The authors also propose a first agent architecture which uses the visual observations as well as the text-based (named BUTLER). The authors tested the generalization capabilities of the proposed BUTLER architecture compared to a seq2seq transformer model.