FILM: Following Instructions in Language with Modular Methods

1 INTRODUCTION . Human intelligence simultaneously processes data of multiple modalities , including but not limited to natural language and egocentric vision , in an embodied environment . Powered by the success of machine learning models in individual modalities ( Devlin et al. , 2018 ; He et al. , 2016 ; Voulodimos et al . ; Anderson et al. , 2018a ) , there has been growing interest to build multimodal embodied agents that perform complex tasks . An incipient pursuit of such interest was to solve the task of Vision Language Navigation ( VLN ) , for which the agent is required to navigate to the goal area given a language instruction ( Anderson et al. , 2018b ; Fried et al. , 2018 ; Zhu et al. , 2020 ) . Embodied instruction following ( EIF ) presents a more complex and human-like setting than VLN or Object Goal Navigation ( Gupta et al. , 2017 ; Chaplot et al. , 2020b ; Du et al. , 2021 ) ; beyond just navigation , agents are required to execute sequences of sub-tasks that entail both navigation and interaction actions from a language instruction ( Fig . 1 ) . The additional challenges posed by EIF are threefold - the agent has to understand compositional instructions of multiple types and subtasks , choose actions from a large action space and execute them for longer horizons , and localize objects in a fine-grained manner for interaction ( Nguyen et al. , 2021 ) . Most existing methods ( Zhang & Chai , 2021 ; Kim et al. , 2021 ; Nottingham et al. , 2021 ) for EIF have relied on neural memory of various types ( transformer embeddings , LSTM state ) , trained end-to-end with expert trajectories upon raw or pre-processed language/visual inputs . However , EIF remains a very challenging task for end-to-end methods as they require the neural net to simultaneously learn state-tracking , building spatial memory , exploration , long-term planning , and low-level control . In this work , we propose FILM ( Following Instructions in Language with Modular methods ) . FILM consists of several modular components that each ( 1 ) processes language instructions into structured forms ( Language Processing ) , ( 2 ) converts egocentric visual input into a semantic metric map ( Semantic Mapping ) , ( 3 ) predicts a search goal location ( Semantic Search Policy ) , and ( 4 ) outputs subsequent navigation/ interaction actions ( Deterministic Policy ) . FILM overcomes some 1At the time of submission , our model was the State of the Art on the official leaderboard of ALFRED : https : //leaderboard.allenai.org/alfred/submissions/public . of the shortcomings of previous methods by leveraging a modular design with structured spatial components . Unlike many of the existing methods for EIF , FILM does not require any input that provides sequential guidance , namely expert trajectories or low-level language instructions . While Blukis et al . ( 2021 ) recently introduced a method that uses a structured spatial memory , it comes with some limitations from the lack of explicit semantic search and the reliance on expert trajectories . On the ALFRED ( Shridhar et al. , 2020 ) benchmark , FILM achieves State-of-the-Art performance ( 24.46 % ) with a large margin ( 8 % absolute ) from the previous SOTA ( Blukis et al. , 2021 ) . Most approaches rely on low-level instructions , and we too find that including them leads to an additional 2 % improvement in success rate ( 26.49 % ) . FILM ’ s strong performance and our analysis indicate that an explicit structured spatial memory coupled with a semantic search policy can provide better state-tracking and exploration , even in the absence of expert trajectories or low-level instructions . 2 RELATED WORK . A plethora of works have been published on embodied vision and language tasks , such as VLN ( Anderson et al. , 2018b ; Fried et al. , 2018 ; Zhu et al. , 2020 ) , Embodied Question Answering ( Das et al. , 2018 ; Gordon et al. , 2018 ) , and topics of multimodal representation learning ( Wang et al. , 2020 ; Bisk et al. , 2020 ) , such as Embodied Language Grounding ( Prabhudesai et al. , 2020 ) . On Visual Language Navigation , which is the most comparable to the setting of our work , methods with impressive performances Ke et al . ( 2019 ) ; Wang et al . ( 2019 ) ; Ma et al . ( 2019 ) have been proposed since the introduction of R2R Anderson et al . ( 2018b ) . While far from conquering VLN , these methods have shown up to 61 % success rate on unseen test environments Ke et al . ( 2019 ) . On the more challenging task of Embodied Instruction Following ( EIF ) , multiple methods have been proposed with differing levels of modularity in the model structure . As a baseline , Shridhar et al . ( 2020 ) has presented a Seq2Seq model with an attention mechanism and a progress monitor , while Pashevich et al . ( 2021 ) proposed to replace to seq2seq model with an episodic transformer . These methods take the concatenation of language features , visual features , and past trajectories as input and predict the subsequent action end-to-end . On the other hand , Kim et al . ( 2021 ) ; Zhang & Chai ( 2021 ) ; Nguyen et al . ( 2021 ) modularly process raw language and visual inputs into structured forms , while keeping a separate “ action prediction module ” that outputs low-level actions given processed language outputs . Their “ action taking module ” itself is trained end-to-end and relies on neural memory that “ implicitly ” tracks all of spatial , progressive , and states of the agent . Unlike these methods , FILM ’ s structured language/ spatial representations make reasons for failure transparent and elucidates directions to improve individual components . Recently , Blukis et al . ( 2021 ) has proposed a more modular method with a persistent and structured spatial memory . Language and visual input are transformed into respectively high-level actions and the 3D map . With the 3D map and high-level actions as input , low-level actions are predicted with a value-iteration network ( VIN ) . Navigation goals for the VIN are sampled from a model trained on interaction pose labels from expert trajectories . Among all proposed methods for EIF , FILM necessitates the least information ( neither low-level instructions nor expert trajectories are needed , although the former can be taken as an additional input ) . Furthermore , FILM addresses the problem of search/ exploration of goal objects . Various works in visual navigation with semantic mapping are also relevant . Simultaneous Localization and Mapping ( SLAM ) methods , which build 2D or 3D obstacle maps , have been widely used ( Fuentes-Pacheco et al. , 2015 ; Izadi et al. , 2011 ; Snavely et al. , 2008 ) . In contrast to these works , recent methods ( Chaplot et al. , 2020b ; a ) build semantic maps with differentiable projection operations , which restrain egocentric prediction errors amplifying in the map . The task of Chaplot et al . ( 2020b ; a ) is object goal navigation , a much simpler task compared to EIF . Furthermore , while Chaplot et al . ( 2020b ) employs a semantic exploration policy , our and their semantic policies serve fundamentally different purposes ; while their policy guides a general sense of direction among multiple rooms in the search for large objects ( e.g . fridge ) , ours guides the search for potential locations of small and flat objects which have little chance of detection at a distance . Also , our semantic policy is conditioned on language instructions . Blukis et al . ( 2018a ; b ) also successfully utilized semantic 2D maps in grounded language navigation tasks . These works are for quadcopters , whose fields of view almost entirely cover the scene and the need for “ search ” or “ exploration ” is less crucial than for pedestrian agents . Moreover , their settings only involve navigation with a single subtask . 3 TASK EXPLANATION . We utilize the ALFRED benchmark . The agent has to complete household tasks given only natural language instructions and egocentric vision ( Fig . 1 ) . For example , the instruction may be given as “ Put a heated apple on the counter , ” with low-level instructions ( which FILM does not use by default ) further explaining step-by-step lower level actions . In this case , one way to “ succeed ” in this episode is to sequentially ( 1 ) pick up the apple , ( 2 ) put the apple in the microwave , ( 3 ) toggle the microwave on/off , ( 4 ) pick up the apple again , and ( 4 ) place it on the countertop . Episodes run for a significantly longer number of steps compared to benchmarks with only single subgoals ; expert trajectories , which are maximally efficient and perform only the strictly necessary actions ( without any steps to search for an object ) , are often longer than 70 steps . There are seven types of tasks ( Appendix A.1 ) , from relatively simple types ( e.g . Pick & Place ) to more complex ones ( e.g . Heat & Place ) . Furthermore , the instruction may require that an object is “ sliced ” ( e.g . Slice bread , cook it in the microwave , put it on the counter ) . An episode is deemed “ success ” if the agent completes all sub-tasks within 10 failed low-level actions and 1000 max steps . 4 METHODS . FILM consists of three learned modules : ( 1 ) Language Processing ( LP ) , ( 2 ) Semantic Mapping , and ( 3 ) Semantic Search Policy ; and one purely deterministic navigation/ interaction policy module ( Fig . 2 ) . At the start of an episode , the LP module processes the language instruction into a sequence of subtasks . Every time step , the semantic mapping module receives the egocentric RGB frame and updates the semantic map . If the goal object of the current subtask is not yet observed , the semantic search policy predicts a “ search goal ” at a coarse time scale ; until the next search goal is predicted , the agent navigates to the current search goal with the deterministic policy . If the goal is observed , the deterministic policy decides low-level controls for interaction actions ( e.g . “ Pick Up ” object ) . 4.1 LANGUAGE PROCESSING ( LP ) . The language processing ( LP ) module transforms high-level instructions into a structured sequence of subtasks ( Fig . 3 ) . It consists of two BERT ( Devlin et al. , 2018 ) submodules that receive the instruction as an input at the beginning of the episode . The first submodule ( BERT type classification ) receives the instruction and predicts the “ type ” of the instruction - one of the seven types stated in Appendix A.1 . The second submodule ( BERT argument classification ) receives both the instruction and the predicted type as input and predicts the “ arguments ” - ( 1 ) “ obj ” for the object to be picked up , ( 2 ) “ recep ” for the receptacle where “ obj ” should be ultimately placed , ( 3 ) “ sliced ” for whether “ obj ” should be sliced , and ( 4 ) “ parent ” for tasks with intermediate movable receptacles ( e.g . “ cup ” in “ Put a knife in a cup on the table ” of Appendix A.1 ) . An object in ALFRED is always an instance of either “ obj ” or “ recep ” ; “ parent ” objects are a subset of “ recep ” objects . We train a separate BERT model for each argument predictor . The two submodules are easily trainable with supervised learning since the type and the four arguments are provided in the training set . Models use only the CLS token for classification , and they do not share parameters ; all layers of “ bert-base-uncased ” were fine-tuned . Due to the patterned nature of instructions , we can match the predicted “ type ” of the instruction to a “ type template ” with blank arguments . For example , if the instruction is classified as the “ clean & place ” type , it is matched to the template “ ( Obj , PickUp ) , ( SinkBasin , Put ) , ( Faucet , ToggleOn ) , ( Faucet , ToggleOff ) , ( Obj , PickUp ) , ( Recep , Put ) ” . If the “ sliced ” argument is predicted to be true from argument classification , subtasks of “ ( Knife , PickUp ) , ( Obj , Slice ) , ( Sink , PutObject ) ” will be added at the beginning of the template ( with the ( Sink , PutObject ) to make the agent drop the knife ) . Filling in the “ type template ” with predictions of the second model , we obtain a list of subtasks ( bottom of Fig . 3b ) to be completed in the current episode .

Summary: This paper presents a modular system (FILM) for egocentric instruction following in the ALFRED environment. The system does not require expert trajectories and can operate without low-level instruction sequences. The system has the following components:  1. A language processing module that maps high-level instructions "Drop a clean pan on the table" to a low-level sub-task sequence. This module makes use of a number of BERT based classifiers that:     a.  classifiy the high level instruction into one of seven possible goal types, each of which is deterministically associated with a templated sub-task sequence     b. identify object IDs to fill in the slots in the template identified in (a)  2. A semantic mapping module that populates a grid with binary classifications that indicate object presence; obstacle presence; and whether the location has been explored. This module is inspired by previous work on egocentric navigation.  3. A novel semantic search policy that predicts the likely location of small sub-goal objects (bowl) on the basis of large receptacle objects (table). This is trained via supervised learning on sampled trajectories and it samples goals at a coarse tiem scale of every 25 steps.  4. A deterministic policy that operates within each 25 step search goal period. This makes use of the Fast Marching Method and is inspired by previous work on this task.  Together, these modules make up a system that generalizes to unseen environments much better than previous work. The new semantic search policy provide small but consistent improvements over a version of FILM that has this module removed.  FILM is evaluated in both the setting where low level instruction sequences are provided, and the setting where only the high-level goal is provided and FILM must map this goal to low-level instructions via module (1). In both settings, FILM significantly outpeforms previous work on unseen environments. When low-level instructions are provided, FILM does worse than some previous work on seen environments. When low-level instructions aren't provided, FILM does marginally better than previous work on seen environments.  Experiments show that the semantic search policy increases path-weighted metrics proportionally more than absolute metrics, suggesting that this policy improves efficiency as well as overall success of the search. Targeted evaluations also show that this search policy is particularly useful in large rooms, where the agent cannot see many target objects, and for the "Clean and Place" task that involves a difficult to detect object (sink) contained in a simple to detect object (countertop).