FILM: Following Instructions in Language with Modular Methods

1 INTRODUCTION . Human intelligence simultaneously processes data of multiple modalities , including but not limited to natural language and egocentric vision , in an embodied environment . Powered by the success of machine learning models in individual modalities ( Devlin et al. , 2018 ; He et al. , 2016 ; Voulodimos et al . ; Anderson et al. , 2018a ) , there has been growing interest to build multimodal embodied agents that perform complex tasks . An incipient pursuit of such interest was to solve the task of Vision Language Navigation ( VLN ) , for which the agent is required to navigate to the goal area given a language instruction ( Anderson et al. , 2018b ; Fried et al. , 2018 ; Zhu et al. , 2020 ) . Embodied instruction following ( EIF ) presents a more complex and human-like setting than VLN or Object Goal Navigation ( Gupta et al. , 2017 ; Chaplot et al. , 2020b ; Du et al. , 2021 ) ; beyond just navigation , agents are required to execute sequences of sub-tasks that entail both navigation and interaction actions from a language instruction ( Fig . 1 ) . The additional challenges posed by EIF are threefold - the agent has to understand compositional instructions of multiple types and subtasks , choose actions from a large action space and execute them for longer horizons , and localize objects in a fine-grained manner for interaction ( Nguyen et al. , 2021 ) . Most existing methods ( Zhang & Chai , 2021 ; Kim et al. , 2021 ; Nottingham et al. , 2021 ) for EIF have relied on neural memory of various types ( transformer embeddings , LSTM state ) , trained end-to-end with expert trajectories upon raw or pre-processed language/visual inputs . However , EIF remains a very challenging task for end-to-end methods as they require the neural net to simultaneously learn state-tracking , building spatial memory , exploration , long-term planning , and low-level control . In this work , we propose FILM ( Following Instructions in Language with Modular methods ) . FILM consists of several modular components that each ( 1 ) processes language instructions into structured forms ( Language Processing ) , ( 2 ) converts egocentric visual input into a semantic metric map ( Semantic Mapping ) , ( 3 ) predicts a search goal location ( Semantic Search Policy ) , and ( 4 ) outputs subsequent navigation/ interaction actions ( Deterministic Policy ) . FILM overcomes some 1At the time of submission , our model was the State of the Art on the official leaderboard of ALFRED : https : //leaderboard.allenai.org/alfred/submissions/public . of the shortcomings of previous methods by leveraging a modular design with structured spatial components . Unlike many of the existing methods for EIF , FILM does not require any input that provides sequential guidance , namely expert trajectories or low-level language instructions . While Blukis et al . ( 2021 ) recently introduced a method that uses a structured spatial memory , it comes with some limitations from the lack of explicit semantic search and the reliance on expert trajectories . On the ALFRED ( Shridhar et al. , 2020 ) benchmark , FILM achieves State-of-the-Art performance ( 24.46 % ) with a large margin ( 8 % absolute ) from the previous SOTA ( Blukis et al. , 2021 ) . Most approaches rely on low-level instructions , and we too find that including them leads to an additional 2 % improvement in success rate ( 26.49 % ) . FILM ’ s strong performance and our analysis indicate that an explicit structured spatial memory coupled with a semantic search policy can provide better state-tracking and exploration , even in the absence of expert trajectories or low-level instructions . 2 RELATED WORK . A plethora of works have been published on embodied vision and language tasks , such as VLN ( Anderson et al. , 2018b ; Fried et al. , 2018 ; Zhu et al. , 2020 ) , Embodied Question Answering ( Das et al. , 2018 ; Gordon et al. , 2018 ) , and topics of multimodal representation learning ( Wang et al. , 2020 ; Bisk et al. , 2020 ) , such as Embodied Language Grounding ( Prabhudesai et al. , 2020 ) . On Visual Language Navigation , which is the most comparable to the setting of our work , methods with impressive performances Ke et al . ( 2019 ) ; Wang et al . ( 2019 ) ; Ma et al . ( 2019 ) have been proposed since the introduction of R2R Anderson et al . ( 2018b ) . While far from conquering VLN , these methods have shown up to 61 % success rate on unseen test environments Ke et al . ( 2019 ) . On the more challenging task of Embodied Instruction Following ( EIF ) , multiple methods have been proposed with differing levels of modularity in the model structure . As a baseline , Shridhar et al . ( 2020 ) has presented a Seq2Seq model with an attention mechanism and a progress monitor , while Pashevich et al . ( 2021 ) proposed to replace to seq2seq model with an episodic transformer . These methods take the concatenation of language features , visual features , and past trajectories as input and predict the subsequent action end-to-end . On the other hand , Kim et al . ( 2021 ) ; Zhang & Chai ( 2021 ) ; Nguyen et al . ( 2021 ) modularly process raw language and visual inputs into structured forms , while keeping a separate “ action prediction module ” that outputs low-level actions given processed language outputs . Their “ action taking module ” itself is trained end-to-end and relies on neural memory that “ implicitly ” tracks all of spatial , progressive , and states of the agent . Unlike these methods , FILM ’ s structured language/ spatial representations make reasons for failure transparent and elucidates directions to improve individual components . Recently , Blukis et al . ( 2021 ) has proposed a more modular method with a persistent and structured spatial memory . Language and visual input are transformed into respectively high-level actions and the 3D map . With the 3D map and high-level actions as input , low-level actions are predicted with a value-iteration network ( VIN ) . Navigation goals for the VIN are sampled from a model trained on interaction pose labels from expert trajectories . Among all proposed methods for EIF , FILM necessitates the least information ( neither low-level instructions nor expert trajectories are needed , although the former can be taken as an additional input ) . Furthermore , FILM addresses the problem of search/ exploration of goal objects . Various works in visual navigation with semantic mapping are also relevant . Simultaneous Localization and Mapping ( SLAM ) methods , which build 2D or 3D obstacle maps , have been widely used ( Fuentes-Pacheco et al. , 2015 ; Izadi et al. , 2011 ; Snavely et al. , 2008 ) . In contrast to these works , recent methods ( Chaplot et al. , 2020b ; a ) build semantic maps with differentiable projection operations , which restrain egocentric prediction errors amplifying in the map . The task of Chaplot et al . ( 2020b ; a ) is object goal navigation , a much simpler task compared to EIF . Furthermore , while Chaplot et al . ( 2020b ) employs a semantic exploration policy , our and their semantic policies serve fundamentally different purposes ; while their policy guides a general sense of direction among multiple rooms in the search for large objects ( e.g . fridge ) , ours guides the search for potential locations of small and flat objects which have little chance of detection at a distance . Also , our semantic policy is conditioned on language instructions . Blukis et al . ( 2018a ; b ) also successfully utilized semantic 2D maps in grounded language navigation tasks . These works are for quadcopters , whose fields of view almost entirely cover the scene and the need for “ search ” or “ exploration ” is less crucial than for pedestrian agents . Moreover , their settings only involve navigation with a single subtask . 3 TASK EXPLANATION . We utilize the ALFRED benchmark . The agent has to complete household tasks given only natural language instructions and egocentric vision ( Fig . 1 ) . For example , the instruction may be given as “ Put a heated apple on the counter , ” with low-level instructions ( which FILM does not use by default ) further explaining step-by-step lower level actions . In this case , one way to “ succeed ” in this episode is to sequentially ( 1 ) pick up the apple , ( 2 ) put the apple in the microwave , ( 3 ) toggle the microwave on/off , ( 4 ) pick up the apple again , and ( 4 ) place it on the countertop . Episodes run for a significantly longer number of steps compared to benchmarks with only single subgoals ; expert trajectories , which are maximally efficient and perform only the strictly necessary actions ( without any steps to search for an object ) , are often longer than 70 steps . There are seven types of tasks ( Appendix A.1 ) , from relatively simple types ( e.g . Pick & Place ) to more complex ones ( e.g . Heat & Place ) . Furthermore , the instruction may require that an object is “ sliced ” ( e.g . Slice bread , cook it in the microwave , put it on the counter ) . An episode is deemed “ success ” if the agent completes all sub-tasks within 10 failed low-level actions and 1000 max steps . 4 METHODS . FILM consists of three learned modules : ( 1 ) Language Processing ( LP ) , ( 2 ) Semantic Mapping , and ( 3 ) Semantic Search Policy ; and one purely deterministic navigation/ interaction policy module ( Fig . 2 ) . At the start of an episode , the LP module processes the language instruction into a sequence of subtasks . Every time step , the semantic mapping module receives the egocentric RGB frame and updates the semantic map . If the goal object of the current subtask is not yet observed , the semantic search policy predicts a “ search goal ” at a coarse time scale ; until the next search goal is predicted , the agent navigates to the current search goal with the deterministic policy . If the goal is observed , the deterministic policy decides low-level controls for interaction actions ( e.g . “ Pick Up ” object ) . 4.1 LANGUAGE PROCESSING ( LP ) . The language processing ( LP ) module transforms high-level instructions into a structured sequence of subtasks ( Fig . 3 ) . It consists of two BERT ( Devlin et al. , 2018 ) submodules that receive the instruction as an input at the beginning of the episode . The first submodule ( BERT type classification ) receives the instruction and predicts the “ type ” of the instruction - one of the seven types stated in Appendix A.1 . The second submodule ( BERT argument classification ) receives both the instruction and the predicted type as input and predicts the “ arguments ” - ( 1 ) “ obj ” for the object to be picked up , ( 2 ) “ recep ” for the receptacle where “ obj ” should be ultimately placed , ( 3 ) “ sliced ” for whether “ obj ” should be sliced , and ( 4 ) “ parent ” for tasks with intermediate movable receptacles ( e.g . “ cup ” in “ Put a knife in a cup on the table ” of Appendix A.1 ) . An object in ALFRED is always an instance of either “ obj ” or “ recep ” ; “ parent ” objects are a subset of “ recep ” objects . We train a separate BERT model for each argument predictor . The two submodules are easily trainable with supervised learning since the type and the four arguments are provided in the training set . Models use only the CLS token for classification , and they do not share parameters ; all layers of “ bert-base-uncased ” were fine-tuned . Due to the patterned nature of instructions , we can match the predicted “ type ” of the instruction to a “ type template ” with blank arguments . For example , if the instruction is classified as the “ clean & place ” type , it is matched to the template “ ( Obj , PickUp ) , ( SinkBasin , Put ) , ( Faucet , ToggleOn ) , ( Faucet , ToggleOff ) , ( Obj , PickUp ) , ( Recep , Put ) ” . If the “ sliced ” argument is predicted to be true from argument classification , subtasks of “ ( Knife , PickUp ) , ( Obj , Slice ) , ( Sink , PutObject ) ” will be added at the beginning of the template ( with the ( Sink , PutObject ) to make the agent drop the knife ) . Filling in the “ type template ” with predictions of the second model , we obtain a list of subtasks ( bottom of Fig . 3b ) to be completed in the current episode .

Summary: This paper presents FILM (Following Instructions in Language with Modular methods), a model built for the ALFRED dataset. FILM consists of:   1) A language processing module which uses a BERT backbone to (a) predict 1 out of 7 task types in the ALFRED ontology and (b) predicts slot categories for populating a template (1 pre-defined per task).   2) A semantic mapping module which uses segmentation and depth prediction to populate a 2D spatial map of object class locations.   3) A semantic search policy which uses a CNN to map from the semantic map to subgoal object locations.   4) A deterministic policy which is used to navigate to either (a) a subgoal object location if it exists in the semantic map or (b) a sampled location from the search policy if not, and performs a predefined object interaction based on the current subgoal from the task template.  In my perception the main technical contribution of the work comes in the form of the language processing module, which provides much of the structure used for planning actions by the agent.   The paper also contributes a semantic search policy for reasoning over a semantic map to predict subgoal object locations (instead of randomly picking a new search location).   Experimentally, the paper presents:   1) SOTA results on the task when only the high-level instruction is given (low-level instructions witheld).   2) Ablations showing a bottleneck from errors in the perceptual side and marginal gains from adding low-level language instructions.   3) A taxonomy and distribution of error modes for the presented model, showing most errors come from either the language prediction, goal object location prediction, or goal objects being in closed receptacles (which the deterministic policy does not open).   4)  An analysis showing that task type length does not correlate with performance.