FILM: Following Instructions in Language with Modular Methods

1 INTRODUCTION . Human intelligence simultaneously processes data of multiple modalities , including but not limited to natural language and egocentric vision , in an embodied environment . Powered by the success of machine learning models in individual modalities ( Devlin et al. , 2018 ; He et al. , 2016 ; Voulodimos et al . ; Anderson et al. , 2018a ) , there has been growing interest to build multimodal embodied agents that perform complex tasks . An incipient pursuit of such interest was to solve the task of Vision Language Navigation ( VLN ) , for which the agent is required to navigate to the goal area given a language instruction ( Anderson et al. , 2018b ; Fried et al. , 2018 ; Zhu et al. , 2020 ) . Embodied instruction following ( EIF ) presents a more complex and human-like setting than VLN or Object Goal Navigation ( Gupta et al. , 2017 ; Chaplot et al. , 2020b ; Du et al. , 2021 ) ; beyond just navigation , agents are required to execute sequences of sub-tasks that entail both navigation and interaction actions from a language instruction ( Fig . 1 ) . The additional challenges posed by EIF are threefold - the agent has to understand compositional instructions of multiple types and subtasks , choose actions from a large action space and execute them for longer horizons , and localize objects in a fine-grained manner for interaction ( Nguyen et al. , 2021 ) . Most existing methods ( Zhang & Chai , 2021 ; Kim et al. , 2021 ; Nottingham et al. , 2021 ) for EIF have relied on neural memory of various types ( transformer embeddings , LSTM state ) , trained end-to-end with expert trajectories upon raw or pre-processed language/visual inputs . However , EIF remains a very challenging task for end-to-end methods as they require the neural net to simultaneously learn state-tracking , building spatial memory , exploration , long-term planning , and low-level control . In this work , we propose FILM ( Following Instructions in Language with Modular methods ) . FILM consists of several modular components that each ( 1 ) processes language instructions into structured forms ( Language Processing ) , ( 2 ) converts egocentric visual input into a semantic metric map ( Semantic Mapping ) , ( 3 ) predicts a search goal location ( Semantic Search Policy ) , and ( 4 ) outputs subsequent navigation/ interaction actions ( Deterministic Policy ) . FILM overcomes some 1At the time of submission , our model was the State of the Art on the official leaderboard of ALFRED : https : //leaderboard.allenai.org/alfred/submissions/public . of the shortcomings of previous methods by leveraging a modular design with structured spatial components . Unlike many of the existing methods for EIF , FILM does not require any input that provides sequential guidance , namely expert trajectories or low-level language instructions . While Blukis et al . ( 2021 ) recently introduced a method that uses a structured spatial memory , it comes with some limitations from the lack of explicit semantic search and the reliance on expert trajectories . On the ALFRED ( Shridhar et al. , 2020 ) benchmark , FILM achieves State-of-the-Art performance ( 24.46 % ) with a large margin ( 8 % absolute ) from the previous SOTA ( Blukis et al. , 2021 ) . Most approaches rely on low-level instructions , and we too find that including them leads to an additional 2 % improvement in success rate ( 26.49 % ) . FILM ’ s strong performance and our analysis indicate that an explicit structured spatial memory coupled with a semantic search policy can provide better state-tracking and exploration , even in the absence of expert trajectories or low-level instructions . 2 RELATED WORK . A plethora of works have been published on embodied vision and language tasks , such as VLN ( Anderson et al. , 2018b ; Fried et al. , 2018 ; Zhu et al. , 2020 ) , Embodied Question Answering ( Das et al. , 2018 ; Gordon et al. , 2018 ) , and topics of multimodal representation learning ( Wang et al. , 2020 ; Bisk et al. , 2020 ) , such as Embodied Language Grounding ( Prabhudesai et al. , 2020 ) . On Visual Language Navigation , which is the most comparable to the setting of our work , methods with impressive performances Ke et al . ( 2019 ) ; Wang et al . ( 2019 ) ; Ma et al . ( 2019 ) have been proposed since the introduction of R2R Anderson et al . ( 2018b ) . While far from conquering VLN , these methods have shown up to 61 % success rate on unseen test environments Ke et al . ( 2019 ) . On the more challenging task of Embodied Instruction Following ( EIF ) , multiple methods have been proposed with differing levels of modularity in the model structure . As a baseline , Shridhar et al . ( 2020 ) has presented a Seq2Seq model with an attention mechanism and a progress monitor , while Pashevich et al . ( 2021 ) proposed to replace to seq2seq model with an episodic transformer . These methods take the concatenation of language features , visual features , and past trajectories as input and predict the subsequent action end-to-end . On the other hand , Kim et al . ( 2021 ) ; Zhang & Chai ( 2021 ) ; Nguyen et al . ( 2021 ) modularly process raw language and visual inputs into structured forms , while keeping a separate “ action prediction module ” that outputs low-level actions given processed language outputs . Their “ action taking module ” itself is trained end-to-end and relies on neural memory that “ implicitly ” tracks all of spatial , progressive , and states of the agent . Unlike these methods , FILM ’ s structured language/ spatial representations make reasons for failure transparent and elucidates directions to improve individual components . Recently , Blukis et al . ( 2021 ) has proposed a more modular method with a persistent and structured spatial memory . Language and visual input are transformed into respectively high-level actions and the 3D map . With the 3D map and high-level actions as input , low-level actions are predicted with a value-iteration network ( VIN ) . Navigation goals for the VIN are sampled from a model trained on interaction pose labels from expert trajectories . Among all proposed methods for EIF , FILM necessitates the least information ( neither low-level instructions nor expert trajectories are needed , although the former can be taken as an additional input ) . Furthermore , FILM addresses the problem of search/ exploration of goal objects . Various works in visual navigation with semantic mapping are also relevant . Simultaneous Localization and Mapping ( SLAM ) methods , which build 2D or 3D obstacle maps , have been widely used ( Fuentes-Pacheco et al. , 2015 ; Izadi et al. , 2011 ; Snavely et al. , 2008 ) . In contrast to these works , recent methods ( Chaplot et al. , 2020b ; a ) build semantic maps with differentiable projection operations , which restrain egocentric prediction errors amplifying in the map . The task of Chaplot et al . ( 2020b ; a ) is object goal navigation , a much simpler task compared to EIF . Furthermore , while Chaplot et al . ( 2020b ) employs a semantic exploration policy , our and their semantic policies serve fundamentally different purposes ; while their policy guides a general sense of direction among multiple rooms in the search for large objects ( e.g . fridge ) , ours guides the search for potential locations of small and flat objects which have little chance of detection at a distance . Also , our semantic policy is conditioned on language instructions . Blukis et al . ( 2018a ; b ) also successfully utilized semantic 2D maps in grounded language navigation tasks . These works are for quadcopters , whose fields of view almost entirely cover the scene and the need for “ search ” or “ exploration ” is less crucial than for pedestrian agents . Moreover , their settings only involve navigation with a single subtask . 3 TASK EXPLANATION . We utilize the ALFRED benchmark . The agent has to complete household tasks given only natural language instructions and egocentric vision ( Fig . 1 ) . For example , the instruction may be given as “ Put a heated apple on the counter , ” with low-level instructions ( which FILM does not use by default ) further explaining step-by-step lower level actions . In this case , one way to “ succeed ” in this episode is to sequentially ( 1 ) pick up the apple , ( 2 ) put the apple in the microwave , ( 3 ) toggle the microwave on/off , ( 4 ) pick up the apple again , and ( 4 ) place it on the countertop . Episodes run for a significantly longer number of steps compared to benchmarks with only single subgoals ; expert trajectories , which are maximally efficient and perform only the strictly necessary actions ( without any steps to search for an object ) , are often longer than 70 steps . There are seven types of tasks ( Appendix A.1 ) , from relatively simple types ( e.g . Pick & Place ) to more complex ones ( e.g . Heat & Place ) . Furthermore , the instruction may require that an object is “ sliced ” ( e.g . Slice bread , cook it in the microwave , put it on the counter ) . An episode is deemed “ success ” if the agent completes all sub-tasks within 10 failed low-level actions and 1000 max steps . 4 METHODS . FILM consists of three learned modules : ( 1 ) Language Processing ( LP ) , ( 2 ) Semantic Mapping , and ( 3 ) Semantic Search Policy ; and one purely deterministic navigation/ interaction policy module ( Fig . 2 ) . At the start of an episode , the LP module processes the language instruction into a sequence of subtasks . Every time step , the semantic mapping module receives the egocentric RGB frame and updates the semantic map . If the goal object of the current subtask is not yet observed , the semantic search policy predicts a “ search goal ” at a coarse time scale ; until the next search goal is predicted , the agent navigates to the current search goal with the deterministic policy . If the goal is observed , the deterministic policy decides low-level controls for interaction actions ( e.g . “ Pick Up ” object ) . 4.1 LANGUAGE PROCESSING ( LP ) . The language processing ( LP ) module transforms high-level instructions into a structured sequence of subtasks ( Fig . 3 ) . It consists of two BERT ( Devlin et al. , 2018 ) submodules that receive the instruction as an input at the beginning of the episode . The first submodule ( BERT type classification ) receives the instruction and predicts the “ type ” of the instruction - one of the seven types stated in Appendix A.1 . The second submodule ( BERT argument classification ) receives both the instruction and the predicted type as input and predicts the “ arguments ” - ( 1 ) “ obj ” for the object to be picked up , ( 2 ) “ recep ” for the receptacle where “ obj ” should be ultimately placed , ( 3 ) “ sliced ” for whether “ obj ” should be sliced , and ( 4 ) “ parent ” for tasks with intermediate movable receptacles ( e.g . “ cup ” in “ Put a knife in a cup on the table ” of Appendix A.1 ) . An object in ALFRED is always an instance of either “ obj ” or “ recep ” ; “ parent ” objects are a subset of “ recep ” objects . We train a separate BERT model for each argument predictor . The two submodules are easily trainable with supervised learning since the type and the four arguments are provided in the training set . Models use only the CLS token for classification , and they do not share parameters ; all layers of “ bert-base-uncased ” were fine-tuned . Due to the patterned nature of instructions , we can match the predicted “ type ” of the instruction to a “ type template ” with blank arguments . For example , if the instruction is classified as the “ clean & place ” type , it is matched to the template “ ( Obj , PickUp ) , ( SinkBasin , Put ) , ( Faucet , ToggleOn ) , ( Faucet , ToggleOff ) , ( Obj , PickUp ) , ( Recep , Put ) ” . If the “ sliced ” argument is predicted to be true from argument classification , subtasks of “ ( Knife , PickUp ) , ( Obj , Slice ) , ( Sink , PutObject ) ” will be added at the beginning of the template ( with the ( Sink , PutObject ) to make the agent drop the knife ) . Filling in the “ type template ” with predictions of the second model , we obtain a list of subtasks ( bottom of Fig . 3b ) to be completed in the current episode .

Summary: The paper proposes a modular framework for solving instruction following tasks from the ALFRED benchmark. The approach uses a language module that takes the high-level instruction, predicts the instruction type and uses the oracle template for the instruction type to fill in the arguments (objects/receptacles) and generate a list of sub-tasks to be executed on those objects and receptacles. The agent builds and maintains a spatial+semantic map to keep track of state and uses it to plan the next waypoint (which object/receptacle to navigate to next) as its intermediate goal. Their approach demonstrates the ability to ground small objects on the top-down semantic map and yields SoTA performance on the ALFRED benchmark. The authors also present a comprehensive analysis of their ablations and failure modes.