Year: 2021

Abstract:
Recent methods for embodied instruction following are typically trained end-toend using imitation learning. This often requires the use of expert trajectories and low-level language instructions. Such approaches assume learned hidden states will simultaneously integrate semantics from the language and vision to perform state tracking, spatial memory, exploration, and long-term planning. In contrast, we propose a modular method with structured representations that (1) builds a semantic map of the scene, and (2) performs exploration with a semantic search policy, to achieve the natural language goal. Our modular method achieves SOTA performance (24.46%) with a substantial (8.17 % absolute) gap from previous work while using less data by eschewing both expert trajectories and low-level instructions. Leveraging low-level language, however, can further increase our performance (26.49%).1 Our findings suggest that an explicit spatial memory and a semantic search policy can provide a stronger and more general representation for state-tracking and guidance, even in the absence of expert trajectories or lowlevel instructions.

heading: 1 INTRODUCTION
text: Human intelligence simultaneously processes data of multiple modalities, including but not limited to natural language and egocentric vision, in an embodied environment. Powered by the success of machine learning models in individual modalities (Devlin et al., 2018; He et al., 2016; Voulodimos et al.; Anderson et al., 2018a), there has been growing interest to build multimodal embodied agents that perform complex tasks. An incipient pursuit of such interest was to solve the task of Vision Language Navigation (VLN), for which the agent is required to navigate to the goal area given a language instruction (Anderson et al., 2018b; Fried et al., 2018; Zhu et al., 2020).
Embodied instruction following (EIF) presents a more complex and human-like setting than VLN or Object Goal Navigation (Gupta et al., 2017; Chaplot et al., 2020b; Du et al., 2021); beyond just navigation, agents are required to execute sequences of sub-tasks that entail both navigation and interaction actions from a language instruction (Fig. 1). The additional challenges posed by EIF are threefold - the agent has to understand compositional instructions of multiple types and subtasks, choose actions from a large action space and execute them for longer horizons, and localize objects in a fine-grained manner for interaction (Nguyen et al., 2021).
Most existing methods (Zhang & Chai, 2021; Kim et al., 2021; Nottingham et al., 2021) for EIF have relied on neural memory of various types (transformer embeddings, LSTM state), trained end-to-end with expert trajectories upon raw or pre-processed language/visual inputs. However, EIF remains a very challenging task for end-to-end methods as they require the neural net to simultaneously learn state-tracking, building spatial memory, exploration, long-term planning, and low-level control.
In this work, we propose FILM (Following Instructions in Language with Modular methods). FILM consists of several modular components that each (1) processes language instructions into structured forms (Language Processing), (2) converts egocentric visual input into a semantic metric map (Semantic Mapping), (3) predicts a search goal location (Semantic Search Policy), and (4) outputs subsequent navigation/ interaction actions (Deterministic Policy). FILM overcomes some
1At the time of submission, our model was the State of the Art on the official leaderboard of ALFRED: https://leaderboard.allenai.org/alfred/submissions/public.
of the shortcomings of previous methods by leveraging a modular design with structured spatial components. Unlike many of the existing methods for EIF, FILM does not require any input that provides sequential guidance, namely expert trajectories or low-level language instructions. While Blukis et al. (2021) recently introduced a method that uses a structured spatial memory, it comes with some limitations from the lack of explicit semantic search and the reliance on expert trajectories.
On the ALFRED (Shridhar et al., 2020) benchmark, FILM achieves State-of-the-Art performance (24.46%) with a large margin (8% absolute) from the previous SOTA (Blukis et al., 2021). Most approaches rely on low-level instructions, and we too find that including them leads to an additional 2% improvement in success rate (26.49%). FILM’s strong performance and our analysis indicate that an explicit structured spatial memory coupled with a semantic search policy can provide better state-tracking and exploration, even in the absence of expert trajectories or low-level instructions.

heading: 2 RELATED WORK
text: A plethora of works have been published on embodied vision and language tasks, such as VLN (Anderson et al., 2018b; Fried et al., 2018; Zhu et al., 2020), Embodied Question Answering (Das et al., 2018; Gordon et al., 2018), and topics of multimodal representation learning (Wang et al., 2020; Bisk et al., 2020), such as Embodied Language Grounding (Prabhudesai et al., 2020). On Visual Language Navigation, which is the most comparable to the setting of our work, methods with impressive performances Ke et al. (2019); Wang et al. (2019); Ma et al. (2019) have been proposed since the introduction of R2R Anderson et al. (2018b). While far from conquering VLN, these methods have shown up to 61% success rate on unseen test environments Ke et al. (2019).
On the more challenging task of Embodied Instruction Following (EIF), multiple methods have been proposed with differing levels of modularity in the model structure. As a baseline, Shridhar et al. (2020) has presented a Seq2Seq model with an attention mechanism and a progress monitor, while Pashevich et al. (2021) proposed to replace to seq2seq model with an episodic transformer. These methods take the concatenation of language features, visual features, and past trajectories as input and predict the subsequent action end-to-end. On the other hand, Kim et al. (2021); Zhang & Chai (2021); Nguyen et al. (2021) modularly process raw language and visual inputs into structured forms, while keeping a separate “action prediction module” that outputs low-level actions given processed language outputs. Their “action taking module” itself is trained end-to-end and
relies on neural memory that “implicitly” tracks all of spatial, progressive, and states of the agent. Unlike these methods, FILM’s structured language/ spatial representations make reasons for failure transparent and elucidates directions to improve individual components.
Recently, Blukis et al. (2021) has proposed a more modular method with a persistent and structured spatial memory. Language and visual input are transformed into respectively high-level actions and the 3D map. With the 3D map and high-level actions as input, low-level actions are predicted with a value-iteration network (VIN). Navigation goals for the VIN are sampled from a model trained on interaction pose labels from expert trajectories. Among all proposed methods for EIF, FILM necessitates the least information (neither low-level instructions nor expert trajectories are needed, although the former can be taken as an additional input). Furthermore, FILM addresses the problem of search/ exploration of goal objects.
Various works in visual navigation with semantic mapping are also relevant. Simultaneous Localization and Mapping (SLAM) methods, which build 2D or 3D obstacle maps, have been widely used (Fuentes-Pacheco et al., 2015; Izadi et al., 2011; Snavely et al., 2008). In contrast to these works, recent methods (Chaplot et al., 2020b;a) build semantic maps with differentiable projection operations, which restrain egocentric prediction errors amplifying in the map. The task of Chaplot et al. (2020b;a) is object goal navigation, a much simpler task compared to EIF. Furthermore, while Chaplot et al. (2020b) employs a semantic exploration policy, our and their semantic policies serve fundamentally different purposes; while their policy guides a general sense of direction among multiple rooms in the search for large objects (e.g. fridge), ours guides the search for potential locations of small and flat objects which have little chance of detection at a distance. Also, our semantic policy is conditioned on language instructions. Blukis et al. (2018a;b) also successfully utilized semantic 2D maps in grounded language navigation tasks. These works are for quadcopters, whose fields of view almost entirely cover the scene and the need for “search” or “exploration” is less crucial than for pedestrian agents. Moreover, their settings only involve navigation with a single subtask.

heading: 3 TASK EXPLANATION
text: We utilize the ALFRED benchmark. The agent has to complete household tasks given only natural language instructions and egocentric vision (Fig. 1). For example, the instruction may be given as “Put a heated apple on the counter,” with low-level instructions (which FILM does not use by default) further explaining step-by-step lower level actions. In this case, one way to “succeed” in this episode is to sequentially (1) pick up the apple, (2) put the apple in the microwave, (3) toggle the microwave on/off, (4) pick up the apple again, and (4) place it on the countertop. Episodes run for a significantly longer number of steps compared to benchmarks with only single subgoals ; expert trajectories, which are maximally efficient and perform only the strictly necessary actions (without any steps to search for an object), are often longer than 70 steps.
There are seven types of tasks (Appendix A.1), from relatively simple types (e.g. Pick & Place) to more complex ones (e.g. Heat & Place). Furthermore, the instruction may require that an object is “sliced” (e.g. Slice bread, cook it in the microwave, put it on the counter). An episode is deemed “success” if the agent completes all sub-tasks within 10 failed low-level actions and 1000 max steps.

heading: 4 METHODS
text: FILM consists of three learned modules: (1) Language Processing (LP), (2) Semantic Mapping, and (3) Semantic Search Policy; and one purely deterministic navigation/ interaction policy module (Fig. 2). At the start of an episode, the LP module processes the language instruction into a sequence of subtasks. Every time step, the semantic mapping module receives the egocentric RGB frame and updates the semantic map. If the goal object of the current subtask is not yet observed, the semantic search policy predicts a “search goal” at a coarse time scale; until the next search goal is predicted, the agent navigates to the current search goal with the deterministic policy. If the goal is observed, the deterministic policy decides low-level controls for interaction actions (e.g. “Pick Up” object).

heading: 4.1 LANGUAGE PROCESSING (LP)
text: The language processing (LP) module transforms high-level instructions into a structured sequence of subtasks (Fig. 3). It consists of two BERT (Devlin et al., 2018) submodules that receive the instruction as an input at the beginning of the episode. The first submodule (BERT type classification)
receives the instruction and predicts the “type” of the instruction - one of the seven types stated in Appendix A.1. The second submodule (BERT argument classification) receives both the instruction and the predicted type as input and predicts the “arguments” - (1) “obj” for the object to be picked up, (2) “recep” for the receptacle where “obj” should be ultimately placed, (3) “sliced” for whether “obj” should be sliced, and (4) “parent” for tasks with intermediate movable receptacles (e.g. “cup” in “Put a knife in a cup on the table” of Appendix A.1). An object in ALFRED is always an instance of either “obj” or “recep”; “parent” objects are a subset of “recep” objects. We train a separate BERT model for each argument predictor. The two submodules are easily trainable with supervised learning since the type and the four arguments are provided in the training set. Models use only the CLS token for classification, and they do not share parameters; all layers of “bert-base-uncased” were fine-tuned.
Due to the patterned nature of instructions, we can match the predicted “type” of the instruction to a “type template” with blank arguments. For example, if the instruction is classified as the “clean & place” type, it is matched to the template “(Obj, PickUp), (SinkBasin, Put), (Faucet, ToggleOn), (Faucet, ToggleOff), (Obj, PickUp), (Recep, Put)”. If the “sliced” argument is predicted to be true from argument classification, subtasks of “(Knife, PickUp), (Obj, Slice), (Sink, PutObject)” will be added at the beginning of the template (with the (Sink, PutObject) to make the agent drop the knife). Filling in the “type template” with predictions of the second model, we obtain a list of subtasks (bottom of Fig. 3b) to be completed in the current episode.

heading: 4.2 SEMANTIC MAPPING MODULE
text: We designed the semantic mapping module (Appendix A.2) with inspirations from prior work (Chaplot et al., 2020b). Egocentric RGB is first processed into depth map and instance segmentation, with MaskRCNN (He et al., 2017) (and its implementation by Shridhar et al. (2021)) and the depth prediction method of Blukis et al. (2021); details of the training are explained in Section 5 2 . These pre-trained, off-the-shelf models were finetuned on the training scenes of ALFRED. Once processed, the depth observation is transformed to a point cloud, of which each point is associated with the predicted semantic categories. Finally, the point cloud is binned into a voxel representation; this summed over height is the semantic map. The map is locally updated and aggregated over time.
The resulting semantic map is a (C+2)×M×M binary grid, where C is the number of object categories and each of the M×M cells represents a 5cm× 5cm space of the scene. The C channels each represent whether a particular object of interest was observed; the two extra channels denote whether obstacle exists and whether exploration happened in a particular 5cm× 5cm space. Thus, the C+2
2We use the publicly released code of Shridhar et al. (2021). We thank the authors of Blukis et al. (2021) for sharing their code with us.
channels are a semantic/spatial summary of the corresponding space. We use M = 240 (12 meters in the physical world) and C = 28+(number of additional subgoal objects in the current task). “28” is the number of “receptacle” objects (e.g. “Table”, “Bathtub”), which are usually large and easily detected; in the example of Fig. 1, there is one additional subgoal object (“Apple”). Please see Appendix A.2 on details of the dynamic handling of C.

heading: 4.3 SEMANTIC SEARCH POLICY
text: The semantic search policy outputs a coarse 2D distribution for potential locations of a small subgoal object (Fig. 4), given a semantic map with the 28 receptacle objects only (e.g. “Countertop”, “Shelf”). The discovery of a small object is difficult in ALFRED due to three reasons - (1) many objects are tiny (some instances of “pencil” occupies less than 200 pixels even at a very close view), (2) the field of view is small due to the camera horizon mostly being downward3, (3) semantic segmentation, despite being fine-tuned, cannot detect small objects at certain angles. The role of the semantic search policy is to predict search locations for small objects, upon the observed spatial configuration of larger ones. While existing works surmise the “implicit” learning of search locations from expert trajectories, we directly learn an explicit policy without expert trajectories.
The policy is trained via supervised learning. For data collection, we deploy the agent without the policy in the training set and gather the (1) semantic map with only receptacle objects and (2) the ground truth location of the subgoal object after every 25 steps. A model of 15 layers of CNN with max-pooling in between (details in Appendix A.3) outputs an N ×N grid, where N is smaller than the original map size M ; this is a 2D distribution for the potential location of the subgoal object. Finally, the KL divergence between this and a pseudo-ground truth “coarse” distribution whose mass is uniformly distributed over all cells with the true location of the subgoal object is minimized (minp KL(p||q) where p is the coarse ground truth and q is the coarse prediction). At deployment, the “search goal” is sampled from the predicted distribution, resized to match the original map size of M ×M (e.g. 240 × 240), with mass in the coarse N × N (e.g. 8 × 8) grid uniformly spread out to the ⌊MN ⌋ × ⌊ M N ⌋ area centered on it. Because arriving at the search goal requires time, the policy operates at a “coarse” time scale of 25 steps; the agent navigates towards the current search goal until the next goal is sampled or the subgoal object is found (more details in Section 4.4).
Fig. 4 shows a visualization of the semantic search policy’s outputs. The policy provides a reasonably close estimate of the true distribution; the predicted mass of “bowl” is shared around observed furniture that it can appear on, and that of “faucet” peaks around the sink/ the end of the bathtub. While we chose N = 8 as the grid size, Appendix A.4 provides a general bound for choosing N .

heading: 4.4 DETERMINISTIC POLICY
text: Given (1) the predicted subtasks, (2) the most recent semantic map, and (3) the search goal sampled at a coarse time scale, the deterministic policy outputs a navigation or interaction action (Fig. 2).
Let [(obj1, action1), ... , (objk, actionk)] be the list of subtasks and the current subtask be (obji, actioni). If obji is observed in the current semantic map, the closest obji is selected as the goal; otherwise, the sample from the semantic search policy is chosen as the goal (Section 4.3). The agent then navigates towards the goal via the Fast Marching Method (Sethian, 1996) and performs
3The agent mostly looks down 45◦ in FILM for correct depth prediction. Looking down is common in existing models as well (Kim et al., 2021; Zhang & Chai, 2021; Blukis et al., 2021).
the required interaction actions. While this “low-level” policy could be learned with imitation or reinforcement learning, we used a deterministic one based on the findings of earlier work that observed that the Fast Marching Method performs as well as a learned local navigation policy (Chaplot et al., 2020b). When the agent successfully executes the required interaction actioni (which can be determined by the change in the egocentric RGB), the pointer of subtasks is advanced to i+1 or the task is completed. More details and pseudocode are provided in Appendix A.5.

heading: 5 EXPERIMENTS AND RESULTS
text: We explain the metrics, evaluation splits, and baselines against which FILM is compared. Furthermore, we describe training details of each of the learned components of FILM.
Metrics Success Rate (SR) is a binary indicator of whether all subtasks were completed. The goalcondition success (GC) of a model is the ratio of goal-conditions completed at the end of an episode. For example, in the example of Fig. 1, there are three goal-conditions - a pan must be “cleaned”, a pan should rest on a countertop, and a “clean” pan should rest on a countertop. Both SR and GC can be weighted by (path length of the expert trajectory)/ (path length taken by the agent); these are called path length weighted SR (PLWSR) and path length weighted GC (PLWGC).
Evaluation Splits The test set consists of “Tests Seen” (1533 epsiodes) and “Tests unseen” (1529 episodes); the scenes of the latter entirely consist of rooms that do not appear in the training set, while those of the former only consist of scenes seen during training. Similarly, the validation set is partitioned into “Valid Seen” (820 epsiodes) and “Valid Unseen” (821 epsiodes). The official leaderboard ranks all entries by the SR on Tests Unseen.
Baselines There are two kinds of baselines: those that use low-level sequential instructions (Kim et al., 2021; Zhang & Chai, 2021; Nguyen et al., 2021; Pashevich et al., 2021) and those that do not (Nottingham et al., 2021; Blukis et al., 2021). While FILM does not necessitate low-level instructions, we report results with and without them and compare them against methods of both kinds.
Training Details of Learned Components In the LP module, BERT type classification and argument classification were trained with AdamW from the Transformer (Wolf et al., 2019) package; learning rates are 1e-6 for type classification and {1e-4,1e-5,5e-5,5e-5} for each of “object”, “parent”, “recep”, “sliced” argument classification. In the Semantic Mapping module, separate depth models for camera horizons of 45◦and 0◦were fine-tuned from an existing model of HLSM (Blukis et al., 2021), both with learning rate 1e-3 and the AdamW optimizer (epsilon 1e-6, weight decay 1e-2). Similarly, separate instance segmentation models for small and large objects were fine-tuned, starting from their respective parameters released by Shridhar et al. (2021), with learning rate 1e-3 and the SGD optimizer (momentum 0.9, weight decay 5e-4). Finally, the semantic search policy was trained with learning rate 1e-3 and the AdamW optimizer (epsilon 1e-6). Appendix A.2 and
A.3 discuss more details on the architectures of semantic mapping/ semantic search policy modules. We will release the trained models and the code so that readers can reproduce all experiments.

heading: 5.1 RESULTS
text: Table 1 shows test results. FILM achieves state-of-the-art performance across both seen and unseen scenes in the setting where only high-level instructions are given. It achieves 8.17% absolute (50.15% relative) gain in SR on Tests Unseen, and 0.66% absolute (2.63% relative) gain in SR on Tests Seen over HLSM, the previous SOTA.
FILM performs competitively even compared to methods that require low-level step-by-step instructions. They can be used as additional inputs to the LP module, with the low-level instruction appended to the high-level instruction for both BERT type classification and BERT argument classification. Under this setting, FILM achieves 11.06% absolute (71.68% relative) gain in SR on Tests Unseen compared to ABP. Notably, FILM performs similarly across Tests Seen and Tests Unseen, which implies FILM’s strong generalizability. This is in contrast to that methods that require lowlevel instructions, such as ABP, E.T., LWIT, MOCA, perform very well on Tests Seen but much less so on unseen scenes. In a Sim2Real situation, these methods will excel if the agent can be trained in the exact household it will be deployed in with multiple low-level instructions and expert trajectories. In the more realistic and cost-efficient setting where the agent is trained in a centralized manner and has to generalize to new scenes, FILM will be more adequate.
It is also notable that the semantic search policy significantly increases not only SR and GC, but also their path-length weighted versions. On Tests Seen, the gap of PLWSR between FILM with/ without semantic search is larger than the corresponding gap of SR (for both with/ without low-level instructions). This suggests that the semantic policy boosts the efficiency of trajectories.

heading: 5.2 ABLATIONS STUDIES AND ERROR ANALYSIS
text: Errors due to perception and language processing. To understand the importance of FILM’s individual modules, we consider ablations on the base method, the base method with low-level language, and with ground truth visual/ language inputs. Table 2 shows ablations on the development sets. While the improvement from gt depth is large in unseen scenes (10.64%), it is incremental on seen scenes (1.48%); on the other hand, gt segmentation significantly boosts performances in both cases (9.26% / 9.26%). Thus, among visual perception, segmentation is a bottleneck in both seen/ unseen scenes, and depth is a bottleneck only in the latter. On the other hand, while much gain in SR comes from using ground truth language (7.43 % / 4.22 %), that from adding low-level language as input is rather incremental.
Error modes. Table 3 shows common error modes of FILM; the metric is the percent of episodes that failed (in SR) from a particular error out of all failed episodes. The main failures in valid unseen scenes are due to failures in (1) locating the subgoal object (due to the small field of view, imperfect segmentation, ineffective exploration), (2) locating the subgoal object because it is in a closed re-
ceptacle (cabinet, drawer, etc), (3) interaction (due to object being too far or not in field of view, bad segmentation mask), (4) navigation (collisions), (5) correctly processing language instructions, (6) others, such as the deterministic policy repeating a loop of actions from depth/ segmentation failures and 10 failed actions accruing from a mixture of different errors. A failed episode is classified as one of (1), ..., (6), to the error type that occurs “earlier” (e.g. If the subtasks were processed incorrectly and also there were 10 consecutive collisions, this episode is classified as (5) (failure in incorrectly processsing language instructions) since the LP module comes “earlier” than the collisions). More details in Appendix A.6. As seen in Table 3, goal object not found is the most common error mode, typically due to objects being small and not visible from a distance or certain viewpoints. Results of the next subsection show that this error is alleviated by the semantic search policy in certain cases.
Performance over different task types. To understand FILM’s strengths/ weaknesses across different types of tasks, we further ablate validation results by task type in Table 4. Figure 5 shows the average number of subtasks completed for failed episodes, by task type. First, the SR and GC for “Stack & Place” is remarkably low. Second, the number of the subtasks entailed with the task type does not strongly correlate with performance. While “Heat & Place” usually involves three more subtasks than “Pick & Place”, the metrics for the former are much higher than those of the latter. Since task types inevitably occur in different kinds of scenes (e.g. “Heat & Place” only occurs in kitchens) and therefore involve different kinds of objects (e.g. “Heat & Place” involves food only), the results suggest that the success of the first PickUp action largely depends on the kinds of the scene and size and type of the subgoal objects rather than number of subtasks.
While the above error analysis is specific to FILM, its implications regarding visual perception may generally represent the weaknesses of existing methods for EIF, since most recent methods (ABP, HLSM, HiTUT, LWIT, E.T.) use the same family of segmentation/ detection models as FILM, such as Mask-RCNN and Fast-RCNN (Wang et al., 2017). Specifically, it could be that the inability to find a subgoal object is a major failure mode in the mentioned existing methods as well. On the other hand, FILM is not designed to search inside closed receptacles (e.g. cabinets), although subgoal objects dwell in receptacles quite frequently (Table 3); a future work to extend FILM should learn to perform a more active search.
5.3 EFFECTS OF THE SEMANTIC SEARCH POLICY
Table 5: Dev set results of FILM with/ without the semantic search policy.
Method Validation Unseen
% 1st Goal Found SR
HLSM (Blukis et al., 2021) N/A 11.8
FILM with Search 80.51 20.09 FILM w.o. Search 76.12 19.85 With Valid Unseen as the development set, we observed that the semantic search policy significantly helps to find small objects (Table 5); we use the percent of episodes in which the first goal object was found (%1st Goal Found) as a proxy, since it is usually small from that it can be
picked up (e.g. “Apple”, “Pen”). Thus, we use FILM with semantic search as the “base method” (default) for all experiments/ ablations.
To further analyze when the semantic search policy especially helps, we ablate on room sizes and task types. Table 6 shows the SR and %1st Goal Found with and without search, by room size (details on the assignment of Room Size are in Appendix A.7). As expected, the semantic policy increases both metrics, especially so in large scenes. This is desirable since the policy makes the agent less disoriented in difficult scenarios (large scenes); the model without it is more susceptible to failing even the first subtask. Figure 6 is consistent with the trend of Table 6; it shows example trajectories of FILM with and without the semantic search policy in a large kitchen scene. Since the countertop appears in the bottom right quadrant of the map, it is desirable that the agent travels there to search for a “knife”. While FILM travels to this area frequently (straight red line in Fig.6), FILM without semantic search mostly wanders in irrelevant locations (e.g. the bottom left quadrant).
Table 7 further shows the performance with and without search by task type. Notably, the gap of performance for the “clean & place” type is very large. In the large kitchen scene of “Valid Unseen” (Fig. 6), the “Sink” looks very flat from a distance and is hardly detected. The semantic policy induces the agent to travel near the countertop area and improves the localization of the 1st Recep (“Sink”) for the “clean & place” type (Table 7). In conclusion, the semantic policy improves the localization of small and flat objects in large scenes.

heading: 6 CONCLUSION
text: We proposed FILM, a new modular method for embodied instruction following which (1) processes language instructions into structured forms (Language Processing), (2) converts egocentric vision into a semantic metric map (Semantic Mapping), (3) predicts a likely goal location (Semantic Search Policy), and (4) outputs subsequent navigation/ interaction actions (Algorithmic Planning). FILM achieves the state of the art on the ALFRED benchmark without any sequential supervision.

heading: ETHICS STATEMENT
text: This research is for building autonomous agents. While we do not perform any experiments with humans, practitioners may attempt to extend and apply this technology in environments with humans. Such potential applications of this research should take privacy concerns into consideration.
All learned models in this research were trained using Ai2Thor (Kolve et al., 2019). Thus, they may be biased towards North American homes.

heading: REPRODUCIBILITY STATEMENT
text: We thoroughly explain training details and model architectures in Section 5.1 and Appendix A.2, A.3. We will also release all trained models and the code at a future date.

heading: A APPENDIX

heading: A.1 TASK DEFINITION
text: High and low-level instructions are both available to agents. There are 7 types of tasks (Fig 7. b) and the sequence of subtasks is templated according to the task type.

heading: A.2 SEMANTIC MAPPING MODULE
text: Figure 8 is an illustration of the semantic mapping module. A depth map and instance segmentation is predicted from Egocentric RGB. Then the first and the later are respectively transformed into a point cloud and a semantic label of each point in the cloud, together producing voxels. The voxels are summed across height to produce the semantic map. Partial maps obtained at particular time steps are aggregated to the global map simply via “sum/ logical and.”
We dynamically control the number of objects C for efficiency (because there are more than 100 objects in total). All receptacle objects (for input to the semantic policy) and all non-receptacle objects that appear in the subtasks are counted in C. For example, in an episode with the subtask [(Pan, PickUp), (SinkBasin, Put), (Faucet, ToggleOn), (Faucet, ToggleOff), (Pan, PickUp), (Table, Put)], all receptacle objects and ”Pan”, ”Faucet” will be the C objects indicated on the map.

heading: A.3 SEMANTIC SEARCH POLICY MODULE
text: The map from the previous subsection is passed into 7 layers of convolutional nets, each with kernel size 3 and stride 1. There is maxpooling between any two conv nets, and after the last layer, there is softmax over the 64 (8 × 8) categories, for each of the Co (73) channels. At deployment/ validation, if the agent is currently searching for the cth object, then a search location is sampled from the cth channel of the outputted 8 × 8 × Co grid.
A.4 IMPACT OF GRID SIZE ON THE EFFECTIVENESS OF THE SEMANTIC SEARCH POLICY
While we chose N = 8, ⌊MN ⌋ = 30 for the size of the “coarse” cell of the semantic search policy, the desirable choice of N may be different if a practitioner attempts to transfer FILM to different scenes/ tasks. While a “too fine” semantic policy will be hard to train due to sparseness of labels, a “too coarse” one will spread the mass of the distribution to widely.
Let us examine the “coarse” and “actual” ground truth distributions just in one direction (e.g. the horizontal direction). Let FX(x), CX(x) be the “actual” and “coarse” ground truth CDFs in the horizontal direction. Also, let L = ⌊MN ⌋ If the goal object occurs “k” times in the horizontal direction, then,
sup x |FX(x)− CX(x)| ≤
1 k (1− 1 L ).
A similar result holds in the vertical direction. The bound above suggests that if the goal object occurs more frequently (smaller 1k ), then a coarser L (larger 1− 1 L ) is tolerable. On the other hand, if the goal object occurs very infrequently (larger 1k ), then a coarse M (larger 1 − 1 L ) will result in FX and CX becoming too different in the worst case. Thus, it is desirable that practitioners choose L (and in turn, N ) based on the frequency of their goal objects, on average. Furthermore, a search policy with adaptive grid sizing should be explored as future work.

heading: A.5 PSEUDOCODE FOR THE DETERMINISTIC POLICY
text: Following the discussion of Section 4.4, let [(obj1, action1), ... , (objk, actionk)] be the list of subtasks, where the current subtask is (obji, actioni). If obji is observed in the current semantic map, the closest obji is selected as the goal to navigate; otherwise, the sample from the semantic search policy is chosen as the goal (Section 4.3). The agent then navigates towards the closest obji via the Fast Marching Method (Sethian, 1996). Once the stop distance is reached, the agent rotates 8 times to the left (at camera horizon 0, 45, 90,...) until obji is detected in egocentric vision. Once obji is in the current frame, the agents decides to take actioni if two criteria are met: whether obji is in the “center” of the frame, or whether the minimum depth towards obji is in visibility distance of 1.5 meters). Otherwise, the agent “sidesteps” to keep obji in the center frame or continue rotating to the left with horizon 0/45 until obji is seen within visibility distance. If the agent executes actioni and fails, the agent “moves backwards” and the map gets updated.
Below, we present a pseudocode for the deterministic navigation/ interaction policy. We first present explanations of some terms.
• “visible” means that an object is in the current RGB frame, and minimum (predicted) depth from the agent to it is less than or equal to 1.5 meters (which is set by ALFRED).
• “FMM” is Fast Marching Method (Sethian, 1996). • We assume that a new RGB frame is given as time step← time step+ 1 • MoveBehind, SideStep, RotateBack are not actions in ALFRED; they are defined by us.
MoveBehind - RotateRight, MoveAhead, RotateLeft SideStep - RotateRight/Left, MoveAhead, RotateLeft/Right RotateBack - RotateRight, RotateRight
Algorithm 1 Navigation/ interaction algorithm in an episode
1: Input: List of goal tuples - [(obj1, action1), ... , (objk, actionk)] 2: Output: Task Success - True/False 3: 4: timestep← 1 5: goal pointer ← 1 6: Sample g from the semantic search policy 7: execute interaction← False 8: stop← False 9: subtask success← False
10: move pointer ← 0 11: task success← False 12: 13: obji ← objgoal pointer; actioni ← actiongoal pointer 14: 15: while goal pointer ≤ k do 16: while timestep ≤ 1000 do 17: update semantic map 18: 19: if stop then 20: if execute interaction then 21: Execute actioni 22: if actioni done successfully then 23: subtask success← True 24: else 25: if obji visible in current frame and obji in the center of the frame then 26: execute interaction← False 27: Execute LookDown 0◦ ▷ void action 28: else 29: if previous action was OpenObject or CloseObject and not subtask success then 30: Execute MoveBehind 31: else if previous action was PutObject and not subtask success then 32: Re-dilate g in the semantic map 33: Execute RotateBack 34: else if obji visible but not in center of the frame then 35: Execute SideStep 36: else ▷ Rotate with camera horizons 0◦, 45◦until obji is visible 37: if move pointer < 4 then 38: Execute RotateLeft 39: else 40: if move pointer == 4 then 41: Execute LookDown 45◦ 42: Execute RotateLeft 43: move pointer ← move pointer + 1 (mod 8) 44: 45: else 46: if not (obji found) then 47: Execute one of (RotateLeft, RotateRight, MoveAhead) with FMM to g 48: else 49: g ← closest obji in the semantic map 50: while distance to g ≥ 0.65 meters do 51: Execute one of (RotateLeft, RotateRight, MoveAhead) with FMM to g 52: if distance to g < 0.65 meters then 53: stop← True 54: timestep← timestep+ 1 55: 56: if timestep ≡ 0 (mod 25) then 57: Sample new g from the semantic search policy 58: 59: if subtask success then 60: goal pointer ← goal pointer + 1 61: obji ← objgoal pointer; actioni ← actiongoal pointer 62: move pointer ← 0 63: execute interaction← False 64: stop← False 65: subtask success← False 66: Sample new g from the semantic search policy 67: break 68: 69: if goal pointer == k + 1 then 70: task success← True

heading: A.6 MORE EXPLANATIONS ON TABLE 3
text: Table 3 shows common error modes and the percentage they take out of all failed episodes, with regards to SR. More specifically, it is showing the distribution of episodes into exactly one error mode, out of the 79.9% of all “Val Unseen” episodes that have failed (the episodes not in the 20.10% of Table 2). The common error modes are failures in (1) locating the subgoal object (due to the small field of view, imperfect segmentation, ineffective exploration), (2) locating the subgoal object because it is in a closed receptacle (cabinet, drawer, etc), (3) interaction (due to object being too far or not in field of view, bad segmentation mask), (4) navigation (collisions), (5) correctly processing language instructions, (6) others, such as the deterministic policy repeating a loop of actions from depth/ segmentation failures and 10 failed actions accruing from a mixture of different errors. These errors occur in the order of (5), (1)/ (2), (3), (4) in an episode, since the LP module operates in the beginning and the object has to be first localized to be interacted with, etc. If an episode ended with errors in multiple categories, it was classified as an example of an ”earlier” error in making Table 3. For example, if the language processing module made an error and later there were also 10 collisions, this episode shown as a case of error (5) in Table 3.

heading: A.7 ASSIGNMENTS OF ROOMS INTO “LARGE” AND “SMALL” IN VALID UNSEEN
text: There are 4 distinct scenes in Valid Unseen (one kitchen scene, one living room, one bed room, one bathroom). The kitchen (Large) has a significantly larger area than all the others (Small).

Review summaries:
The paper proposes a modular framework for solving instruction following tasks from the ALFRED benchmark. The approach uses a language module that takes the high-level instruction, predicts the instruction type and uses the oracle template for the instruction type to fill in the arguments (objects/receptacles) and generate a list of sub-tasks to be executed on those objects and receptacles. The agent builds and maintains a spatial+semantic map to keep track of state and uses it to plan the next waypoint (which object/receptacle to navigate to next) as its intermediate goal. Their approach demonstrates the ability to ground small objects on the top-down semantic map and yields SoTA performance on the ALFRED benchmark. The authors also present a comprehensive analysis of their ablations and failure modes.

The paper presents a modular approach for solving Vision Language Navigation tasks and test it in ALFRED, a recent benchmark for that task. Compared with other benchmarks, ALFRED features a higher number of actions, objects, and an agent operating with an egoistic view. Solving the task requires multiple high-level actions, combined with low-level actions and exploration. As suggested by the authors of ALFRED, the benchmark requires exploiting the hierarchical structure, reusing skills in multiple contexts, and structured reasoning. The submission uses precisely all these aspects to achieve a significant improvement over the next entry in the ALFRED leaderboard.

This paper presents a modular system (FILM) for egocentric instruction following in the ALFRED environment. The system does not require expert trajectories and can operate without low-level instruction sequences. The system has the following components:  1. A language processing module that maps high-level instructions "Drop a clean pan on the table" to a low-level sub-task sequence. This module makes use of a number of BERT based classifiers that:     a.  classifiy the high level instruction into one of seven possible goal types, each of which is deterministically associated with a templated sub-task sequence     b. identify object IDs to fill in the slots in the template identified in (a)  2. A semantic mapping module that populates a grid with binary classifications that indicate object presence; obstacle presence; and whether the location has been explored. This module is inspired by previous work on egocentric navigation.  3. A novel semantic search policy that predicts the likely location of small sub-goal objects (bowl) on the basis of large receptacle objects (table). This is trained via supervised learning on sampled trajectories and it samples goals at a coarse tiem scale of every 25 steps.  4. A deterministic policy that operates within each 25 step search goal period. This makes use of the Fast Marching Method and is inspired by previous work on this task.  Together, these modules make up a system that generalizes to unseen environments much better than previous work. The new semantic search policy provide small but consistent improvements over a version of FILM that has this module removed.  FILM is evaluated in both the setting where low level instruction sequences are provided, and the setting where only the high-level goal is provided and FILM must map this goal to low-level instructions via module (1). In both settings, FILM significantly outpeforms previous work on unseen environments. When low-level instructions are provided, FILM does worse than some previous work on seen environments. When low-level instructions aren't provided, FILM does marginally better than previous work on seen environments.  Experiments show that the semantic search policy increases path-weighted metrics proportionally more than absolute metrics, suggesting that this policy improves efficiency as well as overall success of the search. Targeted evaluations also show that this search policy is particularly useful in large rooms, where the agent cannot see many target objects, and for the "Clean and Place" task that involves a difficult to detect object (sink) contained in a simple to detect object (countertop).

This paper presents FILM (Following Instructions in Language with Modular methods), a model built for the ALFRED dataset. FILM consists of:   1) A language processing module which uses a BERT backbone to (a) predict 1 out of 7 task types in the ALFRED ontology and (b) predicts slot categories for populating a template (1 pre-defined per task).   2) A semantic mapping module which uses segmentation and depth prediction to populate a 2D spatial map of object class locations.   3) A semantic search policy which uses a CNN to map from the semantic map to subgoal object locations.   4) A deterministic policy which is used to navigate to either (a) a subgoal object location if it exists in the semantic map or (b) a sampled location from the search policy if not, and performs a predefined object interaction based on the current subgoal from the task template.  In my perception the main technical contribution of the work comes in the form of the language processing module, which provides much of the structure used for planning actions by the agent.   The paper also contributes a semantic search policy for reasoning over a semantic map to predict subgoal object locations (instead of randomly picking a new search location).   Experimentally, the paper presents:   1) SOTA results on the task when only the high-level instruction is given (low-level instructions witheld).   2) Ablations showing a bottleneck from errors in the perceptual side and marginal gains from adding low-level language instructions.   3) A taxonomy and distribution of error modes for the presented model, showing most errors come from either the language prediction, goal object location prediction, or goal objects being in closed receptacles (which the deterministic policy does not open).   4)  An analysis showing that task type length does not correlate with performance.